Closed OnewayLab closed 3 weeks ago
Maybe it's an issue from Hugging Face Transformers. I found the following code in transformers/modeling_utils.py
:
class PreTrainedModel(nn.Module, ModuleUtilsMixin, GenerationMixin, PushToHubMixin, PeftAdapterMixin):
def save_pretrained(...):
...
# save the string version of dtype to the config, e.g. convert torch.float32 => "float32"
# we currently don't use this setting automatically, but may start to use with v5
dtype = get_parameter_dtype(model_to_save)
model_to_save.config.torch_dtype = str(dtype).split(".")[1]
...
What could we do to fix it...
Maybe it's an issue with unsloth. I got same error when use_unsloth: true.
### model
model_name_or_path: Qwen/Qwen2-1.5B-Instruct
flash_attn: fa2
# use_unsloth: true
### method
stage: sft
do_train: true
finetuning_type: full
bf16: true
### dataset
dataset: mine
template: qwen
cutoff_len: 4000
overwrite_cache: true
preprocessing_num_workers: 8
### output
output_dir: Qwen2-1.5B-Instruct
logging_steps: 10
save_steps: 10
save_strategy: steps
plot_loss: true
overwrite_output_dir: true
per_device_train_batch_size: 8
gradient_accumulation_steps: 4
learning_rate: 1.0e-4
num_train_epochs: 1
lr_scheduler_type: cosine
ddp_timeout: 180000000
### eval
val_size: 0.1
per_device_eval_batch_size: 8
eval_strategy: steps
eval_steps: 30
### log
report_to: wandb
run_name: Qwen2-1.5B-Instruct
@hiyouga This has fixed in unsloth. And it will be useful in next unsloth version or user can fix it follow now: https://github.com/unslothai/unsloth/pull/874#issue-2447675422
@hiyouga This has fixed in unsloth. And it will be useful in next unsloth version or user can fix it follow now: unslothai/unsloth#874 (comment)
Thanks for your reminding! I will close this issue.
Reminder
System Info
llamafactory
version: 0.8.3.dev0Reproduction
Command
Error Log
Expected behavior
Successful training.
Others
The problem occurs in the first step after saving the first checkpoint. I guess it's because
model.config.torch_dtype
changes fromtorch.bfloat16
to a stringfloat32
while saving the checkpoint.