[Usage] error during finetuning

[zzxslp]

Describe the issue

Issue: error occurred after finetuning is done and model can not be saved.

Command (default v1_5 finetuing script):

deepspeed llava/train/train_mem.py \
    --deepspeed ./scripts/zero3.json \
    --model_name_or_path lmsys/vicuna-13b-v1.5 \
    --version v1 \
    --data_path ./playground/data/llava_v1_5_mix665k.json \
    --image_folder ./playground/data \
    --vision_tower openai/clip-vit-large-patch14-336 \
    --pretrain_mm_mlp_adapter ./checkpoints/llava-v1.5-13b-pretrain/mm_projector.bin \
    --mm_projector_type mlp2x_gelu \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end False \
    --mm_use_im_patch_token False \
    --image_aspect_ratio pad \
    --group_by_modality_length True \
    --bf16 True \
    --output_dir ./checkpoints/llava-v1.5-13b \
    --num_train_epochs 1 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 50000 \
    --save_total_limit 1 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --dataloader_num_workers 4 \
    --lazy_preprocess True \
    --report_to wandb

Log:

File "/usr/local/lib/python3.10/dist-packages/transformers/generation/configuration_utils.py", line 354, in validate
raise ValueError(
ValueError: do_sample is set to False. However, temperature is set to 0.9 -- this flag is only used in sample-based generation modes. Set do_sample=True or unset temperature to continue.

According to this thread, the error is caused with HF 4.37.2 release. Downgrading to 4.36.2 seems to work.

[yzbx]

after update transformer from 4.31.0 to 4.37.2, llama2-based model cannot saved after finetune. the do_sample is set to False conflict with temperature > 0 and top_p > 0

• view generation/configuration_utils.py

  # 1. detect sampling-only parameterization when not in sampling mode
      if self.do_sample is False:
          greedy_wrong_parameter_msg = (
              "`do_sample` is set to `False`. However, `{flag_name}` is set to `{flag_value}` -- this flag is only "
              "used in sample-based generation modes. You should set `do_sample=True` or unset `{flag_name}`."
              + fix_location
          )
          if self.temperature != 1.0:
              warnings.warn(
                  greedy_wrong_parameter_msg.format(flag_name="temperature", flag_value=self.temperature),
                  UserWarning,
              )
          if self.top_p != 1.0:
              warnings.warn(
                  greedy_wrong_parameter_msg.format(flag_name="top_p", flag_value=self.top_p),
                  UserWarning,
              )
          if self.typical_p != 1.0:
              warnings.warn(
                  greedy_wrong_parameter_msg.format(flag_name="typical_p", flag_value=self.typical_p),
                  UserWarning,
              )
          if self.top_k != 50 and self.penalty_alpha is None:  # contrastive search uses top_k
              warnings.warn(
                  greedy_wrong_parameter_msg.format(flag_name="top_k", flag_value=self.top_k),
                  UserWarning,
              )
          if self.epsilon_cutoff != 0.0:
              warnings.warn(
                  greedy_wrong_parameter_msg.format(flag_name="epsilon_cutoff", flag_value=self.epsilon_cutoff),
                  UserWarning,
              )
          if self.eta_cutoff != 0.0:
              warnings.warn(
                  greedy_wrong_parameter_msg.format(flag_name="eta_cutoff", flag_value=self.eta_cutoff),
                  UserWarning,
              )