[Question] Merging LoRA weights with lmsys/vicuna-13b-v1.5 instead of liuhaotian/llava-v1.5-13b

[charismaticchiu]

Question

Hi I finetune my own model with LoRA, new-v1.5-13b-lora-665k-custom using finetune_lora.sh but have trouble merging the LoRA weights with lmsys/vicuna-13b-v1.5 backbone.

Can anyone shed some light? Thank you!

The command I used is

python scripts/merge_lora_weights.py \ --model-path ./checkpoints/new-v1.5-13b-lora-665k-custom \ --model-base lmsys/vicuna-13b-v1.5 \ --save-model-path ./checkpoints/merged/new-v1.5-13b-lora-665k-custom

And error is below

Loading LLaVA from base model... Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]/home1/XXX/.conda/envs/llava2/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() return self.fget.get(instance, owner)() Loading checkpoint shards: 100%|███████████████████████████████████████████████| 3/3 [00:19<00:00, 6.36s/it] Some weights of LlavaLlamaForCausalLM were not initialized from the model checkpoint at lmsys/vicuna-13b-v1.5 and are newly initialized: ['model.mm_projector.0.bias', 'model.mm_projector.0.weight', 'model.mm_projector.2.bias', 'model.mm_projector.2.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. /home1/XXX/.conda/envs/llava2/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:392: UserWarning: do_sample is set to False. However, temperature is set to 0.9 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset temperature. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed. warnings.warn( /home1/XXX/.conda/envs/llava2/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:397: UserWarning: do_sample is set to False. However, top_p is set to 0.6 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset top_p. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed. warnings.warn( Loading additional LLaVA weights... Loading LoRA weights... Merging LoRA weights... Model is loaded... Some non-default generation parameters are set in the model config. These should go into a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model) instead. This warning will be raised to an exception in v4.41. Non-default generation parameters: {'max_length': 4096} Your generation config was originally created from the model config, but the model config has changed since then. Unless you pass the generation_config argument to this model's generate calls, they will revert to the legacy behavior where the base generate parameterization is loaded from the model config instead. To avoid this behavior and this warning, we recommend you to overwrite the generation config model attribute before calling the model's save_pretrained, preferably also removing any generation kwargs from the model config. This warning will be raised to an exception in v4.41. Traceback (most recent call last): File "/home1/XXX/.conda/envs/llava2/lib/python3.10/site-packages/transformers/generation/configuration_utils.py", line 558, in save_pretrained raise ValueError(str([w.message for w in caught_warnings])) ValueError: [UserWarning('do_sample is set to False. However, temperature is set to 0.9 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset temperature.'), UserWarning('do_sample is set to False. However, top_p is set to 0.6 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset top_p.')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/project/LLaVA/scripts/merge_lora_weights.py", line 22, in merge_lora(args) File "/project/LLaVA/scripts/merge_lora_weights.py", line 10, in merge_lora model.save_pretrained(args.save_model_path) File "/home1/XXX/.conda/envs/llava2/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2364, in save_pretrained model_to_save.generation_config.save_pretrained(save_directory) File "/home1/XXX/.conda/envs/llava2/lib/python3.10/site-packages/transformers/generation/configuration_utils.py", line 560, in save_pretrained raise ValueError( ValueError: The generation config instance is invalid -- .validate() throws warnings and/or exceptions. Fix these issues to save the configuration.

Thrown during validation: [UserWarning('do_sample is set to False. However, temperature is set to 0.9 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset temperature.'), UserWarning('do_sample is set to False. However, top_p is set to 0.6 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset top_p.')]

[charismaticchiu]

@haotian-liu

Do you think it is caused by the non-default max_length or do_sample=True?

Should I manually unset max_length or set do_sample: true in the config.json? I actually tried both, but still has the same warning. Maybe the problem comes from the lmsys/vicuna-13b-v1.5 checkpoint?

[PointsCoder]

Same issue here

[PointsCoder]

I solved this issue by downgrading the Transformer version with pip install git+https://github.com/huggingface/transformers@v4.31-release