Open ryf1123 opened 1 week ago
Hi @ryf1123 thanks for creating the issue. I think (2) is the way we intend for this to be done, unfortunately looks like it doesn't currently work! So glad you pointed this out. I think the problem is that for the Llama 3.2 Vision model the config is structured a bit differently.. so the num_attention_heads
field is still there, but it's just under the text_config
field.
The ideal case you described of loading in with PeftModel.from_pretrained
is exactly what we want to have and it should currently work for our text models (see the test plan in #933). But I think we need to make some changes to save the adapter weights properly for the multimodal model. I am gonna take a closer look at this, and will also assign to @pbontrager for further investigation since he's quite familiar with the multimodal key mappings
Hi, thanks for this amazing project. I was trying to finetune the lora model for Llama3.2 Vision which works fine and saved a adapter_0.pt; Then I wanted to use this adapter checkpoint for inference in a huggingface pipeline where i see some issues. I tried two ways as following and thanks in advance!
Weight format conversion: there is a file to convert meta (meta_model_0.pt) to tune, then to HF as in this _convert_weights.py. I tried to use it to convert a full model (non-lora), and it works for inference, but it does not work the adapter checkpoint as the parameters names are different.
Then I tried to set the
_component_
parameter totorchtune.training.FullModelHFCheckpointer
hoping to get a huggingface compatible model directly. Then I got the error: *** KeyError: 'num_attention_heads'.What would be ideal is to have a way to use the finetuned model in a way similar to the following? Note the following code does not work as there is no
adapter_config.json
under thecheckpointer/output_dir
path.To reproduce what I observed:
I am using the following command to run, with slightly modified config file:
tune run --nnodes 1 --nproc_per_node 1 lora_finetune_distributed --config ./finetune-llama32/11B _lora_debug.yaml
Main modification:
from
to
Full config file: