Failed to convert to HF

[arbindpd96]

I am trying to convert a fine tuned checkpoint to hf so i can make the model work with llama.cpp Image attached for reference Any help would be appreciated.

Also in the latest commit

in accessory/main_finetune.py FuseAdam is imported as AdamW but used as FuseAdam. which causes it to break. also we are getting a key error on the key 'image' on this line print(f'Warning for truncation input!\nImage name: {data_item["image"]} question: {data_item["question"][:10]}') in accessory/data/alpaca.py

[ChrisLiu6]

1. fail to convert to HF

Hi, the problem is that the model architecture of llamaPeft is different from the original llama (lora and bias are added), but huggingface only contains the original llama implementation. If you want to convert such models to HF, you need to first reimplement the architecture using huggingface, and then write the conversion logic correspondingly.

2. AdamW and 3. Dataset key error Thank you for pointing out these bugs. We have fixed them in the latest commit.

[anuragdalia]

Hey thanks for that response.

Can these checkpoints be used with llama.cpp or as a quantized single bin file. or even be converted to ggml format? or all these would require custom scripts ?

[anuragdalia]

@ChrisLiu6 Hi Please suggest.

[ChrisLiu6]

I'm sorry, but I'm really not familiar with llama.cpp😣. Unfortunately, no one on our team is working on that. The followings are all I can provide on this topic:

• The convert_weights_to_hf script is designed exclusively for the original llama architecture. Therefore, you may fully finetune your model using Accessory and then convert it to Hugging Face. You can refer to this experiment.
• If you do require PEFT, consider finetuning only the norm and Lora. Afterward, merge the Lora weights into the original weights before the conversion to Hugging Face. This will make a checkpoint compatible with the original llama architecture. However, you'll need to implement the Lora merging logic on your own.
• If you do need to keep the bias terms in the model and convert it to HF, you may need to modify the HF llama implementation to make it contain the bias terms. You also need to modify the convert_weights_to_hf script to deal with biases.