search

microsoft / mttl

Building modular LMs with parameter-efficient fine-tuning.
MIT License
62 stars 3 forks source link

Issues when finetuning pretrained lora checkpoint for downstream tasks. #48

Closed prateeky2806 closed 1 week ago

prateeky2806 commented 8 months ago

Hi Lucas,

I was trying to run the MHR codebase to finetune on a downstream task after loading the pretrained lora checkpoint provided here. I am not able to load the pretrained lora checkpoints into the model and I getting some shape mismatch errors. I am pretty sure I am messing up something with the config files and hyperparameters setting. Would it be possible for you to provide a command that can be used to FT lora checkpoint on any downstream task say RTE?

I was able to load the poly checkpoint and finetune that further using the provided scripts here. It would be great if you can add scripts to finetune lora, ia3, and mhr pretrained checkpoints.

Thanks, Prateek

pclucas14 commented 6 months ago

Hi Prateek,

Sorry for the late reply; did you manage to solve this issue ?