search

sail-sg / lorahub

[COLM 2024] LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
MIT License
572 stars 35 forks source link

Using with CausalLM models #25

Closed aksh555 closed 1 month ago

aksh555 commented 1 month ago

Hi, thanks for the great library! Can the LoRA composition and weight learning approach be used with LoRA modules trained from AutoModelForCausalLM? If so, do you have any supporting scripts etc.

Thanks!

SivilTaram commented 1 month ago

@aksh555 Hi, thanks for your interest on our work! Yes definitely it should work well. First, you should train LoRA module based on your base model (e.g., Llama), then you can use https://github.com/sail-sg/lorahub/blob/main/example.py to do lorahub learning.

aksh555 commented 1 month ago

Thanks @SivilTaram but the current algorithm.py supports only AutoModelForSeq2SeqLM, so for using with Llama like model, the loading and dataset preparation needs to be changed?

SivilTaram commented 1 month ago

@aksh555 Thanks for reporting! Yes this should be fixed easily and I will try to fix it in 1 week. You can also try to directly modify it into AutoModel and then see if that works for your case! Thanks!