search

ContextualAI / gritlm

Generative Representational Instruction Tuning
https://arxiv.org/abs/2402.09906
MIT License
538 stars 39 forks source link

Full Finetuning or LoRA ? #5

Closed hjq133 closed 7 months ago

hjq133 commented 7 months ago

Great job! One question: Are you using full finetuning throughout the entire process? Have you tried LoRA's performance?

I remember in the previous work (https://arxiv.org/pdf/2401.00368.pdf), which also using LLM as embedder, they use LoRA and still get good performance.

Muennighoff commented 7 months ago

We do full-finetuning and did not try LoRA, but I think it should work! Let us know if you try it :)