please add compatibilty for encoder-only models

unslothai / unsloth

Finetune Llama 3.1, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

https://unsloth.ai

Apache License 2.0

13.31k stars 880 forks source link

please add compatibilty for encoder-only models #687

Open bobox2997 opened 1 month ago

bobox2997 commented 1 month ago

any way to get support for XLM-RoBERTA, DeBERTA et similia models? (asking for those specific models mostly because they have 1+ B parameters versions, but could be extended to all BERT and encoder-only models in general)

i see that there was another issue asking the same things, there were any progress since then?

timothelaborie commented 1 month ago

According to Mike, this might be coming in early July

danielhanchen commented 4 weeks ago

Hopefully :)