有计划支持LoRAMoE吗？

hiyouga / LLaMA-Factory

Unified Efficient Fine-Tuning of 100+ LLMs (ACL 2024)

https://arxiv.org/abs/2403.13372

Apache License 2.0

34.54k stars 4.26k forks source link

Open luyuntao92 opened 8 months ago

luyuntao92 commented 8 months ago

https://arxiv.org/pdf/2312.09979.pdf 《LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin》

看论文，效果好像挺不错的

No response

No response

zhhvvv commented 8 months ago

如果有更新，建议再follow一下这篇工作：https://arxiv.org/abs/2402.08562《Higher Layers Need More LoRA Experts》允许自定义设置每一层的专家个数

A11en0 commented 8 months ago

mark:)

BigBean666 commented 7 months ago

mark

Xingxiangrui commented 7 months ago

mark

edisonzf2020 commented 6 months ago

mark

yefangok commented 5 months ago

mark

nuass commented 5 months ago

mark

WenTingTseng commented 5 months ago

請問是否有更新，增加該方法

Krismile-yf commented 2 months ago

十分期待在LLaMA-Factory的基础上，增加LoRA-Moe的设置，这将增加了LLaMA的适用性。

GuanyaZhou1 commented 1 week ago

十分期待能够增加这个训练