search

shibing624 / textgen

TextGen: Implementation of Text Generation models, include LLaMA, BLOOM, GPT2, BART, T5, SongNet and so on. 文本生成模型，实现了包括LLaMA，ChatGLM，BLOOM，GPT2，Seq2Seq，BART，T5，UDA等模型的训练和预测，开箱即用。
Apache License 2.0
937 stars 109 forks source link

how to use qlora ? #43

Open feng-1985 opened 1 year ago

shibing624 commented 1 year ago

just add bnb

MonkeyTB commented 1 year ago

bnb 里的4-bit 8-bit 和 qlora 里的 4bit NoramlFloat 不一样吧，还是我又理解错了😓

MonkeyTB commented 1 year ago

好像确实一样，底层依赖也是 bitsandbytes 这个包，。。。

shibing624 commented 1 year ago

是需要支持qlora训练吗？我加到todo里面。

stale[bot] commented 11 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.(由于长期不活动，机器人自动关闭此问题，如果需要欢迎提问)