search

DLLXW / baby-llama2-chinese

用于从头预训练+SFT一个小参数量的中文LLaMa2的仓库；24G单卡即可运行得到一个具备简单中文问答能力的chat-llama2.
MIT License
2.34k stars 288 forks source link

fix: Fix attribute error and reduce memory usage during data processing #65

Open noahc1510 opened 3 months ago

noahc1510 commented 3 months ago

This PR fixed 'ChatGLMTokenizer' object has no attribute 'tokenizer' when transformers > 4.33.0. Tested on transformers==4.38.1 (#45, #63)

Reduce large memory usage during data processing.