Facico / Chinese-Vicuna

Chinese-Vicuna: A Chinese Instruction-following LLaMA-based Model —— 一个中文低资源的llama+lora方案,结构参考alpaca
https://github.com/Facico/Chinese-Vicuna
Apache License 2.0
4.14k stars 425 forks source link

支持不同词表大小的 llama模型训练 lora #226

Closed greatewei closed 1 year ago

greatewei commented 1 year ago

finetune_chat.py

model = get_peft_model(model, config) 前加上以下代码,可支持不同词表大小的 llama 模型训练

embedding_size = model.get_input_embeddings().weight.shape[0]
if len(train_tokenizer) != embedding_size:
      logger.info("resize the embedding size by the size of the tokenizer")
      model.resize_token_embeddings(len(train_tokenizer))