search

ssbuild / chatglm_finetuning

chatglm 6b finetuning and alpaca finetuning
1.54k stars 175 forks source link

eos_token_id是读的哪里配置呀？ #44

Closed lixiang1991 closed 1 year ago

lixiang1991 commented 1 year ago

开始训练时会打印这个配置，"eos_token_id": 20002 INFO:pytorch_lightning.utilities.rank_zero:HPU available: False, using: 0 HPUs ChatGLMConfig { "architectures": [ "ChatGLMModel" ], "auto_map": { "AutoConfig": "configuration_chatglm.ChatGLMConfig", "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration" }, "bos_token_id": 150004, "eos_token_id": 20002, "hidden_size": 4096, "initializer_range": 0.02,

在config/config.json中明明是"eos_token_id": 150005, 然后我看了chatglm-6b的官方模型中的config.json中也是"eos_token_id": 150005,

这个貌似也会对训练产生影响？因为我看打印的 'input_ids': array([ 20003, 20003, 20003, 20003, 20003, 20003, 20003, 20003, 中填充的空白是20003，如果和原始模型的不一样会出问题吧？

lixiang1991 commented 1 year ago

还有UserWarning: fan_in_fan_out is set to True but the target module is not a Conv1D. 我在data_utils.py中加了一行 'fan_in_fan_out': False, 还是会有这个警告。 也不知道要怎么改配置。求指教

ssbuild commented 1 year ago

Thanks for the feedback

  1. Filling is not recorded in the loss
    1. target modules differ from model to model. You can safely ignore the above warning