在使用checkpoint时，遇到了缺少collate_fn_demo的报错

[athos17]

报错如下： Traceback (most recent call last): File "E:\新建文件夹\FCGEC-main\model\STG-correction\preprocess_data.py", line 4, in from utils.argument import ArgumentGroup File "E:\新建文件夹\FCGEC-main\model\STG-correction\utils__init__.py", line 12, in from utils.collate import collate_fn_base, collate_fn_tagger, collate_fn_joint, collate_fn_tagger_V2, collate_fn_bertbase_tti, collate_fn_tagger_V2TTI, collate_fn_jointV2, collate_fn_demo ImportError: cannot import name 'collate_fn_demo' from 'utils.collate' (E:\新建文件夹\FCGEC-main\model\STG-correction\utils\collate.py) 请问这个函数是需要自己写吗？

[xlxwalex]

你好，你可以重新Clone一下，因为Github仓库的代码是我从我项目里摘的，可能我漏覆盖了collate文件，现在已经加上了，如果运行后还有其他问题，欢迎回复

[athos17]

您好，我在修改之后成功生成了test.csv文件，但在运行joint_evaluate.py时仍出现了如下报错： Traceback (most recent call last): File "E:\新建文件夹\FCGEC-main\model\STG-correction\joint_evaluate.py", line 148, in evaluate(args) File "E:\新建文件夹\FCGEC-main\model\STG-correction\joint_evaluate.py", line 43, in evaluate switch_test = SwitchDataset(args, test_dir, 'test') File "E:\新建文件夹\FCGEC-main\model\STG-correction\DataProcessor\SwitchDataset.py", line 17, in init self.tokenizer = BertTokenizer.from_pretrained(args.lm_path, cache_dir='./.cache') File "E:\新建文件夹\FCGEC-main\venv\lib\site-packages\transformers\tokenization_utils_base.py", line 1771, in from_pretrained resolved_vocab_files[file_id] = cached_file( File "E:\新建文件夹\FCGEC-main\venv\lib\site-packages\transformers\utils\hub.py", line 417, in cached_file resolved_file = hf_hub_download( File "E:\新建文件夹\FCGEC-main\venv\lib\site-packages\huggingface_hub\utils_validators.py", line 112, in _inner_fn validate_repo_id(arg_value) File "E:\新建文件夹\FCGEC-main\venv\lib\site-packages\huggingface_hub\utils_validators.py", line 160, in validate_repo_id raise HFValidationError( huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/datadisk2/xlxw/Resources/pretrained_models/roberta-base-chinese'. Use repo_type argument if needed. 希望您能告诉我怎么解决

[xlxwalex]

这个问题是因为你没有配置预训练模型的参数，你可以看看run_stg_joint.sh的PLM_PATH，需要设置为你自己的，如果你本地没有的话可以从HF里或者哈工大的repo里下载，PLM。

[athos17]

请问是在运行bash文件后运行joint_evaluate.py吗，我在更改参数后还是同样的报错

[xlxwalex]

PLM_PATH是你运行bash文件之前设置的，如果你直接使用joint_evaluate.py，需要在evaluate_joint_config.py文件中lm_path参数的默认值。

[athos17]

您好，在使用您提供的第二种方法后，我解决了上述报错，但又出现了以下报错 RuntimeError: Error(s) in loading state_dict for JointModel: size mismatch for tagger._hidden2t.linear.weight: copying a param with shape torch.Size([6, 768]) from checkpoint, the shape in current model is torch.Size([8, 768]). size mismatch for tagger._hidden2t.linear.bias: copying a param with shape torch.Size([6]) from checkpoint, the shape in current model is torch.Size([8]).

[athos17]

我已经解决了上述报错，谢谢您的帮助

[xlxwalex]

客气啦：）

[yazheng0307]

您好，在使用您提供的第二种方法后，我解决了上述报错，但又出现了以下报错 RuntimeError: Error(s) in loading state_dict for JointModel: size mismatch for tagger._hidden2t.linear.weight: copying a param with shape torch.Size([6, 768]) from checkpoint, the shape in current model is torch.Size([8, 768]). size mismatch for tagger._hidden2t.linear.bias: copying a param with shape torch.Size([6]) from checkpoint, the shape in current model is torch.Size([8]).

你好，我也遇到了相同的问题，请问是怎么解决的？ Unexpected key(s) in state_dict: "switch._encoder._bert.embeddings.position_ids", "tagger._encoder._bert.embeddings.position_ids", "generator._lmodel.bert.embeddings.position_ids". size mismatch for tagger._hidden2t.linear.weight: copying a param with shape torch.Size([6, 768]) from checkpoint, the shape in current model is torch.Size([8, 768]). size mismatch for tagger._hidden2t.linear.bias: copying a param with shape torch.Size([6]) from checkpoint, the shape in current model is torch.Size([8]).

[xlxwalex]

你好，我也遇到了相同的问题，请问是怎么解决的？ Unexpected key(s) in state_dict: "switch._encoder._bert.embeddings.position_ids", "tagger._encoder._bert.embeddings.position_ids", "generator._lmodel.bert.embeddings.position_ids". size mismatch for tagger._hidden2t.linear.weight: copying a param with shape torch.Size([6, 768]) from checkpoint, the shape in current model is torch.Size([8, 768]). size mismatch for tagger._hidden2t.linear.bias: copying a param with shape torch.Size([6]) from checkpoint, the shape in current model is torch.Size([8]).

@yazheng0307 你好，

如果你是直接用的joint_evaluate.py，需要修改evaluate_joint_config.py中的max_generate参数的默认值为5来匹配权重的维度。