search

TsinghuaAI / CPM-2-Finetune

Finetune CPM-2
MIT License
83 stars 21 forks source link

请教：使用中英文双语模型报了一下的错误： #38

Closed Chunhui-Zou closed 2 years ago

Chunhui-Zou commented 2 years ago

Traceback (most recent call last): File "/amax/home/zouchunhui/CPM-2-Finetune-master/finetune_cpm2.py", line 720, in main() File "/amax/home/zouchunhui/CPM-2-Finetune-master/finetune_cpm2.py", line 624, in main prompt_config[t]["init_ids"].extend(tokenizer.convert_tokens_to_ids([prompt_config[t]["default_inittoken"] for in range(pad_num)])) AttributeError: 'EncDecTokenizer' object has no attribute 'convert_tokens_to_ids'

Chunhui-Zou commented 2 years ago

已解决： 在tokenization_enc_dec_encn.py的class EncDecTokenizer加上： def convert_tokens_to_ids(self, tokens): return [self.encoder.get(x, self.encoder[""]) for x in tokens]

def convert_ids_to_tokens(self, ids):
    return [self.decoder[x] for x in ids]