search

CLUEbenchmark / CLUENER2020

CLUENER2020 中文细粒度命名实体识别 Fine Grained Named Entity Recognition
https://arxiv.org/abs/2001.04351
1.45k stars 303 forks source link

TypeError: unhashable type: 'list' #76

Open SmartMapple opened 2 years ago

SmartMapple commented 2 years ago

when i use run_ner_crf.py, i want switch to xlnet model to do ner, i come aross follow error: File "pytorch_version\models\transformers\tokenization_utils.py", line 639, in split_on_tokens if sub_text not in self.added_tokens_encoder \ TypeError: unhashable type: 'list' my configuration is: 'xlnet': (XLNetConfig, XLNetCrfForNer, XLNetTokenizer) i thought this is caused by tokenize process, but i cannot locate the error appreciate for your help

YueCongPKU commented 2 years ago

您好！ 您的邮件我已收到，我会尽快查看的。谢谢哈！

JinFish commented 2 years ago

这个方法的目的是为了将文本分割为一个个token，但实际上在数据的预先处理中就已经将文本分割为token了，这一步实属没有必要。 而且这个方法不能接收list类型，只能接受str类型。