search

AetherCortex / Llama-X

Open Academic Research on Improving LLaMA to SOTA LLM
Apache License 2.0
1.59k stars 101 forks source link

there is a endless loop in convert_tokens_to_ids(self.unk_token) and self._convert_token_to_id_with_added_voc(tokens) #23

Open 2018211801 opened 1 year ago

2018211801 commented 1 year ago

hi,this is part of my error information,thanks!

File "/mnt/data/wxc/workspace/Llama-X/src/transformers/src/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc return self.unk_token_id File "/mnt/data/wxc/workspace/Llama-X/src/transformers/src/transformers/tokenization_utils_base.py", line 1150, in unk_token_id return self.convert_tokens_to_ids(self.unk_token) File "/mnt/data/wxc/workspace/Llama-X/src/transformers/src/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids return self._convert_token_to_id_with_added_voc(tokens)