Problem with tokenizer?

[shokhjakhonone]

I am writing to ask for your help with a problem I am having with the tokenizer. I have been trying to solve it for a while now, but I have been unsuccessful. However, I am having trouble with : Traceback (most recent call last): File "/content/baby-llama2-chinese/eval.py", line 81, in tokenizer=ChatGLMTokenizer(vocab_file='./chatglm_tokenizer/tokenizer.model') File "/content/baby-llama2-chinese/chatglm_tokenizer/tokenization_chatglm.py", line 68, in init super().init(padding_side=padding_side, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils.py", line 367, in init self._add_tokens( File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils.py", line 467, in _add_tokens current_vocab = self.get_vocab().copy() File "/content/baby-llama2-chinese/chatglm_tokenizer/tokenization_chatglm.py", line 112, in get_vocab vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)} File "/content/baby-llama2-chinese/chatglm_tokenizer/tokenization_chatglm.py", line 107, in vocab_size return self.tokenizer.n_words AttributeError: 'ChatGLMTokenizer' object has no attribute 'tokenizer'. Did you mean: 'tokenize'?

I would be very grateful if you could help me solve this problem. I am available to answer any questions you may have.

Thank you for your time and consideration.

[ringwraith]

Reverting to transformers==4.33.0 resolved the problem.

[buhe]

Python version too high,switch to 3.10

[buhe]

https://github.com/DLLXW/baby-llama2-chinese/pull/65