search

BohdanPetryshyn / code-llama-fim-fine-tuning

Fill-in-the-middle fine-tuning for the Code Llama model 🦙
MIT License
11 stars 3 forks source link

AttributeError: 'LlamaTokenizerFast' object has no attribute 'eot_token'. Did you mean: 'eos_token'? #3

Open hwaking opened 1 month ago

hwaking commented 1 month ago 
   self.eot_token_id = tokenizer.encode(
        tokenizer.eot_token, add_special_tokens=False
    )[0]

File "/home/hsai/works/code-llama-fim-fine-tuning/dataset.py", line 45, in init tokenizer.eot_token, add_special_tokens=False AttributeError: 'LlamaTokenizerFast' object has no attribute 'eot_token'. Did you mean: 'eos_token'?

BohdanPetryshyn commented 4 weeks ago

Hi @hwaking! Thank you for reporting this! I think it should be a problem with breaking changes in dependencies. I'll be able to take a look at it on weekends.

Please, let me know if you find a solution on your own!