Closed amulil closed 1 year ago
I believe there IS a tokenizer.json
in the huggingface repo:
https://huggingface.co/meta-llama/Llama-2-7b-hf/blob/main/tokenizer.json
I am not sure about the sentencepiece_model_pb2
error, which looks like a huggingface internal bug?
Feel free to re-open if needed.
Prerequisite
Type
I'm evaluating with the officially supported tasks/models/datasets.
Environment
Reproduces the problem - code/configuration sample
Reproduces the problem - command or script
Reproduces the problem - error message
Other information
But when I save the tokenizer to json, the error is missing.
What's the reason of it? The origin hugging face model has no tokenizer.json file.