Where is tokenizer.model? tokenizer path

Lightning-AI / lit-llama

Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

Apache License 2.0

5.97k stars 518 forks source link

Where is tokenizer.model? tokenizer path #488

Open andreamigliorati opened 5 months ago

andreamigliorati commented 5 months ago

Hi, I'm confused about where to find the tokenizer:

--tokenizer_path checkpoints/lit-llama/tokenizer.model

Referring here to the readme:

Where can I download it?

daxian-lh commented 4 months ago

Did you solve the problem?