search

unslothai / unsloth

Finetune Llama 3, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory
https://unsloth.ai
Apache License 2.0
12.51k stars 815 forks source link

Using custom Llama-3 tokenizer #607

Open MikeMpapa opened 1 month ago

MikeMpapa commented 1 month ago

Hi, I have trained my custom Llama tokenizer and I was wondering how/if it would be feasible to train Llama-3 with Unclothe but use my tokenizer instead?

danielhanchen commented 1 month ago

You can load the tokenizer afterwards and separately after if that helps