Open rasbt opened 1 month ago
There is no modeling_*.py
file in the repo.
And the config.json
file looks pretty standard.
It can be just a matter of adding a config.
Update: there is a custom tokenizer - tekken
.
So, yeah, might not be so easy 🙃.
In case we want to pursue this, some findings from Daniel Han:
My findings for Mistral NeMo 12b:
Longer notes:
EOS token is untrained in the base model but trained in instruct - confirming with @MistralAI if this is a feature or a bug - could make finetunes break with NaNs and infinities. Mistral 7b does not have this issue. Only the embed_tokens, not the lm_head has this issue.
EOS token is auto appended. This can break finetuning and inference - collabed with @xenovacom to fix this quickly :)
3, Not 5120 for Wq but 4096 - HF transformers main branch already has a fix for this - please update transformers! Unsloth auto patches, so no need to update!
Not a Llama Tokenizer - was GPT2 Tokenizer, now generic PreTrainedTokenizer? Very interesting! Tokenizer compresses other languages more efficiently.
Support for tools & FIM (Fill in the middle tasks). Function calling, code completion etc.
Pad_token=10
1 million possible position embeddings - had to support dynamic sizing of Cos & Sin cached matrices to not go OOM (used 20GB!)
More details in our blog: https://unsloth.ai/blog/mistral-nemo
Our free Colab notebook can finetune 12b in a free 16GB Tesla T4 GPU exactly, and can do it 2x faster and use 60% less VRAM than HF+FA2! https://colab.research.google.com/drive/17d3U-CAIwzmbDRqbZ9NnpHxCkmXB6LZ0?usp=sharing
We also have a Kaggle notebook making finetuning 2x faster: https://kaggle.com/code/danielhanchen/kaggle-mistral-nemo-12b-unsloth-notebook
According to some first reports, this new model works great. In case we have the time, it would be a nice model to add as it would fill the "multilingual" niche. (Some people have been asking about models for various non-English languages.) Not sure if Gemma-2 already fulfills that too though.