search

unslothai / unsloth

Finetune Llama 3.2, Mistral, Phi, Qwen 2.5 & Gemma LLMs 2-5x faster with 80% less memory
https://unsloth.ai
Apache License 2.0
18.37k stars 1.28k forks source link

Add Support for Pre-Training #1274

Open dame-cell opened 1 week ago

dame-cell commented 1 week ago

Will you guys be adding support for PreTraining ?

danielhanchen commented 1 week ago

Yes for all model support!

dame-cell commented 2 days ago

@danielhanchen For full pre-training support ,I assume you guys will still use A trainer of some sort to make it easier for people and in this tutorial by hugging-face causal language modeling

they showed how to use the Trainer from transformers to basically pre-train a model from scratch

I would like to contribute