search

jzhang38 / TinyLlama

The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.
Apache License 2.0
7.64k stars 446 forks source link

TinyLlama-1.1B-orca-gpt4 #35

Closed acalatrava closed 12 months ago

acalatrava commented 12 months ago

First, I want to express my gratitude about this project. I think TinyLlama has a lot of potential and we're just starting to see it. Cudos!

I'm pretty new to this exciting field and this is the first time I fine-tuned a model. I used the "base" TinyLlama model (step-240k) to fine-tune using the sam-mosaic/orca-gpt4-chatml dataset but the result seems not as good as your v0.2 chat model.

I will keep working on this and I will share with you the models I create. I think that the RAG approach you guys are experimenting now is the good direction and I'll going to do some experiments with that too.

Anyway the model I produced is here in case you want to take a look: TinyLlama-1.1B-orca-gpt4

jzhang38 commented 12 months ago

Hi acalatrava thanks for that! I will include your model into the EVAL.md later today.

You can try out this dataset https://huggingface.co/datasets/shahules786/orca-best. This is a filtered version of openorca and should produce better results.