search

jzhang38 / TinyLlama

The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.
Apache License 2.0
7.29k stars 423 forks source link

Llama 3 #186

Open cduk opened 1 month ago

cduk commented 1 month ago

Given that we have only Llama 3 70B and 8B, it would be useful to have a Tiny Llama based on the Llama 3 tokenizer so that we can use it as a drafting model for speculative decoding.

Are there plans to create a Llama 3 version?

RonanKMcGovern commented 1 month ago

Might be easier to train a medusa head?

On Wed, May 22, 2024 at 11:01 PM cduk @.***> wrote:

Given that we have only Llama 3 70B and 8B, it would be useful to have a Tiny Llama based on the Llama 3 tokenizer so that we can use it as a drafting model for speculative decoding.

Are there plans to create a Llama 3 version?

— Reply to this email directly, view it on GitHub https://github.com/jzhang38/TinyLlama/issues/186, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASVG6CXCKDERGD5SJZALQQLZDUISXAVCNFSM6AAAAABIEO26WKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGMYTCNJWHE3TONY . You are receiving this because you are subscribed to this thread.Message ID: @.***>