search

Stability-AI / StableLM

StableLM: Stability AI Language Models
Apache License 2.0
15.85k stars 1.04k forks source link

How to expand the sequence length of llama? #79

Closed binganao closed 1 year ago

binganao commented 1 year ago

Great job! Here is a question, how did the team extend the sequence length of llama from 2048 to 4096?

mcmonkey4eva commented 1 year ago

They just trained in on 4096 tokens lol. It's a variable that can be selected while training, wasn't any magic.