search

Stability-AI / StableLM

StableLM: Stability AI Language Models
Apache License 2.0
15.83k stars 1.04k forks source link

More than 4096 context length? #22

Open StoyanStAtanasov opened 1 year ago

StoyanStAtanasov commented 1 year ago

Is it possible to have larger context as this allows to do more complicated things with smaller models? A lot of the negatives of a smaller model can be rectified by pushing more data into the context. For example: Help pages, datasheets, examples, thinking rules, longer conversations trying to fix an issue, etc.

Please excuse me if this is the wrong place to ask this question, but very rarely the context is discussed. Thanks in advance.

jon-tow commented 1 year ago

Sure; You just need to fine-tune to a longer context :)

StoyanStAtanasov commented 1 year ago

@jon-tow You are joking right?

jon-tow commented 1 year ago

Nope; see https://github.com/kyleliang919/Long-context-transformers

NPap0 commented 1 year ago
  1. Training code has to change
  2. Data that you fine tune the model with after training has to change

So no, nothing can be done user-side to change attention span. (But maybe you can summarize blocks of text with the model so you can then feed the already summarized text as a whole to do your thing)

mallorbc commented 1 year ago

Pretty sure the answer is no due to how positional encoding is done.

jon-tow commented 1 year ago

@mallorbc The model uses RoPE - a relative attention mechanism. See https://blog.eleuther.ai/rotary-embeddings/

mallorbc commented 1 year ago

@jon-tow So models like GPTJ can be finetuned and generate more than their sequence length? Whenever I try to generate sequences for GPTJ I have issues. Maybe that is something else unrelated.

Ph0rk0z commented 1 year ago

A lora will not work for this? Would you have to retrain the model from scratch on long context as you have done?