search

test-time-training / ttt-lm-pytorch

Official PyTorch implementation of Learning to (Learn at Test Time): RNNs with Expressive Hidden States
MIT License
1.01k stars 56 forks source link

The definition of Context Length #27

Closed Z-Z188 closed 1 month ago

Z-Z188 commented 1 month ago

image Hello author, thank you for your work. So how is this context length defined? Is one word corresponding to one token, and the corresponding length is 1? That's a lot to say for a 32k sentence. Looking forward to the author's answer, thank you!

karan-dalal commented 1 month ago

The context length is # of tokens. A token can be 1 word or a section of a word, depending on its type.

Z-Z188 commented 1 month ago

Hello author, can you give a specific example? I still don't understand. Look forward to your reply, thank you!