RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.
Amazing work! Thank you for sharing the code and weights.
Assuming that this is based or influenced by the Recurrent Memory Transformer paper, I would have assumed an almost infinite context length.
However, when training the model there appears to be a hard limit on the value of ctx_len. I'm currently fine-tuning the pile model question answering via with document context, but it seems to have difficulty learning the contextual relationship between the answers and context in the training data.
Any ideas on how to overcome this?
This is the format of my training data:
<|START_CONTEXT|>
<|START_DOCUMENT|>
Doc 1
<|END_DOCUMENT|>
<|END_CONTEXT|>
<|START_QUESTION|>
How do I.. ?
<|END_QUESTION|>
<|START_ANSWER|>
To do X, you should first..
<|END_ANSWER|>
Amazing work! Thank you for sharing the code and weights.
Assuming that this is based or influenced by the Recurrent Memory Transformer paper, I would have assumed an almost infinite context length.
However, when training the model there appears to be a hard limit on the value of ctx_len. I'm currently fine-tuning the pile model question answering via with document context, but it seems to have difficulty learning the contextual relationship between the answers and context in the training data.
Any ideas on how to overcome this?
This is the format of my training data: