Closed mashdragon closed 1 year ago
I am currently working on a context-based short term memory to be used with llama/gptneox based models. Its pretty much done so I should have a test version pushed in less than 12 hours. Testing it with the Open Assistant models seems to be a little tricky as they went out of their way to train the models to think that they specifically do not have any sort of short term memory, only the information that is available to them during training. So the model will tell me it does not have a memory and then proceed to use the context memory properly if I ask it to take a guess about something I said previously.
The way that I have it implemented is it will keep adding queries and responses to the context up to the context length (2048 or 4096 usually depending on which model is used). Once the context is full, it will purge the oldest query or response to make room for new next tokens as needed, but I have found that it takes a pretty decent amount of normal conversation before the context actually gets full and needs to be purged.
main-oasst has been updated to have basic context memory, running the chat scripts for the Open Assistant models should now work just fine in terms of short term memory.
I am not sure if this is how it is supposed to work, but no context is carried over to subsequent messages. For example, when running
./scripts/chat-pythia-12B.sh
.You can do the context yourself by inserting the
<|endoftext|><|assistant|>
tokens. I don't think they are true tokens in fact if you do this, so the results from the assistant can sometimes reproduce the literal<|endoftext|>
and<|assistant|>
. But I think the goal is to have a coherent dialogue automatically?For example: