lindebuyer
commented
1 year ago Your theory is correct. Every request is a new chat. At least is the default behavior documented on official website. Model analyze the previous chats send along with the new prompt to generate next answer.
And your observation is also correct some time the request chat go over the max token limit. Model still remember some of the instructions in the beginning. Which is not sent with latest request.
And my observation is model tend to have a default behavior that extract important info in the generated response. To keep it in included next request. For example you tell the model John is a male. In the next maybe 10 or 20 model might mention this information and refer John with Him. So on the maybe 99rounds of chat request. Even it dosen't include the first message that declare John is a male. Model still know he is.
I believe this this is only a fraction part of how model try to remember beyond token limits.
Other has theory that the model remember every user input and it's answers. which is how they use to check if some student write essay by chatGPT. The token limit is a limit has higher priority for higher relevancy for request generation.
I forgot how openAI’s privacy policy for api usage. But surely there isn't a parameter to forbid data be use to train their model. Only on their official web &ios app have this toggle. And model api has a optional user ID they said is use for moderation purposes. But might also be a part of their short term memory logic. (betterchatGPT default request don't include this parameter in the code) Hope this help
When using the GPT 4 8k model, what method is the UI using https://bettergpt.chat/ to have a conversational message. It seems to allow me to go over the max token limit while still remembering context from the very beginning of the conversation.