Enquiry on the use of memory

[enigmablue]

Hi there i'm trying to find ways to maintain conversation history and also managing context. I see there is some method to handle chatmemory but i'm not sure what is the context and purpose. Could you help elaborate?

More specifically i'm trying to find out how conversations / prompts can be linked

[mochi-neko]

@enigmablue

Thank you for your enquiry.

Conversation history is a list of Message according to the API reference.

Message has the "role" label.

• "system" is an order to LLM.
• "user" is an user input message.
• "assistant" is a LLM output message.

Therefore, "system" is prompt and others are conversation history between human and AI.

You can specify message list by IChatMemory to set constructor of ChatCompletionAPIConnection.

ChatCompletionAPIConnection automatically records requests and responses to memory.

Sample implementations of IChatMemory are followings:

• SimpleChatMemory
  • Simple list of messages.
• FiniteQueueChatMemory
  • Finite count queue of messages.
• FiniteTokenLengthQueueChatMemory
  • Finite total token length list of messages.
• LongTermChatMemory
  • Summarize old messages by ChatGPT.

Also you can customize memory logic by implementing IChatMemory.

I hope this will help.

[enigmablue]

Thank you but may i ask a stupid question, what is the intention and benefit of setting up memory?

[mochi-neko]

I intend to be able to extend memory implementation because needed memory specification depends on use cases.

As an example, ChatGPT API has the limit of token length at one request.

• gpt3.5-turbo: 4096 tokens
  • https://platform.openai.com/docs/models/gpt-3-5
• gpt4: 8192 or 32768 tokens
  • https://platform.openai.com/docs/models/gpt-4

Token length for these models are calculated by tiktoken.

Then sometimes it is better to take care of token length in memory e.g. FiniteQueueChatMemory and FiniteTokenLengthQueueChatMemory.

See also LangChain.Memory.

In other cases, if you want to utilize the context of the conversation to request a task to ChatGPT but do not need to record a response, customizing the memory has benefit.

[enigmablue]

thank you for your tips!

[enigmablue]

May i ask a bit more about this, do you know a way to maximize the token handling so that costs are managed?

For example in a regular ChatGPT conversation, it knows the history of the conversations and it appears to include it in the prompt without us having to manage it. I believe they are handling this as a part of the reinforcement learning mechanism in the web app.

Do you have any idea how that is done or it can be implemented here or in Unity?

Maybe i dont fully understand your statement: "if you want to utilize the context of the conversation to request a task to ChatGPT but do not need to record a response, customizing the memory has benefit."

Can benefits of langchain also be implemented here?

Thank you so much