Document design of dial assistant context management - Githubissues

epam / ai-dial-assistant

Model agnostic assistant/addon implementation for AI DIAL. It allows to use self-hosted OpenAI plugins as DIAL addons

https://epam-rail.com

Apache License 2.0

9 stars 2 forks source link

Document design of dial assistant context management #22

Open avavilau opened 10 months ago

avavilau commented 10 months ago

Target state (design)
API Spec for Assistant limits
Smaller task list with ETAs

Oleksii-Klimov commented 9 months ago

Target state:

Assistant provides prompt truncation/tokenization endpoints to ensure that completion requests can fit: https://github.com/epam/ai-dial-assistant/issues/38
Completion request provides an extra parameter to control how many tokens can be dedicated for model dialogue with addons: https://github.com/epam/ai-dial-sdk/issues/51 https://github.com/epam/ai-dial-assistant/issues/40

Usage scenario:

Tokenize the completion request using the tokenization endpoint.
If the request fits then go to step #⁠4.
If the request does not fit, reduce the number of messages or message sizes then repeat step #⁠1.
Send the request to the chat completion endpoint. The truncation endpoint aids in managing history size by allowing for the discarding of old messages.