Open rstrahan opened 3 months ago
Reduced default value of LLM_CHAT_HISTORY_MAX_MESSAGES to 10 as an interim measure
One thing we might could try is producing a summary of the transcript so far and sending that, seems potentially more useful than a set number of messages. But that runs into response time issues as well as we'd be making more calls to the LLMs.
Describe the bug
Sometime the meeting assistant responds: "Default message - you'll see this only if the lambda hook failed"
On investigation, the root cause is reflected by this error in the CW Logs for function
QNA-lma6-BedrockKB-LambdaHook
:which is caused by the prompt size being too long for the Bedrock API call.. due to too much transcript text being included in the prompt.
The transcript by default is truncated to last 12 turns - Qnabot setting
LLM_CHAT_HISTORY_MAX_MESSAGES
- setting this to a lower value resolves the issue.. but we should defend against it happening at all by checking and truncating prompt length regardless of MAX_MESSAGES (since individual messages can be long)To Reproduce Steps to reproduce the behavior:
Generate a transcript with long turns.. eg where one person is presenting..
After several long turns, invoke the meeting assistant via either 'OK Assistant' voice prompt or bot button..
It will respond with "Default message - you'll see this only if the lambda hook failed"
Expected behavior
It should not fail.
Screenshots
Additional context Add any other context about the problem here.