search

aws-samples / aws-genai-llm-chatbot

A modular and comprehensive solution to deploy a Multi-LLM and Multi-RAG powered chatbot (Amazon Bedrock, Anthropic, HuggingFace, OpenAI, Meta, AI21, Cohere, Mistral) using AWS CDK on AWS
https://aws-samples.github.io/aws-genai-llm-chatbot/
MIT No Attribution
1.07k stars 321 forks source link

Enabling RAG makes it relatively easy to reach 400KB limit for a dynamodb item on sessions table #580

Open gbone-restore opened 2 weeks ago

gbone-restore commented 2 weeks ago

Amazon DynamoDB limits the size of each item in a table to 400 KB. It's feasible that this limit could be reached on the sessions table because of all the extra metadata stored in the item. Especially when running with RAG configured. The session table stores chat history in a single item attribute.

To work around this limitation, we can store chat history as individual items in the sessions table. I can submit a PR for this change if there is an interest in this issue.

charles-marion commented 6 days ago

Hi @gbone-restore ,

I agree it is an issue but changing the table structure might be a significant change.

A shorter/smaller change could be to reduce the metadata stored and limit the number of message in the sessions.

gbone-restore commented 5 days ago

Here's the data model that I used to test out this change.

Sessions

The partition key is now users and the sort key is sessions. And there are two types of items to make selecting session history and dealing with chat history a little easier. I can submit a PR with a working example. The bulk of the changes are in lib/shared/layers/python-sdk/python/genai_core/langchain/chat_message_history.py and lib/shared/layers/python-sdk/python/genai_core/sessions.py.

charles-marion commented 5 days ago

The partition key is now users and the sort key is sessions.

I think it's a good solution. My concern is it would invalidate existing sessions

I can submit a PR with a working example

If you already have a prototype available it would be welcome.

Note I intend to discuss this topic with the maintainers.