Hung at Waiting Sync [BUG]

aws-samples / bedrock-claude-chat

AWS-native chatbot using Bedrock + Claude (+Mistral)

MIT No Attribution

708 stars 250 forks source link

Hung at Waiting Sync [BUG] #190

Closed coderrrr closed 3 months ago

coderrrr commented 3 months ago

After created a Bot with a PDF knowledge file, it hung at indexing.

NOT Ready This bot has not completed the knowledge synchronization, so the knowledge before the update is used. Screenshot 2024-03-26 at 13 27 33

yianhan commented 3 months ago

I tried couple of times, but not sure whether it's the reason. But when I use more than 8000 characters prompt, the syncing stage will stay in waiting and not move on.

statefb commented 3 months ago

@coderrrr @yianhan Could you try followings to analyze the root cause?

There is a possibility of failure in Event Bridge Pipes, so first check the Pipes logs
If it fails in ECS Run Task, check the details in Cloud Trail > Event History

If succeeded to run ECS task, check:

From the bot details page, use browser developer tools or similar to check the syncLastExecId
Access container execution log via CloudWatch Logs > BedrockChatStack-EmbeddingTaskLogGroupXXXXX > embed-task/Container/syncLastExecId
See what's happening

JoshuaToth commented 3 months ago

Can confirm I also get waiting for sync issue when using a long prompt.

Since the eventbridge pipe loads the entire ddb record into the command it caps out the limit:

EB Pipe error: "Container Overrides length must be at most 8192 (Service: Ecs, Status Code: 400, Request ID: 72b3b7d4-00ef-412d-ac8c-20d8ef8b5833)"

Edit: This will also become an issue when too many documents are added.

statefb commented 3 months ago

@JoshuaToth Thank you for the information. Thanks to you, I was able to identify the cause of the bug. Just for reference, could you please tell me the approximate number of files at the time the error occurred?

JoshuaToth commented 3 months ago

Happy to help @statefb I'm not sure on the exact number, but since the entire DDB record is used as the ECS task override there will be a few factors including filename length, existing prompt length.

But in short. Somewhere between 50 and 300 is what I've tried. I realized it wasn't syncing with the 300 files I initially ingested so I consolidated them down into 50.

JoshuaToth commented 3 months ago

https://github.com/aws-samples/bedrock-claude-chat/pull/233

I had to fix this separately so I've fixed it here as well.