[Bug] Timeout Occurs in Embedding When Adding a Large Number of URLs to Knowledge

[edamame8888]

Describe the bug

When creating a bot, inputting about 100 URLs (not YouTube URLs) results in the following error in BedrockChatStack-EmbeddingTask:

ERROR:embedding.loaders.playwright:Error fetching or processing https://***, exception: Timeout 30000ms exceeded.

version: v1

To Reproduce

• Prerequisites
  • User must be a member of the Admin group(cognito).
• Steps to Reproduce
  1. Click "Bot Console" from the menu.
  2. Click the "Create New Bot" button.
  3. Enter "BugBot" in the Name field, and input about 100 URLs in the URL input form.
  4. Click the "Create" button.
• How to Verify the Issue
  1. Navigate to CloudWatch BedrockChatStack-EmbeddingTaskLogGroup and confirm that the error "ERROR:embedding.loaders.playwright:Error fetching or processing https://, exception: Timeout 30000ms exceeded." has occurred.

Screenshots

Additional context

Workaround Tried

I confirmed an improvement by calling page.close() within a loop of playwright on my end.

https://github.com/aws-samples/bedrock-claude-chat/blob/v1/backend/embedding/loaders/playwright.py#L150-L170

            for url in self.urls:
                try:
                    page = browser.new_page()
                    response = page.goto(url)
                    if response is None:
                        raise ValueError(
                            f"page.goto() returned None for url {url}")

                    text = self.evaluator.evaluate(page, browser, response)
                    metadata = {"source": url}
                    docs.append(Document(page_content=text, metadata=metadata))
+                    page.close()
+                    logger.info(f"Loaded {url} and page closed.")
                except Exception as e:
                    if self.continue_on_failure:
                        logger.error(
                            f"Error fetching or processing {url}, exception: {e}"
                        )
                    else:
                        raise e
            browser.close()
        return docs

Result

[statefb]

Thank you for suggesting workaround! Could you explain or guess why your solution resolve the timeout issue?

[edamame8888]

OK! I'll respond after work, so please wait a bit.

[edamame8888]

Performance Before & After

Could you explain or guess why your solution resolve the timeout issue?

The memory being maxed out at 100% is the cause.

Before the improvement, the memory performance in Cloud Watch Insight was maxed out and the content retrieval never finished. After the improvement, memory performance improved and we confirmed that the embedding was completed in a short period of time.