search

FlowiseAI / Flowise

Drag & drop UI to build your customized LLM flow
https://flowiseai.com
Apache License 2.0
29.91k stars 15.44k forks source link

[BUG] AWSBedrockEmbedding throttling issue #2907

Closed RPKosec closed 1 month ago

RPKosec commented 1 month ago

When using AWS Bedrock Embeddings on larger number of documents (larger Confluence space load for instance), AWS will return throttling error, which in turn will fail the upsert to vector store.

Steps to reproduce the behavior:

  1. Use Confluence document loader (should work with other too) and set it to data source with larger set of documents
  2. Use AWS Bedrock Embeddings
  3. Use In memory vector store (same behaviour was observed with Milvus and Postgres too)
  4. Click Upsert
  5. Error occurs

Expected behavior When upserting larger number of documents using AWS Bedrock Embeddings, no error should be raised. Throttling error should be mitigated using batches (with batch size settable by user). If throttling happens due to to many requests, retry with timeout between retries (increasing with iterations, which should be limited and limit set by user).

Flow Screenshots Basic example of flow, producing an error: AWSBedrockEmbeddingsBug

Setup

Additional context Error trace of the exception:

2024-07-29 12:41:50 [INFO]: ⬆️ POST /api/v1/vector/internal-upsert/cf3dd83a-588e-4856-a53d-9865619492b6
@smithy/node-http-handler:WARN socket usage at capacity=50 and 5273 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
@smithy/node-http-handler:WARN socket usage at capacity=50 and 4261 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
2024-07-29 12:56:13 [ERROR]: ThrottlingException: Too many requests, please wait before trying again. You have sent too many requests.  Wait before trying again.
Error: ThrottlingException: Too many requests, please wait before trying again. You have sent too many requests.  Wait before trying again.
    at InMemoryVectorStore_VectorStores.upsert (/home/node/Flowise/packages/components/dist/nodes/vectorstores/InMemory/InMemoryVectorStore.js:26:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async buildFlow (/home/node/Flowise/packages/server/dist/utils/index.js:408:37)
    at async upsertVector (/home/node/Flowise/packages/server/dist/utils/upsertVector.js:115:32)
    at async Object.upsertVectorMiddleware (/home/node/Flowise/packages/server/dist/services/vectors/index.js:9:16)
    at async createInternalUpsert (/home/node/Flowise/packages/server/dist/controllers/vectors/index.js:28:29)
2024-07-29 12:56:13 [ERROR]: [server]: Error: Error: ThrottlingException: Too many requests, please wait before trying again. You have sent too many requests.  Wait before trying again.
Error: Error: ThrottlingException: Too many requests, please wait before trying again. You have sent too many requests.  Wait before trying again.
    at buildFlow (/home/node/Flowise/packages/server/dist/utils/index.js:483:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async upsertVector (/home/node/Flowise/packages/server/dist/utils/upsertVector.js:115:32)
    at async Object.upsertVectorMiddleware (/home/node/Flowise/packages/server/dist/services/vectors/index.js:9:16)
    at async createInternalUpsert (/home/node/Flowise/packages/server/dist/controllers/vectors/index.js:28:29)
@smithy/node-http-handler:WARN socket usage at capacity=50 and 2677 additional requests are enqueued. See https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html or increase socketAcquisitionWarningTimeout=(millis) in the NodeHttpHandler config.
HenryHengZJ commented 1 month ago

how do we send them in batches?

RPKosec commented 1 month ago

how do we send them in batches?

I added draft PR, how I envisioned the solution. This way, no socket warnings are triggered and throttling error is omitted, but it still has way out in case of to many exceptions in a row or if any other exception is thrown to avoid infinite loop.