search

aws / aws-sdk-js-v3

Modularized AWS SDK for JavaScript.
Apache License 2.0
3.02k stars 564 forks source link

client-bedrock-runtime: Same request via InvokeModelWithResponseStreamCommand throws different exceptions depending on the model used #6266

Open modcab opened 1 month ago

modcab commented 1 month ago

Checkboxes for prior research

Describe the bug

When sending a request that exceeds the allowed token count:

SDK version number

@aws-sdk/client-bedrock-runtime@3.602.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.11.0

Reproduction Steps

Sending a request that exceeds the allowed token count and examine the exception thrown.

Observed Behavior

Expected Behavior

The same exception type for both models, with information that allows us to respond correctly.

Possible Solution

No response

Additional Information/Context

No response

aBurmeseDev commented 1 month ago

Hi @modcab - thanks for reaching out.

To understand the issue better, can you please share your code snippet without any sensitive information? Are you seeing both error in the same request or is it thrown individually in different request? Note that these errors come from service side which means we'll have to reach out to Bedrock team for their input once you've shared the code.

modcab commented 1 month ago

Hi @aBurmeseDev , thanks for your quick response. They're individual, different requests. The actual code is of course much longer and scattered over different functions and files, so let me know if this helps.

First one: 

//
// const params = {
//   body: Uint8Array(4485220) [
//     123,  34, 115, 121, 115, 116, 101, 109,  34,  58,  34,  34,
//       44,  34, 109, 101, 115, 115,  97, 103, 101, 115,  34,  58,
//       91, 123,  34, 114, 111, 108, 101,  34,  58,  34, 117, 115,
//       101, 114,  34,  44,  34,  99, 111, 110, 116, 101, 110, 116,
//       34,  58,  91, 123,  34, 116, 121, 112, 101,  34,  58,  34,
//       116, 101, 120, 116,  34,  44,  34, 116, 101, 120, 116,  34,
//       58,  34, 111, 107,  46,  32,  67, 111, 110, 116, 101, 110,
//       116, 115,  32, 111, 102,  32, 102, 105, 108, 101,  32, 111,
//       122,  95,  49,  46,
// ... 4485120 more items
// ],
//   modelId: 'anthropic.claude-3-haiku-20240307-v1:0',
//     contentType: 'application/json',
//   accept: 'application/json'
// }

const command = new InvokeModelWithResponseStreamCommand(params);
const data = await client.send(command);
const stream = data.body;
// ...
for await (const encodedChunk of stream) {

There, it breaks, and I can see this error in the logs: 

{
    "timestamp": "---",
    "level": "ERROR",
    "requestId": "---",
    "message": {
        "errorType": "ValidationException",
        "errorMessage": "Input is too long for requested model.",
        "stackTrace": [
            "ValidationException: Input is too long for requested model.",
            "    at ----",
            "    at ----",
            "    at ----",
        ],
        "name": "ValidationException",
        "$fault": "client",
        "$metadata": {}
    }
}

Which is great. It has a descriptive message, and I can present it in a helpful way.

But then, if the code is the same, but the modelId is 'anthropic.claude-3-opus-20240229-v1:0' or anthropic.claude-3-5-sonnet-20240620-v1:0' then we get this one:

{
    "timestamp": "-----",
    "level": "ERROR",
    "requestId": "-----",
    "message": "An unexpected error occurred in the chat stream: ModelStreamErrorException: The system encountered an unexpected error during processing. Try your request again.\n    at ---",
    "errorType": "ModelStreamErrorException",
    "errorMessage": "The system encountered an unexpected error during processing. Try your request again.",
    "stackTrace": [
        "ModelStreamErrorException: The system encountered an unexpected error during processing. Try your request again.",
        "    at ----",
        "    at  ----",
        "    at ----",
    ]
}