search

aws / aws-sdk-js

AWS SDK for JavaScript in the browser and Node.js
https://aws.amazon.com/developer/language/javascript/
Apache License 2.0
7.6k stars 1.55k forks source link

ReferenceError in SageMaker invokeEndpointWithResponseStream #4580

Closed cshen-dev closed 5 months ago

cshen-dev commented 9 months ago

Describe the bug

I created an AWS SageMaker endpoint with Mistral Jumpstarter (jumpstart-dft-hf-llm-mistral-7b-ins-20240130).

sagemakerRuntime.invokeEndpoint worked as a charm however when I started exploring its stream capability, I met the issue below.

Error [ReferenceError]: payload is not defined
    at Request.extractData (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/protocol/rest_json.js:78:17)
    at Request.callListeners (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/Users/c.shen/Workspaces/tmp/sagemaker/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'ReferenceError',
  time: 2024-02-02T00:39:53.693Z,
  statusCode: 200,
  retryable: false,
  retryDelay: 62.00239553897755
}

Expected Behavior

[ReferenceError]: payload is not defined

^ This error should not occur which I guess is caused by mis-mapped data structure between SageMaker endpoint's Response and the sdk client's Request.

Current Behavior

It's throwing error and block the downstream processing

Reproduction Steps

  1. Set up a SageMaker endpoint

  2. Use the code block below (which is actually from AWS official doc)

    // In Node.js, events are streamed and can be read as they arrive.
sagemakerruntime.invokeEndpointWithResponseStream({/** params **/}, function(err, data) {
if (err) {
// handle error
return console.error(err);
}

var eventStream = data.Body;

eventStream.on('data', function(event) {
// Check the top-level field to determine which event this is.
if (event.PayloadPart) {
  // handle PayloadPart event
} else if (event.ModelStreamError) {
  // handle ModelStreamError event
} else if (event.InternalStreamFailure) {
  // handle InternalStreamFailure event
}
});
eventStream.on('error', function(err) { /** Handle error events **/});
eventStream.on('end', function() { /** Finished reading all events **/});
});

Possible Solution

No response

Additional Information/Context

No response

SDK version used

2.1549.0

Environment details (OS name and version, etc.)

macos

aBurmeseDev commented 9 months ago

Hi @cshen-dev - thanks for reaching out.

I’m trying to understand if the error you’re getting is actually related to SDK because it seems like a generic JavaScript error. Can you share your code where you’re making client request with SDK? Are you defining the “payload” somewhere?

I’d be happy to further investigate once you share more information requested. Best, John

github-actions[bot] commented 9 months ago

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

inf3rnus commented 8 months ago

FYI @cshen-dev I've created a fix for this with https://github.com/aws/aws-sdk-js/pull/4607, pending Amazon's evaluation of the change.

In the meantime you can pull my fork @ https://github.com/inf3rnus/aws-sdk-js

rjvgupta commented 8 months ago

I had a similar error on this file when using lambda's invokeWithResponseStream. Issue was resolved when making changes as per my pull request (https://github.com/aws/aws-sdk-js/pull/4609). This is clearly a code issue as there is no variable payload referenced in the file but it is an attribute of the rules object and is being used as such in subsequent conditional blocks (e.g. lines 85,87,89). 

2024-03-10T19:54:58.963Z    2a359c5e-a419-406f-9092-84b26d378eeb    ERROR   Error invoking Lambda function: Error [ReferenceError]: payload is not defined
    at Request.extractData (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:78:17)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'ReferenceError',
  time: 2024-03-10T19:54:58.963Z,
  statusCode: 200,
  retryable: false,
  retryDelay: 64.36430770472576
}
rjvgupta commented 8 months ago

Here's a simple repro using 2 lambda functions, with error above, using published SDK. Same code is successful when using lambda layer that has code changes from my pull request.

lambda1:

var AWS = require("aws-sdk")
AWS.config.update({region: 'us-east-1'}); 
var lambda = new AWS.Lambda();

exports.handler = async (event) => {
  var params = {
    FunctionName: 'lambda2', 
    InvocationType: 'RequestResponse',
    LogType: 'Tail',
    Payload: JSON.stringify({message: 'Do Nothing.'})
  };

  const result = await lambda.invokeWithResponseStream(params).promise();
  const events = result.EventStream;
  for await (const event of events) {
    if (event.PayloadChunk) {
      console.log(event.PayloadChunk);
    } 
  }

  const response = {
    statusCode: 200,
    body: JSON.stringify('Hello from Lambda!'),
  };
  return response;
};

lambda2:

exports.handler = awslambda.streamifyResponse(async (event, responseStream, _context) => {
  responseStream.write('message1');
  responseStream.write('message2');
  responseStream.write('message3');
  responseStream.end();
});
aBurmeseDev commented 8 months ago

Hi everyone on the thread, appreciate all your effort into this behavior.

I was able to reproduce this and will be further looking into it with the team. Will post once we have updates.

rjvgupta commented 7 months ago

@siddsriv / @aBurmeseDev - any update?

inf3rnus commented 7 months ago

@rjvgupta if you're only using this part of the AWS sdk, v3 of the sdk solves this problem, and consuming the output of it is a lot easier.

e.g.

async function callSageMakerEndpoint({ endpointName, body }) {
  const params = {
    EndpointName: endpointName,
    ContentType: "application/json",
    Accept: body.stream
      ? "text/event-stream; charset=utf-8"
      : "application/json",
    Body: JSON.stringify(body)
  };

  if (!body.stream) {
    const { Body } = await sageMakerRuntimeClient.send(
      new InvokeEndpointCommand(params)
    );

    const buffer = Buffer.from(Body);

    const jsonResults = JSON.parse(buffer);

    return jsonResults;
  }

  // streaming output
  const { Body } = await sageMakerRuntimeClient.send(
    new InvokeEndpointWithResponseStreamCommand(params)
  );

  const readStream = Readable.from(Body);

  readStream.on("error", error => {
    console.error(`Sagemaker endpoint error: ${endpointName}`, error);
    readStream.emit("end");
  });

  const transformStream = createTransform();

  readStream.pipe(transformStream);

  return transformStream;
}

function createTransform() {
  const transformStream = new Transform({
    readableObjectMode: false, // Output should be Buffers or strings
    writableObjectMode: true, // Input will be objects,
    transform({ PayloadPart: { Bytes } = {} } = {}, encoding, callback) {
      const text = Buffer.from(Bytes)?.toString() ?? "";
      // Push the transformed data to the next handler in the stream
      this.push(text);
      callback();
    }
  });

  return transformStream;
}
rjvgupta commented 7 months ago

Hey Aaron, thanks for following up on this. I was able to try w/ v3 and can confirm, that it worked without needing to override the SDK with my changes. This is a reasonable workaround, but will take some work to modify the v2 code to v3 and think we should deploy the fix requested.

Thank You, Rajiv Gupta

Manager, Analytics Specialist Solutions Architects p: 949-296-5163 | e: @.**@.> Thoughts on our interaction? Provide feedback herehttps://feedback.aws.amazon.com/?ea=rjvgupta&fn=Rajiv&ln=Gupta.

From: Aaron V @.> Reply-To: aws/aws-sdk-js @.> Date: Thursday, March 21, 2024 at 7:43 PM To: aws/aws-sdk-js @.> Cc: "Gupta, Rajiv" @.>, Mention @.***> Subject: Re: [aws/aws-sdk-js] ReferenceError in SageMaker invokeEndpointWithResponseStream (Issue #4580)

@rjvguptahttps://github.com/rjvgupta if you're only using this part of the AWS sdk, v3 of the sdk solves this problem, and consuming the output of it is a lot easier.

e.g.

const { Body } = await sageMakerRuntimeClient.send(

new InvokeEndpointWithResponseStreamCommand(params)

);

const readStream = Readable.from(Body);

readStream.on("error", error => {

console.error(`Sagemaker endpoint error: ${endpointName}`, error);

readStream.emit("end");

});

const transformStream = createTransform();

readStream.pipe(transformStream);

return transformStream;

}

function createTransform() {

const transformStream = new Transform({

readableObjectMode: false, // Output should be Buffers or strings

writableObjectMode: true, // Input will be objects,

transform({ PayloadPart: { Bytes } = {} } = {}, encoding, callback) {

  const text = Buffer.from(Bytes)?.toString() ?? "";

  // Push the transformed data to the next handler in the stream

  this.push(text);

  callback();

}

});

return transformStream;

}

— Reply to this email directly, view it on GitHubhttps://github.com/aws/aws-sdk-js/issues/4580#issuecomment-2014217210, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALLHDRWXZEBLAR2POMIBT6TYZOLC5AVCNFSM6AAAAABCV2Z3CSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJUGIYTOMRRGA. You are receiving this because you were mentioned.Message ID: @.***>

inf3rnus commented 7 months ago

@rjvgupta np, and yeah, I agree. Ended up having to migrate to v3 myself because I ran into another problem where the deleteEndpoint method wouldn't actually delete endpoints, which means there's almost certainly other problems lurking.