Support for testing stream-based functions

[jzyeezy]

First off, I'd like to say that this project is awesome and very well documented. The provided terraform config was really helpful while I was troubleshooting.

My team owns a lambda which returns a response stream. Since the stream pattern allows for responses >6MB payload limitation, the execution of the Step Function's executor will fail with:

Invocation error (running in series): {\n  \"errorMessage\": \"Response payload size exceeded maximum allowed payload size (6291556 bytes).\"

The implementation of the executor lambda uses the V2 aws-sdk, which doesn't include the InvokeWithResponseStream method. I've created a branch to update the executor lambda to be implemented with the V3 aws-sdk and could successfully get past this hurdle using this as the utils invokeLambda method:

module.exports.invokeLambda = async (lambdaARN, alias, payload) => {
    const params = {
        FunctionName: lambdaARN,
        Qualifier: alias,
        Payload: payload,
        LogType: 'Tail', // will return logs
    };
    const lambda = utils.lambdaClientFromARN(lambdaARN);
    const command = new InvokeWithResponseStreamCommand(params);
    const response = await lambda.send(command);
    for await (const item of response.EventStream) {
        if (typeof item.PayloadChunk !== "undefined") {
            console.log(Buffer.from(item.PayloadChunk.Payload.buffer).toString());
        }
    }
    return response;
};

where the response has the shape of:

{
 '$metadata': {
    httpStatusCode: 200,
    requestId: 'REDACTED',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  ExecutedVersion: '323',
  ResponseStreamContentType: 'application/vnd.amazon.eventstream',
  EventStream: SmithyMessageDecoderStream {
    options: {
      messageStream: [MessageDecoderStream],
      deserializer: [AsyncFunction (anonymous)]
    }
  },
  StatusCode: 200
}

and the payload

:event-typePayloadChunk
:content-typeapplication/octet-stream
:message-typeevent
<my streamed chunk>

But it unfortunately appears the LogType: 'Tail' parameter only works with synchronously invoked functions (documented here). This means the logs containing the "Billing Duration" stats aren't returned and the analyzer lambda will fail because there's nothing to analyze.

I was able to work around this issue by updating my lambda to not return a payload, since I've rationalized successfully getting a real response by the power-tuner's executor lambda isn't all that important in measuring cost/compute. However, I thought I'd bring up this use-case and share what I was able to get working in case this comes up for anyone else.

Thanks for putting this together!

[alexcasalboni]

Hi @jzyeezy 👋 thanks for reaching out and sharing your findings!

This is very useful, indeed.

I agree with you that the stream response isn't that useful for the executor to power-tune your function (unless that's all your function is doing). Did you manage to obtain useful power-tuning results even without returning a response?

I haven't converted the whole project to SDK v3 yet, but it'd be a good idea in general :) There's just a lot of tests that will need to be reshaped to work with v3, so I haven't found the time yet.

Any chance you could share your fork/branch with me so I can have a look?

[jzyeezy]

Hey @alexcasalboni, sure happy to help.

Did you manage to obtain useful power-tuning results even without returning a response?

Yes! This tool generated some compelling cost signals which helps us narrow the scope on compute options to measure in real performance testing.

Here are my changes. A couple things to note:

• I only made changes to the executor lambda
• I tried to upgrade the lambda runtime to nodejs 18.x, since that comes with built-in support of the v3 SDK. However, the version of the aws-sdk that comes bundled in Lambda's 18.x nodejs unfortunately didn't contain the InvokeWithResponseStreamCommand. I opted to create my own lambda layer, which is why you see changes in the layer-sdk directory.