How to correctly return binary payload to AWS API Gateway from Lambda

[nardeas]

Hello! I am trying to use serverless-http to wrap a Koa image manipulation API and deploy it on AWS Lambda with APIGW. I ran into some problems returning a binary payload so I tried base64 encoding my body response and then do this in the handler:

const serverless = require('serverless-http');
const server = require('./server');

module.exports.handler = serverless(server, {
  request: function(request, event, context) {
    console.log(request);
  },
  response: function(response, event, context) {
    // Set isBase64Encoded true so API gateway knows the body needs to be decoded
    response.isBase64Encoded = true;
    console.log(response);
  }
});

I've setup CloudWatch logging and it correctly logs the response object having the isBase64Encoded field set. However when I invoke the function using awscli I get:

{
  "isBase64Encoded":false,
  "statusCode":200,
  "headers": {...},
  "body": "..."
}

Any ideas why my response modification is not present in the invocation result?

[dougmoscrop]

Yeah, sorry there's too many "responses" :) it's an ambiguous term.

The request/response hooks manipulate the req/res as seen by the "fake HTTP" pipeline, not the "response object send back to Lambda".

You can, I hope, see sort of how it goes down here: https://github.com/dougmoscrop/serverless-http/blob/master/serverless-http.js#L42

Basically, the Lambda response is a bunch of properties that are picked/calculated off of the "fake HTTP" response. I'm not looking for isBase64Encoded, but rather, determining it based on properties of the response.

What you probably want to do is instead hook in to isBinary and just return true: https://github.com/dougmoscrop/serverless-http/blob/master/lib/is-binary.js

Alternatively, if you genuinely want to 'pipeline' AROUND serverless-http, you can do it like this, it's a bit verbose :

const handle = serverless(app);
module.exports.handler = function(event, context, callback) {
   handle(event, context, (err, res) => {
     if (err) {
       callback(err)
     } else {
       callback(null, Object.assign(res, { isBase64Encoded: true }))
     }
  })
})

[dougmoscrop]

But also if you set up your content types as binary, which you have to do in API gateway anyway, this library just works. You can't just return "isBase64Encoded" and APIg will be happy. You have to turn on content handling options and set MIME types (explicitly too - it doesn't work for "image/* unfortunately) -- these are all APIg limitations that are outside the scope of this library

[nardeas]

Thanks! I set the binary: [ 'image/*' ] option in the handler config and now APIg console tells me that isBase64Encoded: true is correctly returned.

However I still can't get APIg to decode the response for me 😄 I know this is out of scope of this library but I was wondering if you might knew what's wrong with my overall configuration:

• Koa route returns base64 encoded image in body along with appropriate content-type header
• serverless-http handler is configured with binary: [ 'image/*' ] to set the isBase64Encoded flag for the lambda response
• APIg is configured in AWS_PROXY mode to invoke the lambda function
• APIg rest api is configured with binary_media_types = [ 'image/png' ]
• APIg integration response is configured with content handling "CONVERT_TO_BINARY"

So now if I simply curl the endpoint it returns base64 encoded body. Even curl with Accept: image/png header didn't make any difference. If you could point out any obvious problems with this setup I would be eternally grateful, this is starting to be quite annoying... 😅

[nardeas]

Okay now I see:

https://github.com/dougmoscrop/serverless-http/blob/9b9190c976f4d63f0a6b98ed11d3dbd91c078fb5/lib/get-body.js#L12

it automatically does the base64 encode part for your body if the binary type is set. So I am essentially double encoding the body!

[nardeas]

After I removed my own ad-hoc encoding it works with the Accept header. So I still need to find a way to get around that limitation since I would like to use the endpoint for <img src="...">

[nardeas]

OK! I managed to solve all my issues. For anyone else having problems with this here's the recipe to successfully return binary payload from serverless Koa API using AWS lambda and API Gateway:

• Have your Koa route return the payload as binary and set appropriate Content-Type headers
• Wrap your KoaJS API with serverless-http with binary option for your content-type:

module.exports.handler = serverless(server, {
  request: function(request, event, context) {
    console.log(request);
  },
  response: function(response, event, context) {
    console.log(response);
  },
  // Your Content-Type is matched against this and base64 encoding is automatically 
  // done for your payload. This also sets isBase64Encoded true for the Lambda response
  // to API Gateway by this library
  binary: ['image/*']
});

• Setup APIg with binaryMediaTypes as */*
• Example APIg Terraform configuration:

resource "aws_api_gateway_rest_api" "ExampleAPI" {
  name = "Example API"
  description = "Example image manipulation REST API"
  # If you specify "image/jpeg" or similar here, APIg will require the
  # client to send "Accept" header...
  binary_media_types = [ "*/*" ]
}

resource "aws_api_gateway_resource" "ExampleProxy" {
  rest_api_id = "${aws_api_gateway_rest_api.ExampleAPI.id}"
  parent_id = "${aws_api_gateway_rest_api.ExampleAPI.root_resource_id}"
  path_part = "{proxy+}"
}

resource "aws_api_gateway_method" "ExampleProxy" {
  rest_api_id = "${aws_api_gateway_rest_api.ExampleAPI.id}"
  resource_id = "${aws_api_gateway_resource.ExampleProxy.id}"
  http_method = "ANY"
  authorization = "NONE"
}

resource "aws_api_gateway_method" "ExampleProxyRoot" {
  rest_api_id = "${aws_api_gateway_rest_api.ExampleAPI.id}"
  resource_id = "${aws_api_gateway_rest_api.ExampleAPI.root_resource_id}"
  http_method = "ANY"
  authorization = "NONE"
}

resource "aws_api_gateway_integration" "Example" {
  rest_api_id = "${aws_api_gateway_rest_api.ExampleAPI.id}"
  resource_id = "${aws_api_gateway_method.ExampleProxy.resource_id}"
  http_method = "${aws_api_gateway_method.ExampleProxy.http_method}"

  type = "AWS_PROXY"
  integration_http_method = "POST"
  uri = "${aws_lambda_function.Example.invoke_arn}"
}

resource "aws_api_gateway_integration" "ExampleRoot" {
  rest_api_id = "${aws_api_gateway_rest_api.ExampleAPI.id}"
  resource_id = "${aws_api_gateway_method.ExampleProxyRoot.resource_id}"
  http_method = "${aws_api_gateway_method.ExampleProxyRoot.http_method}"

  type = "AWS_PROXY"
  integration_http_method = "POST"
  uri = "${aws_lambda_function.Example.invoke_arn}"
}

# NOTE: Taint this to make changes to APIg
resource "aws_api_gateway_deployment" "ExampleAPI" {
  depends_on = [
    "aws_api_gateway_integration.Example",
    "aws_api_gateway_integration.ExampleRoot"
  ]
  rest_api_id = "${aws_api_gateway_rest_api.ExampleAPI.id}"
  stage_name = "dev"
}

resource "aws_lambda_function" "Example" {
  function_name = "Example"
  handler = "index.handler"
  runtime = "nodejs8.10"
  role = "${aws_iam_role.Lambda.arn}"

  s3_bucket = "${aws_s3_bucket.Lambdas.bucket}"
  s3_key = "example-1.0.0.zip"
}

resource "aws_lambda_permission" "ExampleToGateway" {
  statement_id = "AllowAPIGatewayInvoke"
  action = "lambda:InvokeFunction"
  principal = "apigateway.amazonaws.com"
  function_name = "${aws_lambda_function.Example.arn}"
  source_arn = "${aws_api_gateway_rest_api.ExampleAPI.execution_arn}/*/*/*"
}

resource "aws_iam_role" "Lambda" {
  name = "lambda-role"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

Sorry for long post but it seems a lot of people are having many problems with this thing 😄 This was my setup and it correctly serves images to browser clients as well as allowing me to return non-binary payloads from other routes!