Binary support for HTTP response

[mthenw]

Current implementation support HTTP responses via special JSON object schema. Right now only string is supported as a response body. EG needs to support also binary responses. Requirements:

• no requirement for using any special function wrapper/handler
• works across languages
• toggle per invocation not per function (binary/nonbinary response can be changed between invocation without changing EG configuration)

Possible solution isBinary flag and base64 encoding

Right now the supported object looks like this:

{
    "statusCode": 200,
    "headers": {
      "content-type": "text/html"
    },
    "body": "<hello/>"
}

The proposition is to support additional flag isBinary that will indicate that body is base64 encoded string that carries a binary payload. EG will decode it and return a binary response to the requester. We already pass a Base64 string to a function in case of binary input data so this solution is "symmetrical".

{
    "statusCode": 200,
    "headers": {
      "content-type": "text/html"
    },
    "body": "<base64 string>",
    "isBinary": true
}

Node.js Example

exports.image = (event, context, callback) => {
  fs.readFile("./image.jpg", { encoding: "base64" }, function(err, data) {
    callback(null, {
      statusCode: 200,
      headers: {
        "content-type": "image/jpeg"
      },
      isBinary: true,
      body: data
    });
  });
});

Python Example

def main(event, context):
    with open('./image.jpg', 'rb') as f:
        body = f.read().encode('base64')

    return {
        "statusCode": 200,
        "headers": {
            "content-type": "image/jpeg"
        },
        "isBinary": True,
        "body": body
   }

cc @brianneisler @DavidWells @nikgraf

[xtuc]

Instead of using a binary representation over HTTP I would use end to end binary stream.

• bson (binary json): packing and unpacking is very simple, it's just JSON in a binary representation, it's schemaless which doesn't allow much optimizations.

• protobuf (protocol buffers): require a schema which allows lots of optimizations. Many languages are supported by the SDK. The hidden cost is the schema management.

• gRPC: RPC using protobuf over HTTP2. I would by experience recommend this one. Keep in mind that it uses long-living connections.

[mthenw]

@xtuc the goal is to allow function (e.g. AWS Lambda) returning binary files. Also, it has to be easy to achieve with standard lib across different languages. Not sure if your proposed solutions accomplish that?

[xtuc]

Ok, sorry I didn't know it was to communicate with lambdas.

As I said, bson is really a simple packing technique, it should be implementable in every languages.

[mthenw]

@xtuc the problem is that it requires external libraries. I know that base64 is not perfect but at least it's "universal". In the future, we may support compression between EG <> FaaS.

[mthenw]

Optionally, instead of adding another field in the response object we can just detect binary payload based on returned content type.