Gzipped content cannot be returned from an API Gateway

[AdamDrewsTR]

Description:

For local testing with lambdas behind an API Gateway, I cannot return gzipped content that is base64 encoded that API Gateway expects. Locally I can only return uncompressed API responses. The browser fails to decode the response from the local API Gateway.

Why this matters? Cost, performance, aws limits for large responses.

What if I want gzipped content served from s3 or an API to return gzipped content?

If local responses are too large because they are uncompressed, they will be rejected even though they will work in AWS when compressed.

Steps to reproduce:

Have any API Gateway response return a base64 encoded gziped response. The API Gateway response should indicate that the response is base64 encoded like API Gateway expects. Also ensure the Content-Encoding header is set to gzip.

Observed result:

Base64 gziped content cannot be decoded by any browsers, but works when running in AWS.

Expected result:

Local API Gateway returns a gzipped response.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: MacOS
2. sam --version: 1.24.1
3. AWS region: us-east-1

[wchengru]

Hi @AdamDrewsTR , can you please confirm that the property isBase64Encoded: true works as expected? It seems this is similar to issue #1826.

[AdamDrewsTR]

@wchengru I tested this today, and I still have the same result: a content decoding error that only happens when running locally. It works fine in AWS still, however.

edit: sam cli version: 1.27.2

[AdamDrewsTR]

This is still an issue in version 1.68.0 - returning isBase64Encoded: true with gzipped content works when deployed but not locally via sam.

[dbphuong]

I got similar issue. It turned out the binary_types is empty somehow in _should_base64_decode_body of local_apigw_service.py line 631. The logic is: is_best_match_in_binary_types = best_match_mimetype in binary_types or "*/*" in binary_types

I tried to set BinaryMediaTypes to "/" in Globals but it did not work. To bypass it temporarily for local test, I added len(binary_types) == 0 and build my own custom sam. Does anyone know how to set BinaryMediaTypes for this:

  HelloWorldFunction:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.8
      Architectures:
        - x86_64
      Events:
        HelloWorld:
          Type: Api 
          Properties:
            Path: /hello
            Method: get

I tried

Globals:
  Api:
   BinaryMediaTypes:
     - "*/*"

[royassis]

Having the same issue

I've created a lambda. In the lambda code I use the aws_lambda_powertools package. I am using an APIGatewayRestResolver from the same package. I am returning a binary content with compress=True in one of the lambda's endpoints.

When invoking sam local start-api and sending a get request to said endpoint with the requests package I do get a response back from the api, but I get this error:

requests.exceptions.ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check'))

I do not get this error when invoking the sam lambda after deploying it.

(general-env) [ec2-user@ip-172-30-3-254 server_stack]$ uname -r
5.10.184-175.731.amzn2.x86_64

(general-env) [ec2-user@ip-172-30-3-254 server_stack]$ sam --version
SAM CLI, version 1.91.0

edit: Seems like the data is compressed but is still base64 encoded, I had to decode and decompress manually

res = session.post(url, stream=True, json=body)
res.raise_for_status()

data = res.raw.read()
gzip.decompress(base64.b64decode(data))

[royassis]

Ok my issue was that I returned

from aws_lambda_powertools.event_handler import APIGatewayRestResolver, Response, content_types

default_response = Response(
    status_code=200,
    content_type="application/octet-stream", # was octet-stream and not application/octet-stream
    body=pa.ipc.serialize_pandas(df).to_pybytes()
)

[tanato-cit]

Same error here in SAM version 1.111.0.

My minimal setup to reproduce:

template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  TestFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: nodejs18.x
      Handler: index.handler
      FunctionName: TestGzip
      Events:
        ApiEvent:
          Type: Api
          Properties:
            Path: /test
            Method: GET

index.js

const { gzipSync } = require("zlib");

exports.handler = async (event) => {
    const gzip = gzipSync(JSON.stringify({ message: 'Hello' }));
    const base64 = gzip.toString('base64');

    return {
        isBase64Encoded: true,
        statusCode: 200,
        headers: {
            'Content-Type': 'application/json',
            'Content-Encoding': 'gzip',
        },
        body: base64,
    };
};

It works If I use HTTP Api instead of REST Api as demonstrated bellow, but this is not suitable for my use case where I need to use REST Api due to some limitations of HTTP Api

         ApiEvent:
           Type: Api

to

         ApiEvent:
           Type: HttpApi