Unable to get body from APIGatewayV2HTTPEvent for Spring Cloud Function in AWS Lambda behind a ALB

[beacon-robertbreedt]

Describe the bug

• spring-cloud-starter-function-web v3.2.6
• spring-cloud-function-adapter-aws v3.2.6

I have a Spring Cloud Function application setup in AWS Lambda as a docker image. One of the function is defined as Function<APIGatewayV2HTTPEvent, APIGatewayV2HTTPResponse>. The lambda is behind an Application Load Balancer (ALB).

When I send request to the load balancer, it is successfully routed to the function and is correctly invoked, however, the provided "body" is always removed before it is sent to my Function.

Here are some of the logs from AWS cloudwatch:

Incoming JSON Event: {""requestContext"":{""elb"":{""targetGroupArn"":""arn:aws:elasticloadbalancing:eu-west-1:<removed>:targetgroup/function-1-tg/<removed>""}},""httpMethod"":""POST"",""path"":""/function-1"",""queryStringParameters"":{},""headers"":{""accept"":""*/*"",""accept-encoding"":""gzip, deflate, br"",""connection"":""keep-alive"",""content-length"":""20"",""content-type"":""application/json"",""host"":""alb-test-<removed>.eu-west-1.elb.amazonaws.com"",""postman-token"":""<removed>"",""spring.cloud.function.definition"":""greeter"",""user-agent"":""PostmanRuntime/7.29.2"",""x-amzn-trace-id"":""Root=<removed>"",""x-forwarded-port"":""80"",""x-forwarded-proto"":""http""},""body"":""{
    \""body\"": \""hi\""
}"",""isBase64Encoded"":false}

The above shows the request message has a body - "body": "hi"

When I log the APIGatewayV2HTTPEvent that's provided to my cloud function, I get the below output:

incoming request: APIGatewayV2HTTPEvent(version=null, routeKey=null, rawPath=null, rawQueryString=null, cookies=null, headers=null, queryStringParameters={}, pathParameters=null, stageVariables=null, body=null, isBase64Encoded=false, requestContext=APIGatewayV2HTTPEvent.RequestContext(routeKey=null, accountId=null, stage=null, apiId=null, domainName=null, domainPrefix=null, time=null, timeEpoch=0, http=null, authorizer=null, requestId=null))

In the above, the payload has a null body. I looked through the AWSLambdaUtils and noticed the body is removed here and a new MessageBuilder is returned, however I'm unsure whether this piece is actually hit or working correctly.

I've tested with multiple different input types to the function (string, Map<String, Object>, Message) and all results are the same. I've also tested this through the AWS console by sending an api-gateway-aws-proxy test event and the same thing happens.

Sample

Here's the sample function:

package sample.springcloudfunction.functions;

import com.amazonaws.services.lambda.runtime.events.APIGatewayV2HTTPEvent;
import com.amazonaws.services.lambda.runtime.events.APIGatewayV2HTTPResponse;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;

import java.util.Map;
import java.util.function.Function;

@Component
public class Greeter implements Function<APIGatewayV2HTTPEvent, APIGatewayV2HTTPResponse> {
    private static final Logger LOG = LoggerFactory.getLogger(Greeter.class);

    @Override
    public APIGatewayV2HTTPResponse apply(final APIGatewayV2HTTPEvent event) {
        LOG.info("incoming request: {}", event);
        final var body = event.getBody();
        LOG.info("Greeting: {}", body);
        return APIGatewayV2HTTPResponse.builder()
                .withStatusCode(200)
                .withIsBase64Encoded(false)
                .withHeaders(Map.of(
                        "Content-Type", "application/json"
                ))
                .withBody("Hello " + body + ", welcome to Spring Cloud Functions!")
                .build();
    }
}

I've attached a zip of a sample project as well as cloudwatch event logs.

[spring-cloud-function-sample.zip](https://github.com/spring-cloud/spring-cloud-function/files/9242920/spring-cloud-function-sample.zip)

[olegz]

So, I believe this is related to a similar issue we had with 'headers' where we had similar code to remove headers. I can't exactly recall the reason for it but you may notice there was a TODO comment to get rid of all that code in AWSUtils and delegate to Message Converter etc. . . In any event, i just pushed that change - https://github.com/spring-cloud/spring-cloud-function/commit/8bcdeb5cc2a56102dba84a3cac7df034fe8d5e59 so please try with the latest snapshot - 3.2.7-SNAPSHOT.

I'll keep it open for now

[beacon-robertbreedt]

Thanks, under which repository is the snapshot available?

[olegz]

@beacon-robertbreedt it's here - https://repo.spring.io/snapshot

[beacon-robertbreedt]

Ok, so I've done some testing and here's my results:

Greeter: Function<APIGatewayV2HTTPEvent, APIGatewayV2HTTPResponse>

• headers and body from API request is correctly passed to function
• function result returns correctly as APIGatewayV2HTTPResponse object with full details
• API response returns with different headers than ones set in the APIGatewayV2HTTPResponse object
  • response body has property headers with Content-Type: application/json in it
  • response headers has Content-Type: application/octet-stream
  • is this an issue? idk

Code

```java package sample.springcloudfunction.functions; import com.amazonaws.services.lambda.runtime.events.APIGatewayV2HTTPEvent; import com.amazonaws.services.lambda.runtime.events.APIGatewayV2HTTPResponse; import com.fasterxml.jackson.core.type.TypeReference; import com.fasterxml.jackson.databind.ObjectMapper; import lombok.SneakyThrows; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.stereotype.Component; import java.util.Map; import java.util.function.Function; @Component public class Greeter implements Function { private static final Logger LOG = LoggerFactory.getLogger(Greeter.class); private final TypeReference VALUE_TYPE_REF = new TypeReference<>() { }; private final ObjectMapper objectMapper = new ObjectMapper(); @SneakyThrows @Override public APIGatewayV2HTTPResponse apply(APIGatewayV2HTTPEvent event) { LOG.info("incoming request: {}", event); var body = event.getBody(); var input = objectMapper.readValue(body, VALUE_TYPE_REF); var result = run(input); final var response = APIGatewayV2HTTPResponse.builder() .withStatusCode(200) .withIsBase64Encoded(false) .withHeaders(Map.of( "Content-Type", "application/json" )) .withBody(objectMapper.writeValueAsString(result)) .build(); LOG.info("outgoing response: {}", response); return response; } public String run(final String input) { LOG.info("incoming input: {}", input); var result = "Hello " + input + ", welcome to Spring Cloud Functions!"; LOG.info("outgoing result: {}", result); return result; } } ```

Greeter with body: Function<APIGatewayV2HTTPEvent, APIGatewayV2HTTPResponse>

• same results as above, works as expected

Code

```java package sample.springcloudfunction.functions; import com.amazonaws.services.lambda.runtime.events.APIGatewayV2HTTPEvent; import com.amazonaws.services.lambda.runtime.events.APIGatewayV2HTTPResponse; import com.fasterxml.jackson.databind.ObjectMapper; import lombok.SneakyThrows; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.stereotype.Component; import sample.springcloudfunction.models.Input; import sample.springcloudfunction.models.Output; import java.util.Map; import java.util.function.Function; @Component public class Greeter implements Function { private static final Logger LOG = LoggerFactory.getLogger(Greeter.class); private final ObjectMapper objectMapper = new ObjectMapper(); @SneakyThrows @Override public APIGatewayV2HTTPResponse apply(APIGatewayV2HTTPEvent event) { LOG.info("incoming request: {}", event); var body = event.getBody(); var input = objectMapper.readValue(body, Input.class); var result = run(input); final var response = APIGatewayV2HTTPResponse.builder() .withStatusCode(200) .withIsBase64Encoded(false) .withHeaders(Map.of( "Content-Type", "application/json" )) .withBody(objectMapper.writeValueAsString(result)) .build(); LOG.info("outgoing response: {}", response); return response; } public Output run(final Input input) { LOG.info("incoming input: {}", input); var message = "Hello " + input + ", welcome to Spring Cloud Functions!"; var result = new Output(); result.setMessage(message); LOG.info("outgoing result: {}", result); return result; } } ``` ```java package sample.springcloudfunction.models; import lombok.*; import lombok.extern.jackson.Jacksonized; @Jacksonized @Data @AllArgsConstructor @NoArgsConstructor public class Input { private String name; } ``` ```java package sample.springcloudfunction.models; import lombok.*; import lombok.extern.jackson.Jacksonized; @Jacksonized @Data @AllArgsConstructor @NoArgsConstructor public class Output { private String message; } ```

String Greeter: Function<String, String>

• API request is passed as json encoded string to function
  • i.e. {\"requestContext\":{\"elb\":{\"targetGroupArn\":\"arn:aws:elasticloadbalancing:eu-west-1::targetgroup/function-1-tg/\"}},\"httpMethod\":\"GET\",\"path\":\"/function-1\",\"queryStringParameters\":{},\"headers\":{\"accept\":\"/\",\"accept-encoding\":\"gzip, deflate, br\",\"connection\":\"keep-alive\",\"content-length\":\"8\",\"content-type\":\"application/json\",\"host\":\"alb-test-.eu-west-1.elb.amazonaws.com\",\"postman-token\":\"\",\"spring.cloud.function.definition\":\"greeter\",\"user-agent\":\"PostmanRuntime/7.29.2\",\"x-amzn-trace-id\":\"Root=\",\"x-forwarded-port\":\"80\",\"x-forwarded-proto\":\"http\"},\"body\":\"\\"Robert\\"\",\"isBase64Encoded\":false}
• API response returns String back as result

Code

```java package sample.springcloudfunction.functions; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.stereotype.Component; import java.util.function.Function; @Component public class Greeter implements Function { private static final Logger LOG = LoggerFactory.getLogger(Greeter.class); @Override public String apply(final String input) { LOG.info("incoming input: {}", input); var message = "Hello " + input + ", welcome to Spring Cloud Functions!"; LOG.info("outgoing result: {}", message); return message; } } ```

Object Greeter: Function<Input, Output>

• empty Input object (Input(name=null)) is passed to function as it does not dynamically convert to this from the APIGatewayV2HTTPEvent's body
• API response is sent back correctly as Output class

Code

```java package sample.springcloudfunction.functions; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.stereotype.Component; import sample.springcloudfunction.models.Input; import sample.springcloudfunction.models.Output; import java.util.function.Function; @Component public class Greeter implements Function { private static final Logger LOG = LoggerFactory.getLogger(Greeter.class); public Output run(final Input input) { LOG.info("incoming input: {}", input); var message = "Hello " + input + ", welcome to Spring Cloud Functions!"; var result = new Output(); result.setMessage(message); LOG.info("outgoing result: {}", result); return result; } } ```

[srunnoe]

Olegz (asking on a thread that is still open),

I noticed that the AWSCompanionAutoConfiguration in not included in the spring.factories file. How is AWSTypesMessageConverter ever suppose to make its way into the path of execution?

[olegz]

Look at the start() method of FunctionInvoker

ConfigurableApplicationContext context = ApplicationContextInitializer.class.isAssignableFrom(startClass)
                ? FunctionalSpringApplication.run(new Class[] {startClass, AWSCompanionAutoConfiguration.class}, properties)
                        : SpringApplication.run(new Class[] {startClass, AWSCompanionAutoConfiguration.class}, properties);

That said, i should probably rename it to just AWSCompanionConfiguration as I can now understand your confusion.

[beacon-robertbreedt]

@olegz when can we expect the next release with the fixes above?

[olegz]

next week or the week after

[olegz]

@beacon-robertbreedt while I am still looking I am also trying to figure out if you believe there is still an issue. It's a little inconclusive from your report - the first scenario where you say returns correctly as APIGatewayV2HTTPResponse object with full details and than response body has property headers with Content-Type: application/json in it response headers has Content-Type: application/octet-stream. Can you please clarify?

[beacon-robertbreedt]

Sure, so if the output of a function is APIGatewayV2HTTPResponse, it seems to double wrap the response if the request is determined to be from an API gateway/LB. It's best shown with screenshots.

Request and response via LB from Postman:

Response headers: Content-Type is application/json

From AWS console, using the exact same request as above ^ retrieved from logs:

Response in AWS console:

• first object's headers show application/json

Looks like the function's APIGatewayV2HTTPResponse is wrapped within a APIGatewayV2HTTPResponse in the console, but I'm not sure if this is an issue. If the request originates through the LB, it'll consume the first APIGatewayV2HTTPResponse as that's what it expects and forward the "body" content, which is a json encoded APIGatewayV2HTTPResponse string. This is likely where the application/octet-stream is coming from, AWS's LB itself.

I can easily work around this by specifying the output of a function to some object/class. Screenshots for that:

Postman:

response headers also show application/octet-stream

AWS Console:

• headers show application/json

So this isn't an issue with s.c.f, but found the test result interesting and worth sharing.

[beacon-robertbreedt]

I am wondering on the last report for the Function<Input, Output> test if it's possible to somehow determine/deserialise the object from the APIGatewayV2HTTPEvent body. Possibly not as it'd probably need a lot of conditional checks and added logic to determine what type it needs to deserialise into and the format of the provided data based on the Content-Type headers.

[srunnoe]

FunctionInvoker

FunctionInvoker works fine for normal java lambda functions, but what about native java lambda solutions. The FunctionInvoker is not run and instead its just CustomRuntimeInitializer > CustomRuntimeEventLoop. As you know the current solutions is doing message transformations in the AWSLambdaUtils, so how would the 4.x code base call AWSTypesMessageConverter when run in a custom runtime?

[olegz]

I just had a discussion with some of the AWS engineers and it all comes down to the fact that you are using AWS type that do not match the type of the request you are doing. For example if you are using LB then you would have to be mapping it to ApplicationLoadBalancerRequestEvent, and if plain gateway, then APIGatewayV2HTTPEvent. Now. . . that does sound as very inconvenient thing to do and solution for that as you already pointed out is to use POJO or Spring Message if you interested not only in the payload but also a metadata (i.e., headers) that comes with the request.

Anyway, i'll close this as the original issue is addressed. Feel free to raise a new one if you still believe there is a problem.

[beacon-robertbreedt]

@olegz any update on when 3.2.7 will be released?