S3Events are not deserialized via AWS Lambda RequestHandler APIs

[AjitDas]

I am facing issue while de-serializing JSON to S3Event with Jackson APis throwing error because of non compliant POJO models for these S3Event classes.

Error received after executing the S3Event JSON is something like this.

Can we have simple POJOs version of this S3Event so that it can be de-serialized accordingly.

{
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "aws:s3",
      "awsRegion": "us-east-1",
      "eventTime": "1970-01-01T00:00:00.000Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "EXAMPLE"
      },
      "requestParameters": {
        "sourceIPAddress": "127.0.0.1"
      },
      "responseElements": {
        "x-amz-request-id": "EXAMPLE123456789",
        "x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "testConfigRule",
        "bucket": {
          "name": "test-bucket",
          "ownerIdentity": {
            "principalId": "EXAMPLE"
          },
          "arn": "arn:aws:s3:::test-bucket"
        },
        "object": {
          "key": "test-file.txt",
          "size": 1024,
          "eTag": "0123456789abcdef0123456789abcdef",
          "sequencer": "0A1B2C3D4E5F678901"
        }
      }
    }
  ]
}

Cannot construct instance of `com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification$S3EventNotificationRecord` (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
 at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: com.amazonaws.services.lambda.runtime.events.S3Event["Records"]->java.util.ArrayList[0]): java.lang.IllegalArgumentException
java.lang.IllegalArgumentException: Cannot construct instance of `com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification$S3EventNotificationRecord` (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
 at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: com.amazonaws.services.lambda.runtime.events.S3Event["Records"]->java.util.ArrayList[0])
    at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3938)
    at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:3869)
    at com.serverless.handler.CustomFunctionInvoker.generateMessage(CustomFunctionInvoker.java:259)
    at com.serverless.handler.CustomFunctionInvoker.handleRequest(CustomFunctionInvoker.java:112)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.base/java.lang.reflect.Method.invoke(Unknown Source)
Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification$S3EventNotificationRecord` (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
 at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: com.amazonaws.services.lambda.runtime.events.S3Event["Records"]->java.util.ArrayList[0])
    at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
    at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1592)
    at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1058)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1297)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:286)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:27)
    at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:369)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159)
    at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3933)
    ... 7 more

import com.amazonaws.services.lambda.runtime.events.S3Event;
import com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification;

public class CustomS3EventRequestHandler implements RequestHandler<S3Event, String> {

        @Override
    public Object handleRequest(S3Event event, Context context) {
                // do your operations
                S3EventNotification.S3EventNotificationRecord record = event.getRecords().get(0);
        StringBuilder sb = new StringBuilder().append("Region:").append(record.getAwsRegion()).append(",EventName:")
                    .append(record.getEventName()).append(",Bucket:").append(record.getS3().getBucket().getName())
                    .append(",Object:").append(record.getS3().getObject().getKey());
        return sb.toString();
        }
}

[carlzogh]

Hi @AjitDas - what environment is this exception being thrown in? Was this running in Lambda or elsewhere?

[AjitDas]

Hi @carlzogh this error is coming on my local SAM testing, I have not deployed my code to AWS environment yet. Having said that I think this will come in AWS lambda env as well since serialization would be done same way just like my local SAM testing. Also FYI am using this along with spring framework. I was able to test with other type of events like SQSEvent , SNSEvent etc but for S3Event it doesn't work due to jackson de serialization error.

[TuomasKiviaho]

I'm facing the same issue.

My workaround was to recompile com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification$S3EventNotificationRecord with parameters since https://docs.oracle.com/javase/tutorial/reflect/member/methodparameterreflection.html.

Maven has support for the switch so the workaround could be easily incorporated to the project by default.

<properties>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
    <maven.compiler.parameters>true</maven.compiler.parameters>
</properties>

Note that with older Jackson, one must have an accompanying module to support this (and of course take constructors with multiple parameters into consideration which Jackson doesn't do by default). I verified with it that the arguments got their names via refection by debugging com.fasterxml.jackson.module.paramnames.ParameterNamesAnnotationIntrospector https://github.com/FasterXML/jackson-modules-java8/tree/master/parameter-names.

[TuomasKiviaho]

I forgot to mention that JsonCreator from Jackson tag is still needed even when the parameter names are present and the binding would otherwise work. This is so that the de-serialization can deal with ambiguities (duplicate constructors etc.) and it's baked into how Jackson is meant to work.

Perhaps a pull request for Jackson's ParameterNamesAnnotationIntrospector option would solve this that would make it to consider all constructors that are not deprecated as explicitly declared constructors. This would mimic the presence of the JsonCreator without actually having the annotation present.

BTW: There's an alternative JDK7 based java.beans.ConstructorProperties annotation via com.fasterxml.jackson.databind.ext.Java7Support if direct Jackson dependency is a concern. I'm just afraid that this kind of binding would cause issues with java11 module system.

[TuomasKiviaho]

Having still the ability to parse InputStream using the model would be great, but currently this is impossible without using own mixins etc to guide the process which would break easily whenever the model is updated.

There would be also the official JSON-B that give the capabilities analogous to what Jackson annotations used to provide in previous versions.

I think that even Jackson 2.11 could be then used as a JSON-P parser for Yasson that is the RI for the JSON-B standard.

[CraigGoodspeed]

I think this will resolve your problem.

https://github.com/aws/aws-lambda-java-libs/tree/master/aws-lambda-java-tests

[rehevkor5]

I think what @CraigGoodspeed is trying to point out is this module:

    <dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>aws-lambda-java-serialization</artifactId>
      <version>1.0.0</version>
    </dependency>

Which you can use like this:

final PojoSerializer<S3EventNotification> s3EventSerializer =
        LambdaEventSerializers.serializerFor(S3EventNotification.class, ClassLoader.getSystemClassLoader());

S3EventNotification eventNotification = s3EventSerializer.fromJson(message.body());

[mo-rjr]

@rehevkor5 for some reason I was having trouble with that solution on the AWS lambda environment though it worked fine on my machine -- it couldn't find the S3EventNotification class. I tried a few things but in the end this worked for me, so I've settled for it: S3EventNotification s3EventNotification = S3EventNotification.parseJson(snsMessage); with the import import com.amazonaws.services.s3.event.S3EventNotification; though it meant I had to import the v1 SDK S3 library.

[gregRzn]

@rehevkor5

I think what @CraigGoodspeed is trying to point out is this module:

    <dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>aws-lambda-java-serialization</artifactId>
      <version>1.0.0</version>
    </dependency>

Which you can use like this:

final PojoSerializer<S3EventNotification> s3EventSerializer =
        LambdaEventSerializers.serializerFor(S3EventNotification.class, ClassLoader.getSystemClassLoader());

S3EventNotification eventNotification = s3EventSerializer.fromJson(message.body());

Thanks a lot, this saved me a lot of trouble.

[DeepController]

@mo-rjr Use Thread.currentThread().getContextClassLoader() instead of ClassLoader.getSystemClassLoader() would solve this issue.

Ref: https://github.com/aws/aws-lambda-java-libs/issues/262#issuecomment-887005624