DynamoDB events have milliseconds, not seconds, in their timestamps which crashes deserialization

cdegroot commented 3 years ago

Description

I'm creating a lambda to process DynamoDB events streamed through Kinesis, which means they're wrapped in base64, etcetera. During processing I pull them out and then deserialize them to DynamoDBv2 records:

            var serializer = new Amazon.Lambda.Serialization.Json.JsonSerializer();
            var dynamoEvent = serializer.Deserialize<Amazon.DynamoDBv2.Model.Record>(BaseObserver.GenerateStreamFromString(dynamoData));

Reproduction Steps

This gist: https://gist.github.com/cdegroot/6e8e1957d9d08a595b3918dce3e16d84. Note that all I did was to add a couple of zeros to the test event.

Logs

Stacktrace:

  X TestDynamoParsing [8ms]
  Error Message:
   Test method Canary.EventStore.Tests.BaseObserverTests.TestDynamoParsing threw exception: 
Amazon.Lambda.Serialization.Json.JsonSerializerException: Error converting the Lambda event JSON payload to type Amazon.DynamoDBv2.Model.Record: Value to add was out of range. (Parameter 'value') ---> System.ArgumentOutOfRangeException: Value to add was out of range. (Parameter 'value')                                                                                    
  Stack Trace:
      at System.DateTime.Add(Double value, Int32 scale)
   at System.DateTime.AddSeconds(Double value)
   at Amazon.Lambda.Serialization.Json.JsonNumberToDateTimeDataConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at Amazon.Lambda.Serialization.Json.JsonSerializer.Deserialize[T](Stream requestStream)
--- End of inner exception stack trace ---
    at Amazon.Lambda.Serialization.Json.JsonSerializer.Deserialize[T](Stream requestStream

Environment

Build Version: 1.2.0
OS Info: Ubuntu 20.10
Build Environment: Terminal dotnet test etc.
Targeted .NET Platform: 3.1

Resolution

[ ] :wave: I can/would-like-to implement a fix for this problem myself

I can look later, will just kill the last three digits with a regex for now, but the usual fix is to take a cut-off value (100000000000 - somewhere in the 6th millenium) and if it's above, interpret as millisecs, if not, as secs.

Of course, the proper fix is to make DynamoDB emit always one or the other :)

This is a :bug: bug-report

ashishdhingra commented 3 years ago

Hi @cdegroot,

Good afternoon.

Looking at the code, looks like it is an automated unit test code where your test data is real record from DynamoDB Streams. Could you please share the source for the JSON string? In you JSON string, the value for ApproximateCreationDateTime is 1480642020000. However, the example at https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.Lambda.Tutorial.html, has value 1479499740 for example. Notice the extra last 3 digits, as pointed by you as well. Not sure if there is an issue with your test data. While I didn't got a chance to examine the actual event received from DynamoDB stream, just wanted to validate your test data.

Thanks, Ashish

cdegroot commented 3 years ago

I created that test based on real data I saw flowing so I could reproduce (and work around the issue). I won't share our full data in public, but hopefully the following exposition helps.

From another (JS-based) Kinesis consumer, which is logging full Kinesis events, I copied an event:

var kinesisData = @"{""Records"":[{""kinesis"":{""kinesisSchemaVersion"":""1.0"",""partitionKey"":""D7FE246F3E699C3B143D2A8378D17223"",""sequenceNumber"":""49616371596847055066813995925388293329944250222301413394"",""data"":""eyJhd3....1vZGIifQ=="",""approximateArrivalTimestamp"":1616427312.824},""eventSource"":""aws:kinesis"",""eventVersion"":""1.0"",""eventID"":""shardId-000000000001:49616371596847055066813995925388293329944250222301413394"",""eventName"":""aws:kinesis:record"",""invokeIdentityArn"":""arn:aws:iam::..."",""awsRegion"":""us-east-1"",""eventSourceARN"":""arn:aws:kinesis:us-east-1:147803588724:stream...""}]}";

I snipped most of the payload for brevity and privacy, but base64 decoding it:

{
  "awsRegion": "us-east-1",
  "dynamodb": {
    "ApproximateCreationDateTime": 1616427312786,
    "Keys": {
      "Id": {
        "S": "SHARED/User/0gKQCSqc3bQUW90Fch6oCv"
      },
      "Version": {
        "N": "1"
      }
    },
    "NewImage": {
      "Event": {
        "S": "..."
      },
      "Version": {
        "N": "1"
      },
      "Id": {
        "S": "SHARED/User/0gKQCSqc3bQUW90Fch6oCv"
      },
      "Type": {
        "S": "InviteDeleted"
      }
    },
    "SizeBytes": 452
  },
  "eventID": "408c266f-06b0-43e6-9e31-68f530664e6c",
  "eventName": "INSERT",
  "userIdentity": null,
  "recordFormat": "application/json",
  "tableName": "dev1-EventLog",
  "eventSource": "aws:dynamodb"
}

(again edited for brevity and privacy). As you can see, there's a 13 digit timestamp in there. I added three zeroes to the existing test data (which I think I snipped from here] to reproduce the issue.

ashishdhingra commented 3 years ago

Hi @cdegroot,

Good afternoon.

Unfortunately I'm unable to reproduce the issue. Used the below steps:

Created DynamoDb table HighScoresWithStream with:

Username (String) as Primary partition key and Game (String) as Primary sort key
DynamoDB stream enabled (View type as New and old images)

Created the following Lambda function: .csproj

<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
<AWSProjectType>Lambda</AWSProjectType>

<!-- This property makes the build directory similar to a publish directory and helps the AWS .NET Lambda Mock Test Tool find project dependencies. -->
<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Amazon.Lambda.Core" Version="2.0.0" />
<PackageReference Include="Amazon.Lambda.DynamoDBEvents" Version="2.0.0" />
<PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.1.0" />
</ItemGroup>
</Project>

Function.cs


using Amazon.Lambda.Core;
using System.IO;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class. [assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace DynamoDbStream_Issue839 { public class Function { public void FunctionHandler(DynamoDBEvent dynamoDBEvent, ILambdaContext context) { context.Logger.LogLine($"DynamoDBEvent received with {dynamoDBEvent.Records.Count} records."); foreach (var record in dynamoDBEvent.Records) { context.Logger.LogLine($"{record.EventID}:{record.EventName}, Dynamodb.ApproximateCreationDateTime:{record.Dynamodb.ApproximateCreationDateTime}"); } } } }

**aws-lambda-tools-defaults.json**
```JSON

{
    "Information" : [
        "This file provides default values for the deployment wizard inside Visual Studio and the AWS Lambda commands added to the .NET Core CLI.",
        "To learn more about the Lambda commands with the .NET Core CLI execute the following command at the command line in the project root directory.",
        "dotnet lambda help",
        "All the command line options for the Lambda command can be specified in this file."
    ],
    "profile"     : "default",
    "region"      : "us-east-2",
    "configuration" : "Release",
    "framework"     : "netcoreapp3.1",
    "function-runtime" : "dotnetcore3.1",
    "function-memory-size" : 256,
    "function-timeout"     : 30,
    "function-handler"     : "DynamoDbStream_Issue839::DynamoDbStream_Issue839.Function::FunctionHandler",
    "function-name"        : "StreamConsumer",
    "package-type"         : "Zip",
    "function-role"        : "arn:aws:iam::139480602983:role/lambda_exec_StreamConsumer-0",
    "tracing-mode"         : "PassThrough",
    "environment-variables" : "",
    "image-tag"             : "",
    "function-description"  : ""
}

NOTE: To keep things simple for reproduction, the role lambda_exec_StreamConsumer-0 has AWSLambdaExecute and AmazonDynamoDBFullAccess policies attached.

Published the Lambda function to AWS.
In AWS Lambda console, added DynamoDB trgigger for table HighScoresWithStream
Added some records to DynamoDB database.

Examined the CloudWatch logs:

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                                                                              message                                                                                                                                               |
|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1617405537095 | START RequestId: 4c7331ac-6882-449b-8488-e78994536976 Version: $LATEST                                                                                                                                                                                                                             |
| 1617405537203 | DynamoDBEvent received with 1 records.                                                                                                                                                                                                                                                             |
| 1617405537282 | 1d507e4abd270fc658c78593be3ad773:INSERT, Dynamodb.ApproximateCreationDateTime:4/2/2021 11:18:18 PM                                                                                                                                                                                                 |
| 1617405537300 | END RequestId: 4c7331ac-6882-449b-8488-e78994536976                                                                                                                                                                                                                                                |
| 1617405537300 | REPORT RequestId: 4c7331ac-6882-449b-8488-e78994536976 Duration: 202.58 ms Billed Duration: 203 ms Memory Size: 256 MB Max Memory Used: 70 MB                                                                                                                                                      |
| 1617405565785 | START RequestId: 5bbf64d3-7f10-437d-bd00-4185c5b3c8fe Version: $LATEST                                                                                                                                                                                                                             |
| 1617405565790 | DynamoDBEvent received with 1 records.                                                                                                                                                                                                                                                             |
| 1617405565790 | 421616e0bcb058e190f341ed8227f4e5:INSERT, Dynamodb.ApproximateCreationDateTime:4/2/2021 11:19:25 PM                                                                                                                                                                                                 |
| 1617405565790 | END RequestId: 5bbf64d3-7f10-437d-bd00-4185c5b3c8fe                                                                                                                                                                                                                                                |
| 1617405565790 | REPORT RequestId: 5bbf64d3-7f10-437d-bd00-4185c5b3c8fe Duration: 3.31 ms Billed Duration: 4 ms Memory Size: 256 MB Max Memory Used: 70 MB                                                                                                                                                                                                                                                                                                                   |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Modified the code in Function.cs to below to capture JSON:
```
using Amazon.Lambda.Core;
using System.IO;
```

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class. [assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace DynamoDbStream_Issue839 { public class Function { public Stream FunctionHandler(Stream userInput, ILambdaContext context) { string jsonString; using (var reader = new StreamReader(userInput)) { jsonString = reader.ReadToEnd(); }

        context.Logger.LogLine($"JSON received: {jsonString}");

        byte[] bytes = System.Text.Encoding.UTF8.GetBytes(jsonString);
        MemoryStream stream = new MemoryStream(bytes);
        return stream;
    }
}

}


8. Republished the Lambda.
9. Added record to DynamoDB table.
10. Examined the CloudWatch logs:

JSON received: { "Records": [ { "eventID": "1416c843df272af3718f3173fb9b0363", "eventName": "INSERT", "eventVersion": "1.1", "eventSource": "aws:dynamodb", "awsRegion": "us-east-2", "dynamodb": { "ApproximateCreationDateTime": 1617406031, "Keys": { "Game": { "S": "testgame" }, "Username": { "S": "multiple" } }, "NewImage": { "Game": { "S": "testgame" }, "Username": { "S": "multiple" } }, "SequenceNumber": "5800000000005804693158", "SizeBytes": 56, "StreamViewType": "NEW_AND_OLD_IMAGES" }, "eventSourceARN": "arn:aws:dynamodb:us-east-2:139480602983:table/HighScoresWithStream/stream/2021-04-02T22:37:03.187" } ] }



As you could see from JSON captured above, the `ApproximateCreationDateTime` has value with 10 digits (not 13 digits).

Please let me know if this helps.

Thanks,
Ashish

github-actions[bot] commented 3 years ago

This issue has not recieved a response in 2 weeks. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

josh-c-kendrick commented 2 years ago

I am seeing this issue as well, and receiving the same "value out of range" exception related to the "ApproximateCreationDateTime" field during deserialization.

This is only occurring when trying to processes events that are being streamed through Kinesis (DyanmoDB --> Kinesis --> Lambda). I believe that is why you couldn't reproduce it.

Tragetaschen commented 2 years ago

I can second that analysis, especially after studying the code and what is in the event. I wonder if this issue is going to be reopened or if it's better to open a new one, though.

ashishdhingra commented 2 years ago

@Tragetaschen It would be better to open a new issue linking it to this one. Thanks.

ashishdhingra commented 2 years ago

Issue is reproducible using the following steps:

Created a Kinesis Data Stream test-kinesis-data-stream.
Created a DynamoDb table testdynamoddbtable with recordId (string) as partition key and temp (number) as sort key.\
In DynamoDB table settings, specified the Kinesis data stream created earlier as destination stream in Kinesis data stream details setting.
Created a new Lambda function project named DynamoDbKinesisLambdaNewTest using Simple Kinesis Function AWS blueprint.

Used the below code for testing purposes to demonstrate end-to-end scenario:


using Amazon.DynamoDBv2.Model;
using Amazon.Lambda.Core;
using Amazon.Lambda.KinesisEvents;
using Amazon.Lambda.Serialization.SystemTextJson;
using System;
using System.IO;
using System.Text;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class. [assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace DynamoDbKinesisLambdaNewTest { public class Function { public void FunctionHandler(KinesisEvent kinesisEvent, ILambdaContext context) { context.Logger.LogLine($"Beginning to process {kinesisEvent.Records.Count} records...");

        foreach (var record in kinesisEvent.Records)
        {
            context.Logger.LogLine($"Event ID: {record.EventId}");
            context.Logger.LogLine($"Event Name: {record.EventName}");

            context.Logger.LogLine($"ApproximateArrivalTimestamp: {record.Kinesis.ApproximateArrivalTimestamp}");

            string recordData = GetRecordContents(record.Kinesis);
            context.Logger.LogLine($"Record Data:");
            context.Logger.LogLine(recordData);

            context.Logger.LogLine($"Record Data (base64):");
            context.Logger.LogLine(Convert.ToBase64String(record.Kinesis.Data.ToArray()));

            context.Logger.LogLine($"Deserializing data to 'Amazon.DynamoDBv2.Model.Record'");
            record.Kinesis.Data.Position = 0;
            var serializer = new DefaultLambdaJsonSerializer();
            Record dynamoDBRecord = serializer.Deserialize<Amazon.DynamoDBv2.Model.Record>(record.Kinesis.Data);

            context.Logger.LogLine($"Successfully deserialized data to 'Amazon.DynamoDBv2.Model.Record'");
        }

        context.Logger.LogLine("Stream processing complete.");
    }

    private string GetRecordContents(KinesisEvent.Record streamRecord)
    {
        using (var reader = new StreamReader(streamRecord.Data, Encoding.UTF8, true, -1, true))
        {
            return reader.ReadToEnd();
        }
    }
}

}

- - Published the function with name `TestDynamoDbKinesisLambda` to AWS Lambda. In Lambda publish settings, specified new IAM role to be based out of `AWSLambdaKinesisExecutionRole`.
- - Open the deployed Lambda function in AWS console. Navigate to Configuration > Triggers and add a new Kinesis trigger. Specify the Kinesis stream created earlier, start position as `LATEST` and Batch size 1. 
- Using DynamoDB console, added a new item using the following DynamoDB JSON:
```JSON
{
  "date": {
    "S": "2020-03-07T16:29:50+02:00"
  },
  "reactor": {
    "S": "reactor-1"
  },
  "recordId": {
    "S": "2"
  },
  "temp": {
    "N": "800"
  }
}

RESULT: Lambda function is triggered. Error during deserialization to Amazon.DynamoDBv2.Model.Record is logged in CloudWatch logs:

--------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     message                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1646158314935 | START RequestId: b73097c0-913e-4b06-9f22-6f115d6fb051 Version: $LATEST                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| 1646158315193 | Beginning to process 1 records...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| 1646158315209 | Event ID: shardId-000000000001:49627225517255316116163071331502728924989303313731158034                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| 1646158315209 | Event Name: aws:kinesis:record                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| 1646158315250 | ApproximateArrivalTimestamp: 3/1/2022 6:04:26 PM                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| 1646158315251 | Record Data:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| 1646158315251 | {"awsRegion":"us-east-2","eventID":"4491dcfa-b160-40fc-90b0-7c2a6cf32354","eventName":"INSERT","userIdentity":null,"recordFormat":"application/json","tableName":"testdynamoddbtable","dynamodb":{"ApproximateCreationDateTime":1646157866523,"Keys":{"recordId":{"S":"2"},"temp":{"N":"800"}},"NewImage":{"temp":{"N":"800"},"date":{"S":"2020-03-07T16:29:50+02:00"},"recordId":{"S":"2"},"reactor":{"S":"reactor-1"}},"SizeBytes":75},"eventSource":"aws:dynamodb"}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| 1646158315251 | Record Data (base64):                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| 1646158315251 | eyJhd3NSZWdpb24iOiJ1cy1lYXN0LTIiLCJldmVudElEIjoiNDQ5MWRjZmEtYjE2MC00MGZjLTkwYjAtN2MyYTZjZjMyMzU0IiwiZXZlbnROYW1lIjoiSU5TRVJUIiwidXNlcklkZW50aXR5IjpudWxsLCJyZWNvcmRGb3JtYXQiOiJhcHBsaWNhdGlvbi9qc29uIiwidGFibGVOYW1lIjoidGVzdGR5bmFtb2RkYnRhYmxlIiwiZHluYW1vZGIiOnsiQXBwcm94aW1hdGVDcmVhdGlvbkRhdGVUaW1lIjoxNjQ2MTU3ODY2NTIzLCJLZXlzIjp7InJlY29yZElkIjp7IlMiOiIyIn0sInRlbXAiOnsiTiI6IjgwMCJ9fSwiTmV3SW1hZ2UiOnsidGVtcCI6eyJOIjoiODAwIn0sImRhdGUiOnsiUyI6IjIwMjAtMDMtMDdUMTY6Mjk6NTArMDI6MDAifSwicmVjb3JkSWQiOnsiUyI6IjIifSwicmVhY3RvciI6eyJTIjoicmVhY3Rvci0xIn19LCJTaXplQnl0ZXMiOjc1fSwiZXZlbnRTb3VyY2UiOiJhd3M6ZHluYW1vZGIifQ==                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| 1646158315251 | Deserializing data to 'Amazon.DynamoDBv2.Model.Record'                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| 1646158315509 | Error converting the Lambda event JSON payload to type Amazon.DynamoDBv2.Model.Record: Value to add was out of range. (Parameter 'value'): JsonSerializerException    at Amazon.Lambda.Serialization.SystemTextJson.AbstractLambdaJsonSerializer.Deserialize[T](Stream requestStream)    at DynamoDbKinesisLambdaNewTest.Function.FunctionHandler(KinesisEvent kinesisEvent, ILambdaContext context) in D:\source\repros\DynamoDbKinesisLambdaNewTestScenario\DynamoDbKinesisLambdaNewTest\Function.cs:line 43     at System.DateTime.Add(Double value, Int32 scale)    at System.DateTime.AddSeconds(Double value)    at Amazon.Lambda.Serialization.SystemTextJson.Converters.DateTimeConverter.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options)    at System.Text.Json.JsonPropertyInfoNotNullable`4.OnRead(ReadStack& state, Utf8JsonReader& reader)    at System.Text.Json.JsonPropertyInfo.Read(JsonTokenType tokenType, ReadStack& state, Utf8JsonReader& reader)    at System.Text.Json.JsonSerializer.ReadCore(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& readStack)    at System.Text.Json.JsonSerializer.ReadCore(Type returnType, JsonSerializerOptions options, Utf8JsonReader& reader)    at System.Text.Json.JsonSerializer.ParseCore(ReadOnlySpan`1 utf8Json, Type returnType, JsonSerializerOptions options)    at System.Text.Json.JsonSerializer.Deserialize[TValue](ReadOnlySpan`1 utf8Json, JsonSerializerOptions options)    at Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer.InternalDeserialize[T](Byte[] utf8Json)    at Amazon.Lambda.Serialization.SystemTextJson.AbstractLambdaJsonSerializer.Deserialize[T](Stream requestStream)   |
| 1646158315572 | END RequestId: b73097c0-913e-4b06-9f22-6f115d6fb051                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
------------------------------------------------------------------------------------------------------------------

Looks like in the DynamoDB record sent to Kinesis data stream has ApproximateCreationDateTime with 13 digits (as opposed to 10 digits, e.g. 1646158314147).

For testing purposes, the base64 encoded string in CloudWatch logs could be used with Kinesis example request in Mock Lambda Test tool in Visual Studio.

The issue is probably related to the data sent from DynamoDB to Kinesis data stream.

Related articles:

Thanks, Ashish

ashishdhingra commented 2 years ago

Tracking the above issue internally. Would post here as updates are available.

Tragetaschen commented 2 years ago

Shouldn't this be reopened and kept open then?

ashovlin commented 2 years ago

Agreed, reopening.

It does appear that the service response may vary depending on context.

DynamoDB Streams - https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_StreamRecord.html

The approximate date and time when the stream record was created, in UNIX epoch time format and rounded down to the closest second.
Kinesis - https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/kds.html

ApproximateCreationDateTime indicates the time of the modification in milliseconds.

I think we may be able to either try to recognize that we're within a Kinesis context, or else DateTimeConverter could be expanded to handle both seconds and milliseconds as the original issue suggested.

ashishdhingra commented 6 months ago

@cdegroot Good afternoon. The issue should be addressed as part of PR https://github.com/aws/aws-lambda-dotnet/pull/1710. Could you please try upgrading to latest versions of Amazon.Lambda.Serialization.Json and Amazon.Lambda.Serialization.SystemTextJson, and verify if the issue is resolved?

Thanks, Ashish

github-actions[bot] commented 6 months ago

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

aws / aws-lambda-dotnet