search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.04k stars 4.68k forks source link

JsonSerializer.Deserialize<T> returns invalid object when called by TypeConverter within AWS Lambda Test Tool #44763

Closed alex-bolenok-centralreach closed 1 year ago

alex-bolenok-centralreach commented 3 years ago

Description

I'm running some code in AWS Lambda Test Tool and running into an issue, when System.Text.Json.JsonSerializer.Deserialize<T> returns different results depending on whether it's being called from the top level or from a System.ComponentModel.TypeConverter method.

This is best illustrated with code:

Lambda.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <Nullable>enable</Nullable>
    <RootNamespace>Main</RootNamespace>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Amazon.Lambda.Core" Version="1.2.0" />
    <PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.1.0" />
    <PackageReference Include="Amazon.Lambda.SQSEvents" Version="1.2.0" />
    <PackageReference Include="JetBrains.Annotations" Version="2020.1.0" />
  </ItemGroup>

</Project>

Lambda.cs

using System;
using System.ComponentModel;
using System.Globalization;
using System.Text.Json;
using Amazon.Lambda.Core;
using Amazon.Lambda.SQSEvents;
using JetBrains.Annotations;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Main
{
    public static class JsonOptions
    {
        public static readonly JsonSerializerOptions JsonSerializerOptions = new JsonSerializerOptions
        {
            PropertyNamingPolicy = JsonNamingPolicy.CamelCase
        };
    }

    internal class JsonTypeConverterAdapter<T> : TypeConverter
    {
        public override object? ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, object value)
            => value is string stringValue
                ? JsonSerializer.Deserialize<T>(stringValue, JsonOptions.JsonSerializerOptions)
                : base.ConvertFrom(context, culture, value);

        public override object? ConvertTo(ITypeDescriptorContext context, CultureInfo culture, object value,
            Type destinationType)
            => value is T tValue && destinationType == typeof(string)
                ? JsonSerializer.Serialize(tValue)
                : base.ConvertTo(context, culture, value, destinationType);

        public override bool CanConvertFrom(ITypeDescriptorContext context, Type sourceType)
            => sourceType == typeof(string) || base.CanConvertFrom(context, sourceType);

        public override bool CanConvertTo(ITypeDescriptorContext context, Type destinationType)
            => destinationType == typeof(T) || base.CanConvertTo(context, destinationType);
    }

    [TypeConverter(typeof(JsonTypeConverterAdapter<S3Bucket>))]
    [Serializable]
    internal class S3Bucket
    {
        public string Bucket { get; set; } = string.Empty;
    }

    internal class Lambda
    {
        [PublicAPI]
        public void Handler(SQSEvent sqsEvent, ILambdaContext lambdaContext)
        {
            const string json = "{\"bucket\":\"test\"}";

            var s3BucketWorks = JsonSerializer.Deserialize<S3Bucket>(json, JsonOptions.JsonSerializerOptions);
            Console.WriteLine(s3BucketWorks.Bucket);
            Console.WriteLine((s3BucketWorks.GetType() == typeof(S3Bucket)).ToString());

            var s3BucketFails = TypeDescriptor.GetConverter(typeof(S3Bucket)).ConvertFrom(json);
            if (s3BucketFails == null)
            {
                return;
            }
            Console.WriteLine(s3BucketFails.GetType());
            Console.WriteLine(typeof(S3Bucket));
            Console.WriteLine((s3BucketFails is S3Bucket).ToString());
        }
    }
}

Run:

dotnet tool install -g Amazon.Lambda.TestTool-3.1
dotnet build -c Debug
cd bin\Debug\netcoreapp3.1
dotnet lambda-test-tool-3.1 --no-ui --function-handler Lambda::Main.Lambda::Handler --pause-exit false

Output:

AWS .NET Core 3.1 Mock Lambda Test Tool (0.10.1)                                                                                                         
Loaded local Lambda runtime from project output <...>\Lambda\bin/Debug/netcoreapp3.1                              
Executing Lambda function without web interface                                                                                                          
... Using function handler Lambda::Main.Lambda::Handler                                                                                                  
... No payload configured. If a payload is required set the --payload switch to a file path or a JSON document.                                          
... Setting AWS_PROFILE environment variable to default.                                                                                                 
... No default AWS region configured. The --region switch can be used to configure an AWS Region.                                                        
Captured Log information:                                                                                                                                
test                                                                                                                                                     
True                                                                                                                                                     
Main.S3Bucket                                                                                                                                            
Main.S3Bucket                                                                                                                                            
False

Basically, it tries to serialize some simple JSON string into an object or type Main.S3Bucket, first by calling JsonSerializer.Deserialize<S3Bucket> directly, second by calling it through a registered instance of TypeConverter.

What I'm seeing is that TypeDescriptor.GetConverter(typeof(S3Bucket)).ConvertFrom(json) returns some kind of strange type.

It's named the same as the correct type (Main.S3Bucket, see line 4 in Captured Log Information in the output).

However, it's not the same type. It cannot be cast to Main.S3Bucket, and its GetType() is not equal to typeof(Main.S3Bucket)

I'm expecting TypeDescriptor.GetConverter(typeof(S3Bucket)).ConvertFrom(json) to return an object of the same type as JsonSerializer.Deserialize<S3Bucket>

This only happens when I call it in the test tool. If I try to make a unit test in XUnit for this setup, both TypeConverter and JsonSerializer.Deserialize return the same type.

Configuration

$ dotnet --version
3.1.404

$ systeminfo | findstr /B /C:"OS Name" /C:"OS Version" /C:"System Type"
OS Name:                   Microsoft Windows Server 2016 Datacenter
OS Version:                10.0.14393 N/A Build 14393
System Type:               x64-based PC

Regression?

Other information

layomia commented 3 years ago

@alex-bolenok-centralreach - I was able to repro this issue. From your sample, it looks like the instance returned from JsonSerializer.Deserialize is of the correct type, but the call to TypeConverter.ConvertFrom returns an instance of the wrong type. Given this, I'll label this issue as area-System.ComponentModel. cc @safern

Do you have any tips for running the repro with a .NET 5.0 TFM? I'm running into an error when running locally. Perhaps there's some incompatibility between .NET 5 and the Amazon.Lambda.TestTool-3.1 package (I couldn't find a 5.0-labeled package):

D:\console_apps\AWSRepro\bin\Release\net5.0>dotnet lambda-test-tool-3.1 --no-ui --function-handler AWSRepro::Main.Lambda::Handler --pause-exit false
AWS .NET Core 3.1 Mock Lambda Test Tool (0.10.1)
Loaded local Lambda runtime from project output D:\console_apps\AWSRepro\bin\Release\net5.0
Executing Lambda function without web interface
... Using function handler AWSRepro::Main.Lambda::Handler
... No payload configured. If a payload is required set the --payload switch to a file path or a JSON document.
... Warning: Profile default not found in the aws credential store.
... No default AWS region configured. The --region switch can be used to configure an AWS Region.
Captured Log information:

Request failed to execute
Error:
Failed to find type Main.Lambda
ghost commented 3 years ago

Tagging subscribers to this area: @safern See info in area-owners.md if you want to be subscribed.

Issue Details
### Description I'm running some code in [AWS Lambda Test Tool](https://github.com/aws/aws-lambda-dotnet/tree/master/Tools/LambdaTestTool) and running into an issue, when `System.Text.Json.JsonSerializer.Deserialize` returns different results depending on whether it's being called from the top level or from a `System.ComponentModel.TypeConverter` method. This is best illustrated with code: Lambda.csproj ```CSPROJ netcoreapp3.1 enable Main ``` Lambda.cs ```C# using System; using System.ComponentModel; using System.Globalization; using System.Text.Json; using Amazon.Lambda.Core; using Amazon.Lambda.SQSEvents; using JetBrains.Annotations; [assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))] namespace Main { public static class JsonOptions { public static readonly JsonSerializerOptions JsonSerializerOptions = new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase }; } internal class JsonTypeConverterAdapter : TypeConverter { public override object? ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, object value) => value is string stringValue ? JsonSerializer.Deserialize(stringValue, JsonOptions.JsonSerializerOptions) : base.ConvertFrom(context, culture, value); public override object? ConvertTo(ITypeDescriptorContext context, CultureInfo culture, object value, Type destinationType) => value is T tValue && destinationType == typeof(string) ? JsonSerializer.Serialize(tValue) : base.ConvertTo(context, culture, value, destinationType); public override bool CanConvertFrom(ITypeDescriptorContext context, Type sourceType) => sourceType == typeof(string) || base.CanConvertFrom(context, sourceType); public override bool CanConvertTo(ITypeDescriptorContext context, Type destinationType) => destinationType == typeof(T) || base.CanConvertTo(context, destinationType); } [TypeConverter(typeof(JsonTypeConverterAdapter))] [Serializable] internal class S3Bucket { public string Bucket { get; set; } = string.Empty; } internal class Lambda { [PublicAPI] public void Handler(SQSEvent sqsEvent, ILambdaContext lambdaContext) { const string json = "{\"bucket\":\"test\"}"; var s3BucketWorks = JsonSerializer.Deserialize(json, JsonOptions.JsonSerializerOptions); Console.WriteLine(s3BucketWorks.Bucket); Console.WriteLine((s3BucketWorks.GetType() == typeof(S3Bucket)).ToString()); var s3BucketFails = TypeDescriptor.GetConverter(typeof(S3Bucket)).ConvertFrom(json); if (s3BucketFails == null) { return; } Console.WriteLine(s3BucketFails.GetType()); Console.WriteLine(typeof(S3Bucket)); Console.WriteLine((s3BucketFails is S3Bucket).ToString()); } } } ``` Run: ``` dotnet tool install -g Amazon.Lambda.TestTool-3.1 dotnet build -c Debug cd bin\Debug\netcoreapp3.1 dotnet lambda-test-tool-3.1 --no-ui --function-handler Lambda::Main.Lambda::Handler --pause-exit false ``` Output: ``` AWS .NET Core 3.1 Mock Lambda Test Tool (0.10.1) Loaded local Lambda runtime from project output <...>\Lambda\bin/Debug/netcoreapp3.1 Executing Lambda function without web interface ... Using function handler Lambda::Main.Lambda::Handler ... No payload configured. If a payload is required set the --payload switch to a file path or a JSON document. ... Setting AWS_PROFILE environment variable to default. ... No default AWS region configured. The --region switch can be used to configure an AWS Region. Captured Log information: test True Main.S3Bucket Main.S3Bucket False ``` Basically, it tries to serialize some simple JSON string into an object or type `Main.S3Bucket`, first by calling `JsonSerializer.Deserialize` directly, second by calling it through a registered instance of `TypeConverter`. What I'm seeing is that `TypeDescriptor.GetConverter(typeof(S3Bucket)).ConvertFrom(json)` returns some kind of strange type. It's named the same as the correct type (`Main.S3Bucket`, see line 4 in Captured Log Information in the output). However, it's not the same type. It cannot be cast to `Main.S3Bucket`, and its `GetType()` is not equal to `typeof(Main.S3Bucket)` I'm expecting `TypeDescriptor.GetConverter(typeof(S3Bucket)).ConvertFrom(json)` to return an object of the same type as `JsonSerializer.Deserialize` This only happens when I call it in the test tool. If I try to make a unit test in XUnit for this setup, both `TypeConverter` and `JsonSerializer.Deserialize` return the same type. ### Configuration ``` $ dotnet --version 3.1.404 $ systeminfo | findstr /B /C:"OS Name" /C:"OS Version" /C:"System Type" OS Name: Microsoft Windows Server 2016 Datacenter OS Version: 10.0.14393 N/A Build 14393 System Type: x64-based PC ``` ### Regression? ### Other information
Author: alex-bolenok-centralreach
Assignees: -
Labels: `area-System.ComponentModel`, `untriaged`
Milestone: -
alex-bolenok-centralreach commented 3 years ago

The AWS Lambda Test Tool is supposed to emulate AWS Lambda runtime environments, and as of this moment, .NET 5 is not a supported runtime environment, they only support .NET Core 2.1 and 3.1

alex-bolenok-centralreach commented 1 year ago

This still reproduces with net6.0 and dotnet-lambda-test-tool-6.0

lambda-test-tool.zip

dotnet tool restore
dotnet build
dotnet dotnet-lambda-test-tool-6.0 --no-ui --function-handler Lambda::Main.Lambda::Handler --pause-exit false

Output:

AWS .NET Core 6.0 Mock Lambda Test Tool (0.12.7)
Loaded local Lambda runtime from project output /home/alex.bolenok/Projects/lambda-test-tool/bin/Debug/net6.0
Executing Lambda function without web interface
... Using function handler Lambda::Main.Lambda::Handler
... No payload configured. If a payload is required set the --payload switch to a file path or a JSON document.
... Setting AWS_PROFILE environment variable to default.
... No default AWS region configured. The --region switch can be used to configure an AWS Region.
Captured Log information:
test
True
Main.S3Bucket
Main.S3Bucket
>>>> False <<<<

Request executed successfully
Response:

I expected the marked False to be True.

ghost commented 1 year ago

This issue has been marked needs-author-action and may be missing some important information.

steveharter commented 1 year ago

I converted to a simple Console app and was not able to repro this on 3.1.32. The last Console.WriteLine() returned true for me, not false: Console.WriteLine((s3BucketFails is S3Bucket).ToString());

I also changed this line var s3BucketFails = TypeDescriptor.GetConverter(typeof(S3Bucket)).ConvertFrom(json); to S3Bucket s3BucketFails = (S3Bucket)TypeDescriptor.GetConverter(typeof(S3Bucket)).ConvertFrom(json); which didn't fail with a cast exception or other issue.

Closing as unable to repro and since this is 2.5 years old. Note that .NET 3.1 is no longer supported. Please use 6.0 (latest LTS) or 7.0. Thanks. Can you verify the issue occurs on a console app? If it still repros on 6.0+ and for a console app (or other simple repro), please re-open this issue.