search

Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.46k stars 4.8k forks source link

[BUG] Serializing CloudEvent using Microsoft.Azure.Messaging.EventGrid.CloudNativeCloudEvents is not working as expected #28884

Closed mradulmodi closed 2 years ago

mradulmodi commented 2 years ago

Library name and version

Microsoft.Azure.Messaging.EventGrid.CloudNativeCloudEvents 1.0.0

Describe the bug

SendCloudEventsInternalAsync is not correctly serializing CloudNative.CloudEvents before sending event to Azure Event Grid.

Specifically this line (https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/eventgrid/Microsoft.Azure.Messaging.EventGrid.CloudNativeCloudEvents/src/EventGridPublisherClientExtensions.cs#L106) is not working as expected. The attributes under data in CloudEvent are being ignored. An incorrect Azure.Messaging.CloudEvent is being made and sent to Azure Event Grid

Expected behavior

Incoming CloudNative.CloudEvent

{
            "specVersion": "1.0",
            "data": {
                "requestId": "696824cc-50d7-4f23-93f3-06709fb0cf0a",
                "url": "https://fake.com",
                "tag": "faketag"
            },
            "dataContentType": "application/json",
            "id": "9aeb0fdf-c01e-0131-0922-9eb54906e209",
            "dataSchema": "http://fakeschema.com",
            "source": "com.fake,
            "subject": "42",
            "time": "2021-03-21T15:13:39.4589254+00:00",
            "type": "com.universe.answer.everything",
            "extensionAttributes": [],
            "isValid": true,
            "datacontenttype": "application/json"
        }

Should be converted correctly before being sent to Azure Event Grid. The Azure.Messaging.CloudEvent made should be 

{
            "specVersion": "1.0",
            "data": {
                "requestId": "696824cc-50d7-4f23-93f3-06709fb0cf0a",
                "url": "https://fake.com",
                "tag": "faketag"
            },
            "dataContentType": "application/json",
            "id": "9aeb0fdf-c01e-0131-0922-9eb54906e209",
            "dataSchema": "http://fakeschema.com",
            "source": "com.fake,
            "subject": "42",
            "time": "2021-03-21T15:13:39.4589254+00:00",
            "type": "com.universe.answer.everything",
            "extensionAttributes": [],
            "isValid": true,
            "datacontenttype": "application/json"
        }

Actual behavior

The above cloud event is being incorrectly converted into

{
            "specVersion": "1.0",
            "data": {
                "requestId": [],
                "url": [],
                "tag": []
            },
            "dataContentType": "application/json",
            "id": "9aeb0fdf-c01e-0131-0922-9eb54906e209",
            "dataSchema": "http://fakeschema.com",
            "source": "com.fake,
            "subject": "42",
            "time": "2021-03-21T15:13:39.4589254+00:00",
            "type": "com.universe.answer.everything",
            "extensionAttributes": [],
            "isValid": true,
            "datacontenttype": "application/json"
        }

Issue : Values specified under data attribute requestId, urland tagare not read

Reproduction Steps

Try sending sample cloud event to Azure Event Grid. Add a breakpoint at https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/eventgrid/Microsoft.Azure.Messaging.EventGrid.CloudNativeCloudEvents/src/EventGridPublisherClientExtensions.cs#L106 to inspect the result of conversion or inspect sent event to azure event grid

Environment

.NET SDK (reflecting any global.json):
 Version:   6.0.300
 Commit:    8473146e7d

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22000
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.300\

Host (useful for support):
  Version: 6.0.5
  Commit:  70ae3df4a6

.NET SDKs installed:
  3.1.419 [C:\Program Files\dotnet\sdk]
  5.0.214 [C:\Program Files\dotnet\sdk]
  5.0.408 [C:\Program Files\dotnet\sdk]
  6.0.202 [C:\Program Files\dotnet\sdk]
  6.0.203 [C:\Program Files\dotnet\sdk]
  6.0.300 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 3.1.25 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.25 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.25 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
azure-sdk commented 2 years ago

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'Event Grid:0.32042658,Event Hubs:0.10760262,Storage:0.107433304'

JoshLove-msft commented 2 years ago

@mradulmodi, this library using System.Text.Json to serialize the CloudEvent content. Using this library, your model properties would need to be public. Can you share the definition of your data model?

mradulmodi commented 2 years ago

Attribute Data in CloudNative.CloudEvent is an object and I am not typing before populating it. I am using default get & set properties. An example of how I am making Cloud event is below 


return new CloudEvent
            {
                Type = "com.universe.answer.everything",
                Source = new Uri("com.fake", UriKind.RelativeOrAbsolute),
                Id = Guid.NewGuid().ToString(),
                Time = DateTimeOffset.UtcNow,
                Subject = "42",
                DataSchema = new Uri("http://fakeschema.com"),
                Data = new
                {
                    requestId = Guid.NewGuid(),
                    url = "https://fake.com",
                    tag = "faketag"
                }
            };
JoshLove-msft commented 2 years ago

I just tested this out using an anonymous object for the Data property and everything is working as expected. Can you share your application code and your csproj so I can see what other dependencies you may be using?

mradulmodi commented 2 years ago

Sharing some application code. I am using Cosmos Change Feed Processor to listen to changes in Cosmos DB 

        /// <inheritdoc />
        public async override Task HandleChangesAsync(
            IReadOnlyCollection<CustomType> changes,
            CancellationToken cancellationToken)
        {
            try
            {
                foreach (var change in changes)
                {
                    this.logger.LogInformation($"Received event document with id - {change.Id}.");

                    await this.eventGridPublisherClient.SendCloudNativeCloudEventsAsync(change.EventData).ConfigureAwait(false);
                }
            }
            catch (Exception exception)
            {
                this.logger.LogError(exception, $"Processing failed.");
                throw;
            }
        }

Below is a snapshot of document in Cosmos Collection which this feed is listening to 

{
    "id": "cecb9ead-529a-4953-9ba6-b80cb78b6844",
    "eventData": [
        {
            "specVersion": "1.0",
            "data": {
                "requestId": "696814cc-50d7-4f23-93f3-06709fb0cf0a",
                "url": "https://som.system.net/api/v1.0/signals/4002-1",
                "tag": "\"00000000-0000-0000-65a0-2162e95101d8\""
            },
            "id": "9aeb0fdf-c01e-0131-0922-9eb54906e209",
            "dataSchema": "http://fakeschema.com",
            "source": "com.fake,
            "subject": "42",
            "time": "2021-03-21T15:13:39.4589254+00:00",
            "type": "com.universe.answer.everything",
            "extensionAttributes": [],
            "isValid": true,
            "datacontenttype": "application/json"
        }
    ],
    "_rid": "TzIJAOeOn8YZAAAAAAAAAA==",
    "_self": "dbs/TzIJAA==/colls/TzIJAOeOn8Y=/docs/TzIJAOeOn8YZAAAAAAAAAA==/",
    "_etag": "\"00000000-0000-0000-6bbd-7b6cc9bf01d8\"",
    "_attachments": "attachments/",
    "_ts": 1652991457
}

Sharing all external dependencies in my csporj

Change feed processor package has following external dependencies

<PackageReference Include="Azure.Identity" Version="1.5.0" />
    <PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.20.0" />
    <PackageReference Include="Microsoft.Azure.Cosmos" Version="3.26.2" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions" Version="4.0.1" />
    <PackageReference Include="Microsoft.Extensions.Hosting" Version="3.1.24" />

Processor also depends on another internal package which has these external dependencies

<ItemGroup>
        <PackageReference Include="AutoMapper" Version="10.1.1" />
        <PackageReference Include="Azure.Messaging.EventGrid" Version="4.10.0" />
        <PackageReference Include="CompareNETObjects" Version="4.73.0" />
        <PackageReference Include="EPPlus" Version="4.5.3.2" />
        <PackageReference Include="ExcelDataReader" Version="3.6.0" />
        <PackageReference Include="Microsoft.Azure.Messaging.EventGrid.CloudNativeCloudEvents" Version="1.0.0" />
        <PackageReference Include="Microsoft.Cloud.InstrumentationFramework.Metrics.Extensions" Version="2.2020.825.633" />
        <PackageReference Include="Microsoft.Cloud.InstrumentationFramework" Version="3.2.4.1" />
        <PackageReference Include="Microsoft.CodeAnalysis.BinSkim" Version="1.6.1" />
        <PackageReference Include="Microsoft.CodeAnalysis.FxCopAnalyzers" Version="2.6.1" />
        <PackageReference Include="Microsoft.FeatureManagement" Version="2.4.0" />
        <PackageReference Include="Newtonsoft.Json" Version="12.0.3" />
        <PackageReference Include="StyleCop.Analyzers" Version="1.0.2" />
        <PackageReference Include="System.Text.Encoding.CodePages" Version="4.7.1" />
        <PackageReference Include="System.Net.Security" Version="4.3.2" />
        <PackageReference Include="System.Text.RegularExpressions" Version="4.3.1" />
    </ItemGroup>
JoshLove-msft commented 2 years ago

Thanks, can you confirm that the EventData is populated correctly when it is being processed in HandleChangesAsync?

mradulmodi commented 2 years ago

I can confirm EventData is populated correctly when it is being processed in HandleChangesAsync. I have debugged this locally and the cloudEvent goes correctly till the line https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/eventgrid/Microsoft.Azure.Messaging.EventGrid.CloudNativeCloudEvents/src/EventGridPublisherClientExtensions.cs#L106

It's during EncodeStructuredModeMessage the attributes inside data are stripped of values and an incorrect document is read after it. 

ReadOnlyMemory<byte> bytes = s_eventFormatter.EncodeStructuredModeMessage(cloudEvent, out var `_);`
 using JsonDocument document = JsonDocument.Parse(bytes);
JoshLove-msft commented 2 years ago

Can you try adding an explicit reference to System.Text.Json in your project using a version of 5.0.2 or higher?

JoshLove-msft commented 2 years ago

My hunch is that somehow a lower version of STJ is being pulled in. I believe anonymous type support was added in 5.0

mradulmodi commented 2 years ago

I explicitly added reference to System.Text.Json to 6.0.4, these are my dependencies for service making the call. The problem still exists

image

JoshLove-msft commented 2 years ago

Can you include the full csproj file?

mradulmodi commented 2 years ago

This is the csproj for the service sending events to AEG

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <TargetFramework>netstandard2.0</TargetFramework>
    </PropertyGroup>
    <PropertyGroup>
        <CodeAnalysisRuleSet>..\..\Analyzers.ruleset</CodeAnalysisRuleSet>
    </PropertyGroup>
    <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'">
        <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
        <WarningsAsErrors />
    </PropertyGroup>
    <ItemGroup>
        <AdditionalFiles Include="..\..\stylecop.json">
            <Link>stylecop.json</Link>
        </AdditionalFiles>
    </ItemGroup>
    <ItemGroup>
        <PackageReference Include="AutoMapper" Version="10.1.1" />
        <PackageReference Include="Azure.Messaging.EventGrid" Version="4.10.0" />
        <PackageReference Include="<reference for an internal project>" />
        <PackageReference Include="CompareNETObjects" Version="4.73.0" />
        <PackageReference Include="EPPlus" Version="4.5.3.2" />
        <PackageReference Include="ExcelDataReader" Version="3.6.0" />
        <PackageReference Include="Microsoft.Azure.Messaging.EventGrid.CloudNativeCloudEvents" Version="1.0.0" />
        <PackageReference Include="Microsoft.Cloud.InstrumentationFramework.Metrics.Extensions" Version="2.2020.825.633" />
        <PackageReference Include="Microsoft.Cloud.InstrumentationFramework" Version="3.2.4.1" />
        <PackageReference Include="Microsoft.CodeAnalysis.BinSkim" Version="1.6.1" />
        <PackageReference Include="Microsoft.CodeAnalysis.FxCopAnalyzers" Version="2.6.1" />
        <PackageReference Include="Microsoft.FeatureManagement" Version="2.4.0" />
        <PackageReference Include="Newtonsoft.Json" Version="12.0.3" />
        <PackageReference Include="StyleCop.Analyzers" Version="1.0.2" />
        <PackageReference Include="System.Text.Encoding.CodePages" Version="4.7.1" />
        <PackageReference Include="System.Net.Security" Version="4.3.2" />
        <PackageReference Include="System.Text.Json" Version="6.0.4" />
        <PackageReference Include="System.Text.RegularExpressions" Version="4.3.1" />
    </ItemGroup>
    <ItemGroup>
        Internal Projects part of same solution
    </ItemGroup>

</Project>
JoshLove-msft commented 2 years ago

Chatted offline with @mradulmodi. The issue was that the Data in the CloudNative CloudEvent was a JObject, which could not be serialized using System.Text.Json.

As a workaround, he is going to serialize the CloudNative CloudEvent to bytes using NewtonSoft, and then call SendEncodedCloudEventsAsync. This means that the Azure.Messaging.EventGrid library can be used directly instead of going through the CloudNativeCloudEvents bridge library.