JsonSerializationException in Cosmos DB LINQ Provider when Using OData Queries

[devbrsa]

Description: A JsonSerializationException is occurring when using OData queries in the Azure Cosmos DB LINQ provider. This issue is specifically related to the detection of a self-referencing loop during JSON serialization, particularly within the DeclaringType property of type Microsoft.OData.Edm.EdmComplexType in the object structure.

Assemblies Affected:

• Microsoft.AspNetCore.OData 8.2.3
• Microsoft.Azure.Cosmos 3.35.4

Framework:

• dotnet 6

Dependencies:

<PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="6.0.22" />
<PackageReference Include="Microsoft.AspNetCore.OData" Version="8.2.3" />
<PackageReference Include="Microsoft.AspNetCore.OData.NewtonsoftJson" Version="8.2.0" />
<PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="7.0.5"/>
<PackageReference Include="Microsoft.Azure.Cosmos" Version="3.35.4" />

Issue Details: When executing OData queries against Azure Cosmos DB using the LINQ provider, a JsonSerializationException is thrown due to the detection of a self-referencing loop during JSON serialization. The issue appears to be related to the DeclaringType property of the Microsoft.OData.Edm.EdmComplexType within the object structure.

Steps to Reproduce: In the OData repo issue: https://github.com/OData/AspNetCoreOData/issues/1056

Expected Behavior: The expected behavior is to permit the projection of solely the chosen User or its internal properties while avoiding the JsonSerializationException.

Additional Information:

• The issue is causing a JsonSerializationException within the Cosmos DB LINQ provider.
• Trying to ignore the error leads to excessive memory consumption as the LINQ provider continues serializing the loop.
• Adjusting the MaxDepth property on JsonSerializerSettings to different values, such as 5, does not seem to have any effect. (Changing manually directly from the SDK source code).

Reproducible Code: Link to the original issue: https://github.com/OData/AspNetCoreOData/issues/1056

Stacktrace

 Newtonsoft.Json.JsonSerializationException: Self referencing loop detected for property 'DeclaringType' with type 'Microsoft.OData.Edm.EdmComplexType'. Path 'TypedProperty.SchemaElements[0].DeclaredProperties[0]'.      
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CheckForCircularReference(JsonWriter writer, Object value, JsonProperty property, JsonContract contract, JsonContainerContract containerContract, JsonProp
erty containerProperty)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues(JsonWriter writer, Object value, JsonContainerContract contract, JsonProperty member, JsonProperty property, JsonContract& memberC
ontract, Object& memberValue)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty 
containerProperty)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty con
tainerProperty)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProper
ty containerProperty)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty con
tainerProperty)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty 
containerProperty)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty con
tainerProperty)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProper
ty containerProperty)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty con
tainerProperty)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty 
containerProperty)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty con
tainerProperty)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty 
containerProperty)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty con
tainerProperty)
         at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
         at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
         at Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
         at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer)
         at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Type type, JsonSerializerSettings settings)
         at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, JsonSerializerSettings settings)
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitConstant(ConstantExpression inputExpression, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Linq\Expres
sionToSQL.cs:line 802
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitNonSubqueryScalarExpression(Expression inputExpression, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\
Linq\ExpressionToSQL.cs:line 263
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitNonSubqueryScalarExpression(Expression expression, ReadOnlyCollection`1 parameters, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3
\Microsoft.Azure.Cosmos\src\Linq\ExpressionToSQL.cs:line 1078
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitScalarExpression(Expression expression, ReadOnlyCollection`1 parameters, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.
Azure.Cosmos\src\Linq\ExpressionToSQL.cs:line 1538
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitScalarExpression(Expression expression, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Linq\ExpressionT
oSQL.cs:line 1452
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitMemberAccess(MemberExpression inputExpression, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Linq\Expr
essionToSQL.cs:line 831
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitNonSubqueryScalarExpression(Expression inputExpression, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\
Linq\ExpressionToSQL.cs:line 267
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitNonSubqueryScalarExpression(Expression expression, ReadOnlyCollection`1 parameters, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3
\Microsoft.Azure.Cosmos\src\Linq\ExpressionToSQL.cs:line 1078
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitScalarExpression(Expression expression, ReadOnlyCollection`1 parameters, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.
Azure.Cosmos\src\Linq\ExpressionToSQL.cs:line 1538
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitScalarExpression(Expression expression, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Linq\ExpressionT
oSQL.cs:line 1452
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitMemberAssignment(MemberAssignment inputExpression, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Linq\
ExpressionToSQL.cs:line 886
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitBinding(MemberBinding binding, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Linq\ExpressionToSQL.cs:l
ine 341
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitBindingList(ReadOnlyCollection`1 inputExpressionList, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Li
nq\ExpressionToSQL.cs:line 908
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitMemberInit(MemberInitExpression inputExpression, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Linq\Ex
pressionToSQL.cs:line 966
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitNonSubqueryScalarExpression(Expression inputExpression, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\
Linq\ExpressionToSQL.cs:line 276
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitNonSubqueryScalarExpression(Expression expression, ReadOnlyCollection`1 parameters, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3
\Microsoft.Azure.Cosmos\src\Linq\ExpressionToSQL.cs:line 1078
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitScalarExpression(Expression expression, ReadOnlyCollection`1 parameters, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.
Azure.Cosmos\src\Linq\ExpressionToSQL.cs:line 1538
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitScalarExpression(LambdaExpression lambda, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Linq\Expressio
nToSQL.cs:line 1434
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitSelect(ReadOnlyCollection`1 arguments, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Linq\ExpressionTo
SQL.cs:line 1663
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitMethodCall(MethodCallExpression inputExpression, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Linq\Ex
pressionToSQL.cs:line 1213
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.Translate(Expression inputExpression, TranslationContext context) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Linq\ExpressionToSQL.cs
:line 135
         at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.TranslateQuery(Expression inputExpression, IDictionary`2 parameters, CosmosLinqSerializerOptions linqSerializerOptions) in D:\repos\PoC\azure-cosmos-dotnet
-v3\Microsoft.Azure.Cosmos\src\Linq\ExpressionToSQL.cs:line 105
         at Microsoft.Azure.Cosmos.Linq.SqlTranslator.TranslateQuery(Expression inputExpression, CosmosLinqSerializerOptions linqSerializerOptions, IDictionary`2 parameters) in D:\repos\PoC\azure-cosmos-dotnet-v
3\Microsoft.Azure.Cosmos\src\Linq\SQLTranslator.cs:line 51
         at Microsoft.Azure.Cosmos.Linq.DocumentQueryEvaluator.HandleMethodCallExpression(MethodCallExpression expression, IDictionary`2 parameters, CosmosLinqSerializerOptions linqSerializerOptions) in D:\repos
\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Linq\DocumentQueryEvaluator.cs:line 96
         at Microsoft.Azure.Cosmos.Linq.DocumentQueryEvaluator.Evaluate(Expression expression, CosmosLinqSerializerOptions linqSerializerOptions, IDictionary`2 parameters) in D:\repos\PoC\azure-cosmos-dotnet-v3\
Microsoft.Azure.Cosmos\src\Linq\DocumentQueryEvaluator.cs:line 31
         at Microsoft.Azure.Cosmos.Linq.CosmosLinqQuery`1.CreateFeedIterator(Boolean isContinuationExpected) in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Linq\CosmosLinqQuery.cs:line 219    
         at Microsoft.Azure.Cosmos.Linq.CosmosLinqQuery`1.GetEnumerator()+MoveNext() in D:\repos\PoC\azure-cosmos-dotnet-v3\Microsoft.Azure.Cosmos\src\Linq\CosmosLinqQuery.cs:line 111
         at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSetSerializer.WriteResourceSetAsync(IEnumerable enumerable, IEdmTypeReference resourceSetType, ODataWriter writer, ODataSerializerContext writeContex
t)
         at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSetSerializer.WriteObjectInlineAsync(Object graph, IEdmTypeReference expectedType, ODataWriter writer, ODataSerializerContext writeContext)
         at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSetSerializer.WriteObjectAsync(Object graph, Type type, ODataMessageWriter messageWriter, ODataSerializerContext writeContext)
         at Microsoft.AspNetCore.OData.Formatter.ODataOutputFormatterHelper.WriteToStreamAsync(Type type, Object value, IEdmModel model, ODataVersion version, Uri baseAddress, MediaTypeHeaderValue contentType, HttpRequest req
uest, IHeaderDictionary requestHeaders, IODataSerializerProvider serializerProvider)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isC
ompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
         at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

[sourabh1007]

You can use custom serializer https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.cosmosclientoptions.serializer?view=azure-dotnet in .Net SDK. May be that would help you.

[devbrsa]

@sourabh1007 Please refer to https://github.com/Azure/azure-cosmos-dotnet-v3/blob/9f8d84860d3170d8a95b278a51ac2792a927a1ab/Microsoft.Azure.Cosmos/src/Linq/ExpressionToSQL.cs#L778C60-L778C60 It remains set to "Newtonsoft" when using JsonConvert.SerializeObject, even though altering the serializer won't resolve the problem; it's merely a temporary measure to bypass the loop, if necessary.

Stacktrace

Newtonsoft.Json.JsonSerializationException: Self referencing loop detected for property 'DeclaringType' with type 'Microsoft.OData.Edm.EdmComplexType'. Path 'TypedProperty.SchemaElements[0].DeclaredProperties[0]'.

[devbrsa]

If you're looking for a temporary solution, consider converting the OData syntax tree expression into a CosmosDb query. This can then be used alongside the CosmosDb SDK's QueryDefinition.

• Gist Link 1 - CosmosDbQueryBuilder.cs
• Gist Link 2 - ODataToCosmosDbVisitor.cs