JsonSerializationException when using $select OData queries on .Net 6

[mkulisic]

Assemblies affected 8.0.12

Describe the bug When using $select or the $expand query statements with OData to query a cosmosDB collection I receive the following exceptions: "Newtonsoft.Json.JsonSerializationException: Self referencing loop detected for property 'DeclaringType' with type 'Microsoft.OData.Edm.EdmEntityType'. Path 'TypedProperty.SchemaElements[0].DeclaredKey[0]'."

My use case is similar to the one described in this issue (now closed with no resolution):

• https://github.com/OData/AspNetCoreOData/issues/503
• I attempted to add ".AddODataNewtonsoftJson()" to my services to fix the issue but this only works when I return a local collection. If I attempt to query cosmosDB I hit the same problem (https://github.com/JamesNK/Newtonsoft.Json/issues/2649)

Reproduce steps I've attached my test code (minus crednetials) in the post.

Data Model Please share your Data model, for example, your C# class.

EDM (CSDL) Model Please share your Edm model, for example, CSDL file. You can send $metadata to get a CSDL XML content.

Request/Response Please share your request Uri, head or the request body Please share your response head, body.

Expected behavior Be able to run a query with a select statment. CosmosTest.zip

Screenshots Query I am attempting to run: Error Message:

Additional context I can see that the ".AddODataNewtonsoftJson()" method has an overload that takes parameters and that perhaps I could configure the ReferenceLoopHandling there to ignore this issue, but I can't find an example of how to use this anywhere. I've tried to add this setting in a couple of different places but didn't succeed.

[HolyChen]

I encountered same bug when I try to integrate CosmosDb with OData 8.0.12. '$filter' works correctly, but when query with '$select', it throws:

"Self referencing loop detected for property 'declaringType' with type 'Microsoft.OData.Edm.EdmEntityType'. Path 'typedProperty.schemaElements[0].declaredKey[0]'.",
"   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CheckForCircularReference(JsonWriter writer, Object value, JsonProperty property, JsonContract contract, JsonContainerContract containerContract, JsonProperty containerProperty)",
"   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues(JsonWriter writer, Object value, JsonContainerContract contract, JsonProperty member, JsonProperty property, JsonContract& memberContract, Object& memberValue)",
"   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)",
"   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)",
"   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)",
"   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)",
"   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)",
"   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)",
"   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)",
"   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)",
"   at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitConstant(ConstantExpression inputExpression, TranslationContext context)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitNonSubqueryScalarExpression(Expression expression, ReadOnlyCollection`1 parameters, TranslationContext context)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitScalarExpression(Expression expression, ReadOnlyCollection`1 parameters, TranslationContext context)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitMemberAccess(MemberExpression inputExpression, TranslationContext context)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitNonSubqueryScalarExpression(Expression expression, ReadOnlyCollection`1 parameters, TranslationContext context)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitScalarExpression(Expression expression, ReadOnlyCollection`1 parameters, TranslationContext context)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitMemberAssignment(MemberAssignment inputExpression, TranslationContext context)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitBindingList(ReadOnlyCollection`1 inputExpressionList, TranslationContext context)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitMemberInit(MemberInitExpression inputExpression, TranslationContext context)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitNonSubqueryScalarExpression(Expression expression, ReadOnlyCollection`1 parameters, TranslationContext context)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitScalarExpression(Expression expression, ReadOnlyCollection`1 parameters, TranslationContext context)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitSelect(ReadOnlyCollection`1 arguments, TranslationContext context)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.VisitMethodCall(MethodCallExpression inputExpression, TranslationContext context)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.Translate(Expression inputExpression, TranslationContext context)",
"   at Microsoft.Azure.Cosmos.Linq.ExpressionToSql.TranslateQuery(Expression inputExpression, IDictionary`2 parameters, CosmosLinqSerializerOptions linqSerializerOptions)",
"   at Microsoft.Azure.Cosmos.Linq.SqlTranslator.TranslateQuery(Expression inputExpression, CosmosLinqSerializerOptions linqSerializerOptions, IDictionary`2 parameters)",
"   at Microsoft.Azure.Cosmos.Linq.CosmosLinqQuery`1.CreateFeedIterator(Boolean isContinuationExpected)",
"   at Microsoft.Azure.Cosmos.Linq.CosmosLinqQuery`1.GetEnumerator()+MoveNext()",
"   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)",
"   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)",
"   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)",
"   at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)",
"   at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)",
"   at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)",
"   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultAsync>g__Logged|22_0(ResourceInvoker invoker, IActionResult result)",
      ......

My controller:

[ApiController]
public class MyController : ControllerBase
    [ODataAttributeRouting]
    [EnableQuery]
    [HttpGet("[action]")]
    [Produces("application/json")]
    public IActionResult GetSomething()
    {
        return Ok(cosmosDbContainer.GetItemLinqQueryable<MyType>(allowSynchronousQueryExecution: true));
    }
}

My service configuration:

var modelBuilder = new ODataConventionModelBuilder();
modelBuilder.EntitySet<MyType>("MyTypes");

services.AddControllers()
    .AddNewtonsoftJson()
    .AddOData(option =>
    {
        option.Select().Filter().OrderBy().Expand().Count().AddRouteComponents(
            "odata", modelBuilder.GetEdmModel());
    })
    .AddODataNewtonsoftJson();

I tried to dump the StringBuilder of the JsonWriter, it contains a lot of 'metadata' which are not expected to write:

[julealgon]

[ApiController]
public class MyController : ControllerBase
    [ODataAttributeRouting]
    [EnableQuery]
    [HttpGet("[action]")]
    [Produces("application/json")]
    public IActionResult GetSomething()
    {
        return Ok(cosmosDbContainer.GetItemLinqQueryable<MyType>(allowSynchronousQueryExecution: true));
    }
}

@HolyChen does it make any difference if you rewrite your controller method like this?

 [ApiController]
 public class MyController : ControllerBase
     [ODataAttributeRouting]
     [EnableQuery]
     [HttpGet("[action]")]
     [Produces("application/json")]
     public ActionResult<IQueryable<MyType>> GetSomething()
     {
         return cosmosDbContainer.GetItemLinqQueryable<MyType>(allowSynchronousQueryExecution: true);
     }
 }

Or just:

 [ApiController]
 public class MyController : ControllerBase
     [ODataAttributeRouting]
     [EnableQuery]
     [HttpGet("[action]")]
     [Produces("application/json")]
     public IQueryable<MyType> GetSomething()
     {
         return cosmosDbContainer.GetItemLinqQueryable<MyType>(allowSynchronousQueryExecution: true);
     }
 }

?

[mkulisic]

@julealgon I just attempted the solutions you suggested and got the same error (I think the first solution you provided is missing an Ok() in the return value). I also tried upgrading to .Net 7 and upgrading the version of asp.net I'm using but I still keep getting the same error. In the meantime I have been resorting to using an implementation of this based on the Entity framework. This implementation doesn't seem to have the same issue but it is more complex and I would rather stick with the client based approach if we can figure out what is wrong with it.

[HolyChen]

Thank you @julealgon , I tried your solution, but get same error. I find this error happens when reading data from CosmosDb and then serializing the data to Json string. The object to serialize contains tons of metadata, for example DeclaredKey in the error message, and the class schema of all regarding objects, such as MyType, classes referenced by MyType. I guess OData has should call a specified JsonConverter to handle the serialization, but actually not.

[julealgon]

@julealgon I just attempted the solutions you suggested and got the same error

To clarify, I wasn't necessarily saying those were solutions. Was just curious if those would affect the results in a meaningful way.

(I think the first solution you provided is missing an Ok() in the return value).

It is not. The way ActionResult<T> works is that it automatically converts from a T result, so you are supposed to return just the value without the Ok(...). Adding the OK actually changes the behavior since the underlying DeclaredType in the resulting object is not set, and this actually influences EnableQuery as it relies on this property to check the type of the result and then perform further operations.

If you cannot use just the value, it might be because your service is not returning it as IQueryable. Try changing that so it matches and it should automatically convert.

[mkulisic]

@julealgon is there anyone in particular we could pin on this ticket to try an get some help?

Thanks

[julealgon]

@julealgon is there anyone in particular we could pin on this ticket to try an get some help?

Thanks

@xuzhg can you direct this one to someone in the team to investigate?

[mkulisic]

@xuzhg is there an update on this problem?

[AndreiCsibi-msft]

@xuzhg : I work with @mkulisic. Any progress on this investigation? Let us know if you need more information from our side.

[parvsaxena]

I was facing the similar issue with my project

Self referencing loop detected for property 'declaringType' with type 'Microsoft.OData.Edm.EdmEntityType'. Path '[0].model.schemaElements[0].declaredKey[0]

and was setting the model like this modelBuilder.EntitySet<AssetClassificationAssociation>("Asset_Classification_Association");

and controller action like

[HttpGet]
[Route("assetclassificationassociation")]
public async Task<IActionResult> GetAssetClassificationAssociations(ODataQueryOptions<AssetClassificationAssociation> query, CancellationToken cancellationToken)
{
     // do and return something
}

However, what fixed it was making the route [Route("asset_classification_association")], ie same as the underlying view/table's name in database. This seems like a misleading error, given the issue possibly appears to be with some hard dependency between route and table name, when using the out of box routing for odata.

I later noticed during project startup, I was seeing this warning warn: Microsoft.AspNetCore.OData.Routing.Conventions.AttributeRoutingConvention[0] The path template 'datasets/assetclassificationassociation' on the action 'GetAssetClassificationAssociations' in controller 'DatasetsOData' is not a valid OData path template. Resource not found for the segment 'assetclassificationassociation'.

[julealgon]

@parvsaxena your route does need to match the entityset name: that's how it is supposed to work.

If you need to change the route, you also need to change the entityset name in the EDM setup.

[devbrsa]

Is there by any possibility an update on this?

[CSharpFiasco]

@xuzhg are you able to provide a timeline on this? I am getting this error in .NET 8. This work would provide a lot of value to teams, and it would be very much appreciated 🙏

[Isayaa]

@xuzhg any update? we are also having the same issue...