$select-ing non-key properties inside $expand of contained entity with odata.metadata=full fails despite being autoselected

[Xriuk]

Assemblies affected ASP.NET Core OData 8.2.4

Describe the bug $expanding a contained navigation and $selecting other properties except the keys throws:

Microsoft.OData.ODataException: The entity instance value of type '...' doesn't have a value for property '...'. To compute an entity's metadata, its key and concurrency-token property values must be provided.

Reproduce steps A GET request to:

https://.../ContactForms?$expand=Emails($select=Locale)

with header:

Accept application/json;odata.metadata=full

Throws:

Microsoft.OData.ODataException: The entity instance value of type 'Models.Email.ContactFormEmail' doesn't have a value for property 'Id'. To compute an entity's metadata, its key and concurrency-token property values must be provided.
12:22:13:466       at Microsoft.OData.Evaluation.ODataResourceMetadataContext.TryGetPrimitiveOrEnumPropertyValue(ODataResourceBase resource, String propertyName, String entityTypeName, Boolean isRequired, Object& value)
12:22:13:466       at Microsoft.OData.Evaluation.ODataResourceMetadataContext.GetPropertyValues(IEnumerable`1 properties, ODataResourceBase resource, IEdmEntityType actualEntityType, Boolean isRequired)+MoveNext()
12:22:13:466       at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
12:22:13:466       at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)
12:22:13:466       at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
12:22:13:466       at Microsoft.OData.Evaluation.ODataResourceMetadataContext.ODataResourceMetadataContextWithModel.get_KeyProperties()
12:22:13:466       at Microsoft.OData.Evaluation.ODataConventionalIdMetadataBuilder.ComputeIdForContainment()
12:22:13:466       at Microsoft.OData.Evaluation.ODataConventionalIdMetadataBuilder.ComputeAndCacheId()
12:22:13:466       at Microsoft.OData.Evaluation.ODataConventionalIdMetadataBuilder.get_ComputedId()
12:22:13:466       at Microsoft.OData.Evaluation.ODataConventionalIdMetadataBuilder.GetId()
12:22:13:466       at Microsoft.OData.Evaluation.ODataConventionalEntityMetadataBuilder.TryGetIdForSerialization(Uri& id)
12:22:13:466       at Microsoft.OData.JsonLight.ODataJsonLightResourceSerializer.WriteResourceStartMetadataPropertiesAsync(IODataJsonLightWriterResourceState resourceState)
12:22:13:466       at Microsoft.OData.JsonLight.ODataJsonLightWriter.StartResourceAsync(ODataResource resource)
12:22:13:466       at Microsoft.OData.ODataWriterCore.<>c.<<WriteStartResourceImplementationAsync>b__196_0>d.MoveNext()
12:22:13:466    --- End of stack trace from previous location ---
12:22:13:466       at Microsoft.OData.ODataWriterCore.InterceptExceptionAsync[TArg0](Func`3 action, TArg0 arg0)
12:22:13:466       at Microsoft.OData.ODataWriterCore.WriteStartResourceImplementationAsync(ODataResource resource)
12:22:13:466       at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSerializer.WriteResourceAsync(Object graph, ODataWriter writer, ODataSerializerContext writeContext, IEdmTypeReference expectedType)
12:22:13:466       at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSerializer.WriteObjectInlineAsync(Object graph, IEdmTypeReference expectedType, ODataWriter writer, ODataSerializerContext writeContext)
12:22:13:466       at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSetSerializer.WriteResourceSetItemAsync(Object item, IEdmStructuredTypeReference elementType, Boolean isUntypedCollection, IEdmTypeReference resourceSetType, ODataWriter writer, IODataEdmTypeSerializer resourceSerializer, ODataSerializerContext writeContext)
12:22:13:466       at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSetSerializer.WriteResourceSetAsync(IEnumerable enumerable, IEdmTypeReference resourceSetType, ODataWriter writer, ODataSerializerContext writeContext)
12:22:13:466       at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSetSerializer.WriteObjectInlineAsync(Object graph, IEdmTypeReference expectedType, ODataWriter writer, ODataSerializerContext writeContext)
12:22:13:466       at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSerializer.WriteComplexAndExpandedNavigationPropertyAsync(IEdmProperty edmProperty, SelectItem selectItem, ResourceContext resourceContext, ODataWriter writer)
12:22:13:466       at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSerializer.WriteExpandedNavigationPropertiesAsync(SelectExpandNode selectExpandNode, ResourceContext resourceContext, ODataWriter writer)
12:22:13:466       at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSerializer.WriteResourceAsync(Object graph, ODataWriter writer, ODataSerializerContext writeContext, IEdmTypeReference expectedType)
12:22:13:466       at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSerializer.WriteObjectInlineAsync(Object graph, IEdmTypeReference expectedType, ODataWriter writer, ODataSerializerContext writeContext)
12:22:13:466       at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSetSerializer.WriteResourceSetItemAsync(Object item, IEdmStructuredTypeReference elementType, Boolean isUntypedCollection, IEdmTypeReference resourceSetType, ODataWriter writer, IODataEdmTypeSerializer resourceSerializer, ODataSerializerContext writeContext)
12:22:13:466       at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSetSerializer.WriteResourceSetAsync(IEnumerable enumerable, IEdmTypeReference resourceSetType, ODataWriter writer, ODataSerializerContext writeContext)
12:22:13:466       at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSetSerializer.WriteObjectInlineAsync(Object graph, IEdmTypeReference expectedType, ODataWriter writer, ODataSerializerContext writeContext)
12:22:13:466       at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataResourceSetSerializer.WriteObjectAsync(Object graph, Type type, ODataMessageWriter messageWriter, ODataSerializerContext writeContext)
12:22:13:466       at Microsoft.AspNetCore.OData.Formatter.ODataOutputFormatterHelper.WriteToStreamAsync(Type type, Object value, IEdmModel model, ODataVersion version, Uri baseAddress, MediaTypeHeaderValue contentType, HttpRequest request, IHeaderDictionary requestHeaders, IODataSerializerProvider serializerProvider)
12:22:13:466       at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
12:22:13:466       at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
12:22:13:466       at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
12:22:13:466       at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
12:22:13:466    --- End of stack trace from previous location ---
12:22:13:466       at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
12:22:13:466       at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
12:22:13:466       at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
12:22:13:466       at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
12:22:13:466       at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
12:22:13:466       at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
12:22:13:466       at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
12:22:13:466       at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<<UseMiddlewareInterface>b__1>d.MoveNext()
12:22:13:466    --- End of stack trace from previous location ---
12:22:13:466       at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<<UseMiddlewareInterface>b__1>d.MoveNext()
12:22:13:466    --- End of stack trace from previous location ---
12:22:13:466       at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
12:22:13:466       at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
12:22:13:466       at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
12:22:13:466       at Microsoft.AspNetCore.OData.Routing.ODataRouteDebugMiddleware.Invoke(HttpContext context)
12:22:13:466       at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
12:22:13:466       at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

Data Model

public class Language{
    public string LanguageCode { get; set; }
}

public class ContactFormEmail {
    public int Id { get; set; }

    ...

    public Language? Locale { get; set; }

    // Reverse navigation
    public ContactForm Form { get; set; }
}

public class ContactForm {
    public int Id { get; set; }

    ...

    public ICollection<ContactFormEmail> Emails { get; set; }
}

EDM (CSDL) Model

...
<EntityType Name="ContactForm">
    <Key>
        <PropertyRef Name="Id" />
    </Key>
    <Property Name="Id" Type="Edm.Int32" Nullable="false" />
    ...
    <NavigationProperty Name="Emails" Type="Collection(Models.Email.ContactFormEmail)" ContainsTarget="true" />
</EntityType>
<EntityType Name="ContactFormEmail">
    <Key>
        <PropertyRef Name="Id" />
    </Key>
    <Property Name="Id" Type="Edm.Int32" Nullable="false" />
    ...
    <Property Name="Locale" Type="Models.Language" />
    <NavigationProperty Name="Form" Type="Models.Email.ContactForm" Nullable="false" />
</EntityType>
<ComplexType Name="Language">
    <Property Name="LanguageCode" Type="Edm.String" Nullable="false" />
</ComplexType>
...
<EntitySet Name="ContactForms" EntityType="Models.Email.ContactForm">
    <NavigationPropertyBinding Path="Emails/Form" Target="ContactForms" />
    ...
</EntitySet>
...

Additional context From the resulting query it looks like the key properties are autoselected as they should but they are not being used.

DbSet<ContactForm>()
    .AsSplitQuery()
    .AsNoTrackingWithIdentityResolution()
    .Select($it => new SelectAllAndExpand<ContactForm>{ 
        Model = TypedLinqParameterContainer<IEdmModel>.TypedProperty, 
        Instance = $it, 
        UseInstanceForProperties = True, 
        Container = new NamedProperty<IEnumerable<SelectSome<ContactFormEmail>>>{ 
            Name = "Emails", 
            Value = $it.Emails
                .Select($it => new SelectSome<ContactFormEmail>{ 
                    Model = TypedLinqParameterContainer<IEdmModel>.TypedProperty, 
                    Container = new SingleExpandedPropertyWithNext0<SelectAll<Language>>{ 
                        Name = "Locale", 
                        Value = new SelectAll<Language>{ 
                            Model = TypedLinqParameterContainer<IEdmModel>.TypedProperty, 
                            Instance = $it.Locale, 
                            UseInstanceForProperties = True 
                        }
                        , 
                        Next0 = new AutoSelectedNamedProperty<int?>{ 
                            Name = "Id", 
                            Value = (int?)$it.Id 
                        }
                        , 
                        IsNull = $it.Locale == null 
                    }

                }
                ) 
        }

    }
    )

If I explicitly select the key, the results are correctly returned

https://.../ContactForms?$expand=Emails($select=Id,Locale)

[WanjohiSammy]

@Xriuk

I have performed a few tests to replicate this issue. Using TripPin/TripPinServiceRW/$metadata) service:

1. Query People, expand Friends and select other friends FirstName other than Id/Key. This is giving me the correct data:

GET https://services.odata.org/V4/(S(1ap4mtbir3rypxuywlctvast))/TripPinServiceRW/People?$expand=Friends($select=FirstName)

2. Query People, expand Photo and select photo name. This is returning the expected result:

GET https://services.odata.org/V4/(S(1ap4mtbir3rypxuywlctvast))/TripPinServiceRW/People?$expand=Photo($select=Name)

The query that is not working is when, for example, trying to select property (other than key/id) of navigation property that has its "ContainsTarget" attribute set to true. Example of TripPin service endpoint that will cause the above exception is:

GET https://services.odata.org/V4/(S(1ap4mtbir3rypxuywlctvast))/TripPinServiceRW/People?$expand=Trips($select=Name)

Header: Accept application/json;odata.metadata=full

However, the above query will give me the correct data when used with header Accept application/json;odata.metadata=minimal

For example:

GET https://services.odata.org/V4/(S(1ap4mtbir3rypxuywlctvast))/TripPinServiceRW/People?$expand=Trips($select=Name)

Header: Accept application/json;odata.metadata=minimal

I am still looking into this more

[Xriuk]

@WanjohiSammy yes, that is correct, as I wrote above, this only happens for contained entities (navigation properties with ContainsTarget="true"), and only with odata.metadata=full. So this has something to do with incorrectly generating annotations (@odata.id in this case) for contained entities, when their keys are not explicitly selected (despite still being autoselected).