Exception when projecting from navigation properties after null coalescing operator in self-referencing relationship

[peterwurzinger]

Hi,

To simplify things I will treat the domain model for reproducing the issue as "our domain".

We have an entity Product with a self-referencing optional (!) relationship:

Product WelcomePackage { get; set; }

In a query we project properties from this entity, as well as ones from other related entities - e.g. Documents. For this projections we use some kind of fallback - if WelcomePackage is set, we treat that as the entity to project from, else we fall back to the selected Product. Basically let projectionProduct = (product.WelcomePackage ?? product)

When we try to project projectionProduct.Documents.Select(...)/FirstOrDefault().Id/... an exception is thrown: System.InvalidOperationException : When called from 'VisitLambda', rewriting a node of type 'System.Linq.Expressions.ParameterExpression' must return a non-null value of the same type. Alternatively, override 'VisitLambda' and change it to not visit children of this type. Stack traced included below.

Steps to reproduce

namespace NullCoalescingReproduction
{
    public class NullCoalescingTests : IDisposable
    {
        private readonly Context _context;

        public NullCoalescingTests()
        {
            _context = new Context();
            _context.Database.EnsureDeleted();
            _context.Database.EnsureCreated();
            Seed(_context);
        }

        [Fact]
        public void ShouldTranslateProjectionOnNullCoalescedEntity()
        {
            var query = from product in _context.Set<Product>()
                        //where product.Id == <Some key provided by a user interaction>
                        let welcomePackage = product.WelcomePackage
                        let productToShip = (welcomePackage ?? product)

                        let firstDocument = productToShip.Documents.FirstOrDefault()

                        select new
                        {
                            ProductId = productToShip.Id, //This works
                            EveryDocument = productToShip.Documents, //This works
                            FirstDocument = firstDocument, //This works

                            FirstDocumentId = firstDocument.Id //This does not
                        };

            var result = query.ToList();

            Assert.NotNull(result);
            Assert.NotEmpty(result);
        }

        private static void Seed(Context ctx)
        {
            var welcomePackage = new Product
            {
                Name = "A welcome package with Product A in it"
            };

            var product = new Product
            {
                Name = "Product A",
                WelcomePackage = welcomePackage
            };

            ctx.Set<Product>().AddRange(product, welcomePackage);
            ctx.SaveChanges();
        }

        public void Dispose()
        {
            //I know, that this is not the proper way of implementing IDisposable
            _context.Dispose();
        }
    }

    public class Context : DbContext
    {
        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            var connectionString = "Data Source=(localdb)\\MSSQLLocalDB;Initial Catalog=ReproduceNullCoalescingBug;Integrated Security=True;";
            optionsBuilder.UseSqlServer(connectionString);
        }

        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            modelBuilder.Entity<Product>();
            modelBuilder.Entity<Document>();
        }
    }

    public class Product
    {
        [Key]
        public Guid Id { get; set; }
        public string Name { get; set; }
        public ISet<Document> Documents { get; } = new HashSet<Document>();
        //Consider this being some kind of welcome-package with the ordered product and some additions in it
        public Product WelcomePackage { get; set; }
    }

    public class Document
    {
        [Key]
        public Guid Id { get; set; }

        [Required]
        public Product Product { get; set; }
    }
}

Exception:

 Message: 
    System.InvalidOperationException : When called from 'VisitLambda', rewriting a node of type 'System.Linq.Expressions.ParameterExpression' must return a non-null value of the same type. Alternatively, override 'VisitLambda' and change it to not visit children of this type.
  Stack Trace: 
    ExpressionVisitor.VisitAndConvert[T](T node, String callerName)
    ExpressionVisitorUtils.VisitParameters(ExpressionVisitor visitor, IParameterProvider nodes, String callerName)
    ExpressionVisitor.VisitLambda[T](Expression`1 node)
    Expression`1.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    RelationalProjectionBindingExpressionVisitor.Visit(Expression expression)
    ExpressionVisitorUtils.VisitArguments(ExpressionVisitor visitor, IArgumentProvider nodes)
    ExpressionVisitor.VisitMethodCall(MethodCallExpression node)
    MethodCallExpression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    RelationalProjectionBindingExpressionVisitor.Visit(Expression expression)
    ExpressionVisitorUtils.VisitArguments(ExpressionVisitor visitor, IArgumentProvider nodes)
    ExpressionVisitor.VisitMethodCall(MethodCallExpression node)
    MethodCallExpression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    RelationalProjectionBindingExpressionVisitor.Visit(Expression expression)
    ExpressionVisitor.VisitUnary(UnaryExpression node)
    UnaryExpression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    RelationalProjectionBindingExpressionVisitor.Visit(Expression expression)
    RelationalProjectionBindingExpressionVisitor.VisitNew(NewExpression newExpression)
    NewExpression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    RelationalProjectionBindingExpressionVisitor.Visit(Expression expression)
    RelationalProjectionBindingExpressionVisitor.Translate(SelectExpression selectExpression, Expression expression)
    RelationalQueryableMethodTranslatingExpressionVisitor.TranslateSelect(ShapedQueryExpression source, LambdaExpression selector)
    QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
    RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
    MethodCallExpression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
    Database.CompileQuery[TResult](Expression query, Boolean async)
    QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
    <>c__DisplayClass9_0`1.<Execute>b__0()
    CompiledQueryCache.GetOrAddQueryCore[TFunc](Object cacheKey, Func`1 compiler)
    CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
    QueryCompiler.Execute[TResult](Expression query)
    EntityQueryProvider.Execute[TResult](Expression expression)
    EntityQueryable`1.GetEnumerator()
    List`1.ctor(IEnumerable`1 collection)
    Enumerable.ToList[TSource](IEnumerable`1 source)
    NullCoalescingTests.ShouldTranslateProjectionOnNullCoalescedEntity() line 41

Further technical details

EF Core version: 3.1.2 Database provider: Microsoft.EntityFrameworkCore.SqlServer 3.1.2 Target framework: .NET Core 3.1 Operating system: Windows 10 x64 IDE: Visual Studio 2019 16.4

[ajcvickers]

@smitpatel @maumar Thoughts?

[smitpatel]

We cannot translate Coalesce over entityType to Sql. This particular form will always client eval. It could be written to have coalesce over properties in final projection so it can be evaluated on client. productToShip.Documents is never going to be Sql representable.

[peterwurzinger]

@smitpatel @ajcvickers I digged a little deeper into that, and I got a better understanding of why EF Core can't translate coalescing on an entity level. For the sake of completeness, with that understanding I rephrased the projection in the example above to:

FirstDocumentId = (welcomePackage.Documents.FirstOrDefault()
                ?? product.Documents.FirstOrDefault()).Id

It's not pretty, but it works.

What I want to elaborate now is another alternative of the projection, which also didn't work

FirstDocumentId = (product.WelcomePackage ?? product).Documents.FirstOrDefault().Id

Because this expression has quite an easy equivalent in SQL:

select FirstDocumentId = doc.Id
from Product product
left join Document doc
on doc.ProductId = coalesce(product.WelcomePackageId, product.Id)

The difference between those two expressions

product.Documents.FirstOrDefault().Id
//and
(product.WelcomePackage ?? product).Documents.FirstOrDefault().Id

is only represented in the join condition of the resulting SQL:

on doc.ProductId = product.Id
-- and
on doc.ProductId = coalesce(product.WelcomePackageId, product.Id)

I might be wrong, but I could think about this being not too hard to implement..?

[cjblomqvist]

Most likely related/dupe: #17782