Support lazy and explicit loading from entities queried with NoTracking behavior

[JoesGab]

Explicit loading of a Reference does not populate the navigation property when QueryTrackingBehavior.NoTracking is set and AsTracking() is used as an override.

Explicit loading of a Collection does work as expected though.

Steps to reproduce

Minimalistic example for various combinations included. The issue arises only in the last assert statement of the second using block.

Program.cs

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using Microsoft.EntityFrameworkCore;

namespace NoTrackingBug
{
    public class BloggingContext : DbContext
    {
        public DbSet<Blog> Blogs { get; set; }
        public DbSet<Post> Posts { get; set; }

        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder.UseSqlite("Data Source=blogging.db");
        }
    }

    public class Blog
    {
        public int BlogId { get; set; }
        public string Url { get; set; }

        public List<Post> Posts { get; set; }

        public int OwnerId { get; set; }
        public Person Owner { get; set; }
    }

    public class Post
    {
        public int PostId { get; set; }
        public string Title { get; set; }

        public int BlogId { get; set; }
        public Blog Blog { get; set; }
    }

    public class Person
    {
        public int PersonId { get; set; }
        public string Name { get; set; }

        public List<Blog> OwnedBlogs { get; set; }
    }

    class Program
    {
        static void Main(string[] args)
        {
            // Setup the example
            using (var db = new BloggingContext())
            {
                if (!db.Blogs.Any())
                {
                    db.Blogs.Add(new Blog
                    {
                        Url = "http://blogs.msdn.com/adonet",
                        Posts = new List<Post>{
                            new Post { Title = "SomeTitle" }
                        },
                        Owner = new Person
                        {
                            Name = "Kobert"
                        }
                    });

                    db.SaveChanges();
                }
            }

            // Explicit load and .NoTracking => does not work for References
            using (var db = new BloggingContext())
            {
                db.ChangeTracker.QueryTrackingBehavior = QueryTrackingBehavior.NoTracking;

                var blog = db.Blogs
                    .AsTracking()
                    .Single();

                db.Entry(blog)
                    .Collection(b => b.Posts)
                    .Load();

                db.Entry(blog)
                    .Reference(b => b.Owner)
                    .Load();

                Debug.Assert(blog.Posts != null);
                Debug.Assert(blog.Owner != null); // <== This will fail!
            }

            // Eager load and .NoTracking => does work
            using (var db = new BloggingContext())
            {
                db.ChangeTracker.QueryTrackingBehavior = QueryTrackingBehavior.NoTracking;

                var blog = db.Blogs
                    .Include(b => b.Posts)
                    .Include(b => b.Owner)
                    .AsTracking()
                    .Single();

                Debug.Assert (blog.Posts != null);
                Debug.Assert (blog.Owner != null);
            }

            // Explicit load and .TrackAll => does work
            using (var db = new BloggingContext())
            {
                var blog = db.Blogs
                    .Single();

                db.Entry(blog)
                    .Collection(b => b.Posts)
                    .Load();

                db.Entry(blog)
                    .Reference(b => b.Owner)
                    .Load();

                Debug.Assert(blog.Posts != null);
                Debug.Assert(blog.Owner != null);
            }
        }
    }
}

Further technical details

EF Core version: 1.1.0 and 2.0.0 Database Provider: Microsoft.EntityFrameworkCore.SqlServer and Micrisoft.EntityFrameworkCore.Sqlite Operating system: Windows 7 Enterprise IDE: Visual Studio 2017 Enterprise and Visual Studio Code

[ajcvickers]

Note for triage: behavior is to throw in all cases, as it was in 2.0. Implementation is a bit tricky because it requires running a no-tracking query where the root entity is already materialized but not tracked. Proposing we push this to post 2.1.

[ajcvickers]

Triage: Leaving this in the backlog for now. Please vote for this feature if you need it.

[Suriman]

Both the Lazy Loading feature and the AsNoTraking feature are fundamental for implementing applications that use the WebRule tool from https://codeeffects.com/.

This tool allows to evaluate expressions on graphs of objects where these expressions are defined in a declarative way, which provides a lot of power to the users of an application. You can see examples in the section: https://codeeffects.com/Business-Rule-Demo

The tool allows navigating through the graph of objects accessing properties and collections of objects and it is precisely in this feature that the EF Core and the support of Lazy Loading come into play. For WebRule it is transparent to access a property of an object that is itself a collection of other objects thanks to the Lazy Loading mechanism of EF Core. Since WebRule only requires access to the properties to read and evaluate them, it is not necessary for the data to be tracked, which greatly increases the performance if the Lazy Loading combined with the AsNoTraking is allowed.

Please consider giving priority to this issue as it is essential in many scenarios, including this one that we particularly use.

[Starkie]

@divega told us that if we wanted to try and solve it we should contact you on the issue, as it's important to our use case. Could you please guide us a bit on how could it be done?

Thanks in advance

[ajcvickers]

@Starkie Can you be more explicit about what you are asking?

[divega]

@ajcvickers I met @Starkie and @Suriman recently. They expressed a lot of interest on creating a PR to address this issue. They are asking for guidance and details on what makes the issue tricky to fix.

[Starkie]

Sorry for being so ambiguous @ajcvickers 😅

As Diego said, we'd like to try to solve this issue. He commented that you may have some ideas on how could this be done, and might be able to offer us some guidance.

Sorry again, and thanks for your time.

[ajcvickers]

@Starkie The difficult aspect of this is changing the query pipeline so it can handle performing a no-tracking query with fixup when the root entity has already been materialized. This is the tricky bit that will need input from query people like @smitpatel and @maumar. I can't really give any kind of deep guidance on this without doing more digging and experimenting.

[smitpatel]

Filed #12208 to have better support to do fixup in QueryBuffer so that it can be reused for scenario like above.

[divega]

@Starkie, @Suriman, just to close the loop on this: we identified #12208, which would be a pre-requisite for #10042. All in all, this issue is too complicated for a first PR. I would recommend trying something simpler first.

[Suriman]

@divega, thank you for the interest shown. We will try to think of a shortcut.

[ajcvickers]

See also #12780

[smitpatel]

We need to re-evaluate this. Query does not do any fixup other than eager loading.

[rosenbjerg]

Separating lazy-loading from change-tracking, perhaps still using a Castle proxy but a more lightweight one, could be useful for CQRS. This could help gain some performance, if the change-tracking part of the proxying is costly, but I suspect the overhead is almost exclusively the creation of the proxy instance?

[seekingtheoptimal]

Maybe I'm missing something, but it seems to me the issue ("attempt was made to lazy-load navigation property on detached entity") can exists also when not using AsNoTracking() at all. When throw is turned off for this, then the lazy loading just does not work, with or without Include() the navigation properties are null. Any idea when and why this could happen?

The current query where I noticed this, is way too complicated to start digging into that and couldn't find a consistent pattern (some corporate hackathon abomination I inherited sadly), but would be nice to know if there are any specific cases when such thing could happen. Because for now for some dbsets it works, and for some it doesn't, and I'm stuck on where to look.

[ajcvickers]

@seekingtheoptimal I'm not aware of any specific bugs we know of in this area.

[klepeis]

Any update on this?

We are designing a service implementing CQRS and I am leveraging Entity on our read side. I am leveraging an object that wraps my DbContext for my queries as shown below.

private readonly DbContext _dbContext;

public ReadOnlyContext(DbContext dbContext)
{
   _dbContext = dbContext;
}

public IQueryable<TEntity> Set<TEntity>() where TEntity : class
{
   return _dbContext.Set<TEntity>().AsNoTracking();
}

I'd like to be able to lazy load associated objects returned from my queries as needed.

Thanks!

[ajcvickers]

@klepeis This issue is in the Backlog milestone. This means that it is not planned for the next release (EF Core 5.0). We will re-assess the backlog following the this release and consider this item at that time. However, keep in mind that there are many other high priority features with which it will be competing for resources.

[klepeis]

Thanks @ajcvickers!

[dazbradbury]

@ajcvickers - Just wondering where this sits now EF Core 5.0 has been released?

We're running into this issue for our "Read Only" data context(s), where it's non-obvious that a read-only context wouldn't be able to lazy load via navigation properties.

In a way, it's nice that you have to force a single query/operation. But the fact it's a runtime exception makes it a bit of a "gotcha". Alternatively, if you know of an analyser that could highlight/flag things like this, would be a more than adequate solution.

[ajcvickers]

@dazbradbury Currently no change in status.

[gokhanabatay]

Hi @ajcvickers, I dont understand why LazyLoading only available with "AsTracking()" queries, we are trying to migrate ef 7 from NHibernate. This is major issue since must support LazyLoading but "TrackAll" will kill application performance and memory. Is there any workaround to support lazy loading with non trackable queries?

[ajcvickers]

@dazbradbury If tracking is really too slow, then only workaround I can think of is to inject the DbContext into your entities (or your own proxies) and implement the behavior there.

[gokhanabatay]

@ajcvickers could you show us custom proxy sample, replacing ef core default proxy generator is it possible? https://github.com/dotnet/efcore/issues/14554 , we have too much entity we cannot manuely add lazy initialization all of them. How to by pass EF Core LazyLoading exception for non tracking queries and allow lazy load?

[ajcvickers]

@gokhanabatay That's a fairly big job; I'm not sure I'm going to write a full example on this. If it's not something you feel like you can tackle, then probably best to wait for the feature to be implemented.

[MoazAlkharfan]

I think this made it so that proxies are being created and lazy loading is enabled even when ChangeTracker.LazyLoadingEnabled = false.

[ajcvickers]

@MoazAlkharfan Turning off lazy-loading does not disable proxy creation. Disabling proxy creation is tracked by #12895.