Support filtered Include

[0xdeafcafe]

We keep this issue to track specifying filters inline when you specify eager loading in a query. However there are many scenarios that can be satisfied with global query filters:

Please learn about global query filters

We are seeing that a large portion of the scenarios customers want this feature for can already be better addressed using global query filters. Please try to understand that feature before you add your vote to this issue.

We are keeping this issue open only to represent the ability to specify filters on Include on a per-query basis, which we understand can be convenient on some cases.

Original issue:

Doing a .Where() inside a .Include() is allowed (by the syntax, it fails in execution). However, if you then follow that up with a .ThenInclude(), like so:

.Include(t => t.Ratings.Where(r => !r.IsDeleted))
.ThenInclude(r => r.User)

You get the following error:

'IEnumerable' does not contain a definition for 'User' and no extension method 'User' accepting a first argument of type 'IEnumerable' could be found (are you missing a using directive or an assembly reference?)

Is this something you're aware of and going to allow in a future version, or am I just doing something wrong and It's already supported?

[hidegh]

This should do the trick (any additional where clausule can be added to the joins/Where), but... Need to check out: http://www.thinqlinq.com/Post.aspx/Title/Left-Outer-Joins-in-LINQ-with-Entity-Framework - since depending how the model is set up and how you mix lambda and linq - the resulting query might do a bit of surprise (like in some cases: matching 2 table rows where ID is null)...

            var q =
                from od in dbContext.Set<Document>()
                from p in dbContext.Set<Packet>().Where(i => i.PacketId == od.Packet.PacketId)
                from c in dbContext.Set<Company>().Where(i => i.CompanyId == p.Company.CompanyId)
                select new DocumentMetadataDto()
                {

[anpete]

Includes can now be filtered at the model-level (see b1379b10)

[Bartmax]

@anpete can you give us more insight? I don't see the Include(x=> x.Where(y => y.Something)) combination (or by any other syntax that supposed to address this issue) on that PR (at least not on tests). What model-level means in this context ?

off topic: the Select(x => x is Kiwi) is beautiful 👍

[anpete]

Model-level means you specify the filters on the model (usually in OnModelCreating). E.g.

modelBuilder.Entity<Customer>().HasFilter(c => !c.IsDeleted);
modelBuilder.Entity<Order>().HasFilter(o => o.CustomerId == this._tenantId);

So, this is not ad-hoc Include filtering, but does enable things like soft-delete and multi-tenancy etc.

[Bartmax]

as long as one would be able to override that filtering on per query (aka showing soft-deleted records on admin dashboard) it's fine for me 👍 thanks!

[anpete]

https://github.com/aspnet/EntityFramework/commit/b1379b10fba013bcb70b60250e39eae8e895a6c3#diff-05dcdf754d61d5e939e36c51beab1374R99

[urig]

This looks like a big step forward. Thank you! Do I understand correctly this is only for EFCore?

[anpete]

Yep, Core only.

[gdoron]

Wow a great improvement @anpete kudos! I didn't think it will be done any time soon.

Do you have an estimation when filler includes will be available not just in the model level?

[johnkwaters]

Nice! Any chance you would support interfaces as well as types? All of our multitenant entities have an ITenant, all of our soft deletables have an ISoftDelete...

[anpete]

@gdoron Nothing at the moment, sorry. @johnkwaters Do you mean modelBuilder.Entity<ITenant>()? If so, we don't, but it should be very easy to write a helper method that loops over all entities in the model and applies to filter to those that implement the interface.

[johnkwaters]

Could you sketch the pseudo code for that? Have tried it in the past and not had much luck expressing it!

John Waters, MVP

CTO4Hire LLC

c: 831.295.3218

http://www.cto4hire.net www.cto4hire.net

From: Andrew Peters [mailto:notifications@github.com] Sent: Monday, May 8, 2017 1:47 PM To: aspnet/EntityFramework EntityFramework@noreply.github.com Cc: John Waters john.waters@cto4hire.net; Mention mention@noreply.github.com Subject: Re: [aspnet/EntityFramework] Support filtered Include (#1833)

@gdoron https://github.com/gdoron Nothing at the moment, sorry. @johnkwaters https://github.com/johnkwaters Do you mean modelBuilder.Entity()? If so, we don't, but it should be very easy to write a helper method that loops over all entities in the model and applies to filter to those that implement the interface?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aspnet/EntityFramework/issues/1833#issuecomment-299986017 , or mute the thread https://github.com/notifications/unsubscribe-auth/ADQ5rWYWhbEYo0sJ0k0PaDYz6rHBMVjQks5r339MgaJpZM4DvDZS . https://github.com/notifications/beacon/ADQ5rSTEtpzkBLsP5yjMf_zhdvCK3uDBks5r339MgaJpZM4DvDZS.gif

[anpete]

@johnkwaters It is not as easy as it could be but this works:

public interface IFilter
{
    string CustomerID { get; set; }
}

foreach (var entityType
    in modelBuilder.Model.GetEntityTypes()
         .Where(et => typeof(IFilter).IsAssignableFrom(et.ClrType)))
{
    var entityParameter = Expression.Parameter(entityType.ClrType, "e");

    var filter
        = Expression.Lambda(
            Expression.Equal(
                Expression.Property(entityParameter, "CustomerID"),
        Expression.Constant("ALFKI")),
             entityParameter);

    modelBuilder.Entity(entityType.ClrType).HasQueryFilter(filter);
}

[johnkwaters]

Thanks a ton!

John Waters, MVP

CTO4Hire LLC

c: 831.295.3218

http://www.cto4hire.net www.cto4hire.net

From: Andrew Peters [mailto:notifications@github.com] Sent: Monday, May 8, 2017 3:23 PM To: aspnet/EntityFramework EntityFramework@noreply.github.com Cc: John Waters john.waters@cto4hire.net; Mention mention@noreply.github.com Subject: Re: [aspnet/EntityFramework] Support filtered Include (#1833)

@johnkwaters https://github.com/johnkwaters It is not as easy as it could be but this works:

public interface IFilter { string CustomerID { get; set; } }

foreach (var entityType in modelBuilder.Model.GetEntityTypes() .Where(et => typeof(IFilter).IsAssignableFrom(et.ClrType))) { var entityParameter = Expression.Parameter(entityType.ClrType, "e");

var filter
    = Expression.Lambda(
        Expression.Equal(
            Expression.Property(entityParameter, "CustomerID"),
           Expression.Constant("ALFKI")),
         entityParameter);

modelBuilder.Entity(entityType.ClrType).HasQueryFilter(filter);

}

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aspnet/EntityFramework/issues/1833#issuecomment-300007854 , or mute the thread https://github.com/notifications/unsubscribe-auth/ADQ5rbHzavO3y1Iaorsbdf69Fe-L17QBks5r35XegaJpZM4DvDZS . https://github.com/notifications/beacon/ADQ5rf6Z80sFrl6ImIJWCtLoeZoPRhEsks5r35XegaJpZM4DvDZS.gif

[jnm2]

Here's a little helper method.

Usage:

modelBuilder.AddQueryFilterToAll<IFilter>(_ => _.CustomerID == "ALFKI");

Definition:

public static class TemporaryUtils
{
    public static void AddQueryFilterToAll<T>(this ModelBuilder modelBuilder, Expression<Func<T, bool>> predicate) where T : class
    {
        if (modelBuilder == null) throw new ArgumentNullException(nameof(modelBuilder));
        if (predicate == null) throw new ArgumentNullException(nameof(predicate));

        foreach (var entityType
            in modelBuilder.Model.GetEntityTypes()
                .Where(et => typeof(T).IsAssignableFrom(et.ClrType)))
        {
            var entityParameter = Expression.Parameter(entityType.ClrType);

            modelBuilder.Entity(entityType.ClrType).HasQueryFilter(
                Expression.Lambda(
                    new ExpressionReplacer(predicate.Parameters[0], entityParameter).Visit(predicate.Body),
                    entityParameter));
        }
    }

    private sealed class ExpressionReplacer : ExpressionVisitor
    {
        private readonly Expression find;
        private readonly Expression replace;

        public ExpressionReplacer(Expression find, Expression replace)
        {
            this.find = find;
            this.replace = replace;
        }

        public override Expression Visit(Expression node)
        {
            if (node == find) return replace;
            return base.Visit(node);
        }
    }
}

[im1dermike]

@anpete I'm using EF Core 1.1.0. How do I get access to HasFilter()?

[jnm2]

@im1dermike Since the HasQueryFilter code was only merged 8 days ago, I tested against the nightlies (https://dotnet.myget.org/feed/aspnetcore-dev/package/nuget/Microsoft.EntityFrameworkCore.SqlServer).

[johnkwaters]

Nice!

John Waters, MVP

CTO4Hire LLC

c: 831.295.3218

http://www.cto4hire.net www.cto4hire.net

From: Joseph Musser [mailto:notifications@github.com] Sent: Monday, May 8, 2017 4:21 PM To: aspnet/EntityFramework EntityFramework@noreply.github.com Cc: John Waters john.waters@cto4hire.net; Mention mention@noreply.github.com Subject: Re: [aspnet/EntityFramework] Support filtered Include (#1833)

Here's a little helper method.

Usage:

modelBuilder.AddQueryFilterToAll( => .CustomerID == "ALFKI");

Definition:

public static class TemporaryUtils { public static void AddQueryFilterToAll(this ModelBuilder modelBuilder, Expression<Func<T, bool>> predicate) where T : class { if (modelBuilder == null) throw new ArgumentNullException(nameof(modelBuilder)); if (predicate == null) throw new ArgumentNullException(nameof(predicate));

    foreach (var entityType
        in modelBuilder.Model.GetEntityTypes()
                .Where(et => typeof(T).IsAssignableFrom(et.ClrType)))
    {
        var entityParameter = Expression.Parameter(entityType.ClrType);

        modelBuilder.Entity(entityType.ClrType).HasQueryFilter(
            Expression.Lambda(
                new ExpressionReplacer(predicate.Parameters[0], entityParameter).Visit(predicate.Body),
                entityParameter));
    }
}

private sealed class ExpressionReplacer : ExpressionVisitor
{
    private readonly Expression find;
    private readonly Expression replace;

    public ExpressionReplacer(Expression find, Expression replace)
    {
        this.find = find;
        this.replace = replace;
    }

    public override Expression Visit(Expression node)
    {
        if (node == find) return replace;
        return base.Visit(node);
    }
}

}

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aspnet/EntityFramework/issues/1833#issuecomment-300017675 , or mute the thread https://github.com/notifications/unsubscribe-auth/ADQ5rXayjIiu4r1sF1TadyLqfeggo2itks5r36NTgaJpZM4DvDZS . https://github.com/notifications/beacon/ADQ5rXn70LQZW98lklC_UNhKkcOaAGoRks5r36NTgaJpZM4DvDZS.gif

[ajbeaven]

Where is "ad-hoc" Include filtering on the roadmap now? I read here under High priority features:

Filtered loading allows a subset of related entities to be loaded. This has been addressed by global query filters in EF Core 2.0 Preview 1

Global query filters refer to the model-level filtering mentioned above does it? I can understand that its addition has dampened enthusiasm for this feature but there are still use cases for Include filtering that don't make sense on a model level. I'm just after clarification here - is this still high priority?

[dodegaard]

I have now encountered a .Include() that crashes a C# console app when the amount of data that an include brings with it exceeds a certain threshold. I don't know the exact level of data but this illustrates why a filtered include would be beneficial. The main issue for me was the Unhandled exception when iterating the in-memory list of the objects was never the same. I suppose one could argue that this could be an overall .ToList() memory issue but without the ability to filter on the Include I was forced to alter data in a down system situation last night to overcome the limitation. This would be a huge win for those of us who find Include very helpful yet prone to data overload.

[divega]

@dodegaard In case it helps, have you had a chance to look at the new query filters feature supported by EF Core 2.0? Would that help in your scenario?

[divega]

@ajbeaven As you very well understand, the team considers that the priority of this feature is less because the overlap with query filters. It is very hard to asses what the current priority should be. One way you may be able to help is by articulating the scenarios in which you think this feature is still necessary but query filters can’t help.

[ajbeaven]

@divega There has been a handful of times where I've needed to load a subset of data for one query, but not for another. Here are a couple of examples:

var parts = _context.Parts
    .Include(c => c.Performances.Where(p => p.PerformerId == _currentUser.UserId));

var plays = _context.Plays
    .Include(c => c.Parts.Where(p => p.CanAudition));

Neither of these includes make sense on global level filter.

I'm also using AutoMapper's .ProjectTo to easily map EF datasets directly to ViewModels without loading superfluous data. Without filtered includes allowing me to load the exact dataset, I need to do some additional filtering after the mapper had done its work, which somewhat defeats the purpose.

[popcatalin81]

@ajbeaven Projections aren't done using 'Include' and includes are not used during projections. (https://github.com/AutoMapper/AutoMapper/wiki/Queryable-Extensions)

[ajbeaven]

@popcatalin81 Yikes, thanks for putting me straight there! I was confused between one mapping that used ProjectTo and another that used the Mapper.Map (the latter did require Includes to be specified).

That aside, the examples above are still valid use-cases as far as I'm concerned.

[xr280xr]

I absolutely need this and can't imagine any serious medium to large-scale, data-centered application not needing it. I'm shocked it has sat in the backlogs for coming up on a decade.

[niccha]

I also need this feature and encountered it on my first project with EF. Like others say, can't see many projects that won't need this feature...

[jhvanderven]

The workaround looks ugly. .Include(..).Where(..) is a readable, understandable syntax. Please Include this feature fast.

[mhosman]

1+, this is a must-have feature (I think... this is a very basic feature. All of my projects need this).

Example: var users = _context.Users.Include(u => u.Permissions.Where(p => p.PermissionType == 1));

[mhosman]

Hey @rowanmiller @ajcvickers this is something that can be done? You have any estimated time for this? Thanks!!!

[ajcvickers]

@mhosman This issue is in the Backlog milestone. This means that it is not going to happen for the 2.1 release. We will re-assess the backlog following the 2.1 release and consider this item at that time. However, keep in mind that there are many other high priority features with which it will be competing for resources.

[mhosman]

Hey @ajcvickers thank you very much for your response. The only approach here is to make a raw SQL query? Maybe there is some example to mix a full lambda query with include and add a FromSQL to filter the Include? Or maybe make an include and then filter the results manually? (this last option is not performant).

[anpete]

@mhosman Global query filters (HasQueryFilter API in OnModelCreating) are one way to do this.

[mhosman]

Hey @anpete I know that, but I need to make a filter based on a specific value that changes all the time.

[TsengSR]

@mhosman IIRC you can use member properties of the DbContext to do that already with the filtering API we have now. You just need to set the value, before executing the query. There are no concurrency issues as EF isn't thread-safe, so you can only do one query at a time

[mhosman]

@anpete @TsengSR Could you please provide some basic workaround example based on this example that I'm trying to apply (using lambda)??

var users = _context.Users.Include(u => u.Permissions.Where(p => p.PermissionType == 1));

Thank you very much!

[anpete]

@mhosman

Add a _permissionType field to your context and initialize the value from a ctor argument. Setup a filter on the Permission ET like: modelBuilder.Entity<Permission>().HasQueryFilter(p => p.PermissionType == _permissionType); Now any query for Permission will be filtered by the current value of _permissionType.

[mhosman]

@anpete Do you mean to initialize the value in the PermissionType Class? (sorry my confusion but I can't see where to initialize that specific value for that specific query in the controller).

[anpete]

The normal way is to have the value initialized when creating the context:

var context = new MyContext(permissionType: 1);

In this case, the value is "set" for the lifetime of that specific context instance. Other instances can have difference values.

You should also be able to add a PermissionType property to your context.

var context = new MyContext();
context.PermissionType = 1;
var q = context.Users.Include(u => u.Permissions).ToList();

This is less common but should allow you to change the value on the fly.

[mhosman]

Great @anpete! Thank you very much for your help!

[mhosman]

@anpete Just one last question. How to access _permissionType attribute from a Self-contained type configuration file?

[anpete]

@smitpatel Passing in the context to the entity configuration works, right?

[smitpatel]

No. See issue #10301

[mhosman]

Yes, passing the context to a file with this example code:

class CustomerConfiguration : IEntityTypeConfiguration<Customer>
{
  public void Configure(EntityTypeBuilder<Customer> builder)
  {
     builder.HasKey(c => c.AlternateKey);
     builder.Property(c => c.Name).HasMaxLength(200);
   }
}

[anpete]

Thanks @smitpatel

@mhosman For now you will need to define the filters in OnModelCreating directly

[weitzhandler]

How about:

.Include(f => f.Thumbnails
  .Where(t => t.ThumbnailSize < maxThumbSize)
  .OrderBy(t => t.Rank)
  .Take(3))
.ThenInclude(...)

[stap123]

Just want to throw a +1 onto this, would be excellent to have this feature I commonly hit this limitation.

[n123r]

Also adding a +1 onto this feature. The ability to filter on include on a per query basis would be an excellent addition to EF Core

[conterio]

It's been a couple years now! Work on this feature please!!!!! Query on Includes. I'm going to Ignite this year, have it done by then or else. Or else what? Or else i'll just have to keep waiting.

[olivierr91]

I also upvote this feature, and yes I know about global query filters and it doesn't solve any of my use cases.