Runtime Dynamic Global Query Filters (centralized where clauses for entities)

Danielku15 commented 5 days ago

Ask a question

Is there a hidden or official way to generate global query filters dynamically as part of query execution?

My data filters depend on some dynamic runtime information which can change the filters quite significantly. Creating an overall expression for all scenarios is quite a risky and challenging task.

In a normal query you would apply custom filters with .Where() on the DbSet but I also need to ensure any .Include() collections are also filtered according to the ruleset (OData style expands). AFAIK Global Query Filters have the benefit of being applied on the entities regardless of whether it is a main query or an include. If this assumption is wrong, I think we can close the discussion here.

It would be great if there is some service/callback/extension point where I can provide an additional .Where() for a given IQueryable<T> regardless whether it is a top-level or expand query.

Include your code

Assume a similar setup like the following project. You want to ensure that users can only see the projects they are members of and if the special access token is used, you can only see your own project. With this simplified setup you would want to generate a query filter matching your current state of authentication.

While adding filter clauses manually to all sets and expands might be technically possible but its risky and prone to errors. Global query filters would allow a central point of access control.

In my case I have an OData style API where users can specify the expands and the .Include clauses are dynamically added. Also here I could add a custom hook to apply where clauses. But if I extend my whole API and application framework to support this I was hoping to tackle this on the data access layer via EF directly.

class UserPrivilege { MasterAdmin, NormalUser }
class User 
{
    public UserPrivilege Privilege {get;set;}
}

class ProjectMember 
{
    public User User {get;set;}
    public Project Project {get;set;}
}

class ProjectGroup
{
    public IList<Project>? Projects {get;set;}
}

class Project 
{
    public ProjectGroup Group {get;set;}
    public string ProjectMasterToken {get;set;}
    public IList<ProjectMember>? Members {get;set;}
}

public class MyDbContext(IAuthInfo authInfo) : DbContext 
{
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<Project>(e =>
        {
            // how to do something like this: 
            e.HasQueryFilter(() =>
            {
                if(authInfo.IsTokenAuth)
                {
                    return (Project p) => p.ProjectMasterToken == authInfo.ProjectMasterToken;
                }
                else 
                {
                    return authInfo.UserPrivilege switch
                    {
                        UserPrivilege.MasterAdmin => ((Project p) => true),
                        UserPrivilege.NormalUser => ((Project p) => p.Members.Any(m=>m.User.Id == authInfo.UserId),
                    };
                }
            });
        };
    }
}

var groupsWithProjects = myContext.Set<ProjectGroup>()
    .Include(g => g.Projects)
    .ToArray();

Diving deeper

As far I could see query filters are applied in the NavigationExpandingExpressionVisitor:

https://github.com/dotnet/efcore/blob/e84a366a480ee6fabb87c85945f0818fd88fe60f/src/EFCore/Query/Internal/NavigationExpandingExpressionVisitor.cs#L1755-L1760

Seems it would not be a huge change to have GetQueryFilter() providing a dynamic value instead of pre-registered object in the model.

I was thinking if I add a custom Expression visitor via QueryTranslationPreprocessor and IQueryTranslationPreprocessorFactory

Would be great to get some feedback on this matter.

roji commented 5 days ago

The global query filter is, by design, meant to apply the same filter(s) to all queries (except when IgnoreQueryFilters is specified), without variance across queries. It sounds like you want different filters to be applied to different queries, so they're no longer global at that point.

Now, if I understand correctly, you want to simply apply filtering - within a specific query - that would apply on an entity type within that query regardless of where/how it's being used; for example, the filtering would apply to the entity type when it's the root (DbSet), and when it's being Included.

If we're indeed within the scope a single query, then what's the problem with simply explicitly specifying the filter in that query wherever you reference the entity type? Most queries don't reference an entity type more than once (i.e. don't self-join), so in any case the filter has to only be specified once - why not just do it at that point?

Danielku15 commented 5 days ago

It sounds like you want different filters to be applied to different queries, so they're no longer global at that point.

It depends a bit on the definition/understanding of "global filter". I would still be registering one single "global" callback per entity defining the general business rules to access this entity. The callback might have multiple logic paths resulting in an expression. You're right that its maybe not stictly one single global filter expression anymore but the expressions are somehow dynamic.

Now, if I understand correctly, you want to simply apply filtering - within a specific query - that would apply on an entity type within that query regardless of where/how it's being used; for example, the filtering would apply to the entity type when it's the root (DbSet), and when it's being Included.

I'm not having really a specific query here but rather a general "query endpoint". Think of an API offering inputs where the end-user can specify what to expand on HTTP level (like a GraphQL or OData API). These expands internally result in .Include() calls being appended to the base query. Regardless of how the user might expand down the data graph the appropriate filtering rules need to be applied.

Most queries don't reference an entity type more than once (i.e. don't self-join), so in any case the filter has to only be specified once - why not just do it at that point?

I could theoretically extend my API query framework where I apply filtering on my level when translating the API request. Similar to the ASP.net core OData framework, I am passing a IQueryable<T> to my framework to apply any filters and expands.

From architecture standpoint I would prefer to define the rules and have the filtering on the business & data access layer than relying on the framework translating the user inputs. This mitigates the risk that though side-channels some data might be exposed.

roji commented 5 days ago

It depends a bit on the definition/understanding of "global filter". I would still be registering one single "global" callback per entity defining the general business rules to access this entity. The callback might have multiple logic paths resulting in an expression. You're right that its maybe not stictly one single global filter expression anymore but the expressions are somehow dynamic.

Sure; definition-wise, I was referring to the "global query filter" feature which EF already has (doc); this is very different from what you're describing.

The callback might have multiple logic paths resulting in an expression. You're right that its maybe not stictly one single global filter expression anymore but the expressions are somehow dynamic.

I understand the need here; but the problem here is going to be what that callback will get as its input, and how that's going to be passed from the context of a specific query. In other words, I'm not sure how one could provide a completely generic piece of logic (the dynamic global callback), which then somehow gets some specific, per-query information that determines which actual filter(s) get added when.

I'd suggest trying to come up with a very concrete example of such a dynamic filter that you'd like to define globally; that could provide a basis for further conversation here.

dotnet / efcore