Add Select-Methods that are being executed after materialization only

[angelaki]

I really love the idea of the last Select being executed after materialization if it can't be translated to SQL. Now I have a use-case where I'd love to append Select methods that are being executed only after materialization.

My service provides an IQueryable of objects that get user-defined properties set after materialization (via Select). Right now some methods can't set these values because the consuming services need to append some more filters, etc. (Where, modify Select fields e.g.) so right now the consuming service gets responsible of setting these properties.

If my service providing these entities could append it via e.g. .AfterExecuteSelect() I could solve this issue.

[roji]

You could simply add some trivial, no-op operator, such as Where(b => b != null) after the Select, this should make the query fail if the Select contains anything that requires client evaluation. You could also wrap this Where and the Select in a single extension method, and call it AfterExecuteSelect.

[angelaki]

Thank you for the recommendation but I do not want it to fail (I actually hate failing code 😉). I'd like to be able to put some kind of Select statement in the IQueryable pipe that never gets translated but only executed after materialization (just like it was the last select statement).

I guess that feature request is reasonable and could be quite handy.

[roji]

I'm not sure I fully understand - you want to integrate a Select somewhere in the middle of the query, but which will only be evaluated at the end of the query? I'm not sure how that could work, given that the query result shape can change by query operators (i.e. the shape in the middle isn't necessarily the same as the shape in the end).

Maybe submit a small code sample showing what you're actually looking for?

[angelaki]

given that the query result shape can change by query operators (i.e. the shape in the middle isn't necessarily the same as the shape in the end).

I actually forgot about that point. So yes, if the resulting model does not fit the type of that extension method an InvalidOperationException must be thrown. I gues I'd need to show you my usecase so you understand:

Some properties of my model change depending on the user viewing the entity. So I'm setting the request's IAuthorizationService this way:

public IEntity SetAuthorizationService(IAuthorizationService? authorizationService) { _authorizationService = authorizationService; return this; }

Now I can do dbContext.Entities.Where(e => true).Select(e => e.SetAuthorizationService(this)); and it works quite fine - unless someone consuming my IQueryable appends some filters e.g..

I confirm this is a pretty special use-case but it would be super awesome to allow this. Actually the method doesn't need (or even may not) have the Select's functionallity of transforming the model but rather modify it. I obviously totally confused you recommend using the Select method since it's more an abuse I'm doing here. So the method I'm recommend should not look like public static IQueryable<TResult> Select<TSource, TResult>(this IQueryable<TSource> source, Expression<Func<TSource, TResult>> selector) but rather public static IQueryable<TSource> AfterExecution<TSource>(this IQueryable<TSource> source, Expression<Action<TSource>> selector) (throwing an Exception if the result set does not fit TSource or having an optional overload that just skips it, handles it as object or similar).

I really hope you like the idea!

[roji]

Thanks for the additional details. It seems to me that you're trying to force SetAuthorizationService into EF Core's query pipeline, where in fact it doesn't belong there, but should rather be composed by the consumers of your query's results; your method doesn't actually operate on IQueryable, in the sense that SetAuthorizationService can't ever be translated - it could simply be composed on the Enumerable results that come out after evaluating the query (e.g. with AsEnumerable).

Another way to think about this, is to imagine that instead of EF Core you're actually getting results via plain old LINQ to Objects in memory. There's no way there to integrate a think in the middle of the query that would somehow be executed at the very end - and I don't see why EF Core would be different. This, aside from the more practical issues of the shape changing and various other difficulties I can see with this.

But we can see what others in the team think about it.

[angelaki]

Totally understand your doubts! In my case there is an IObjectService and an ObjectController. The controller consumes IQueryables from the service and is still able to append some paginations etc. on the result - what is quite cool! Materializing them before causes horrible performance, guess we both agree here.

Now the controller has absolutely no idea of the service's internal stuff like authorization etc.. He just acts as the interface between service and client.

I guess this use case is pretty clean - but forces me to do something like this.

[roji]

Sure - I'm definitely not telling you to materialize before applying pagination. But your controller must be made aware of the authorization in some sense, possibly by having your service return it out as a final operator to be composed on the query after the pagination etc.

As above, my recommendation is for you to think about what you would do if this were simply LINQ to Objects rather than EF Core.

[angelaki]

If it was simply LINQ and not EF Core I could easily use my approach. But that's the great about EF Core! It leaves the elements on the server only querying them in the very last moment necessary. The simply LINQ objects already exists so there is no need to delay execution.

If you don't agree on my idea of the feature - is there currently at least any chance to hook the elements creation? Worst case via reflection? So I can at least apply my service on every instance (independent of the query generating it).

[roji]

If you don't agree on my idea of the feature - is there currently at least any chance to hook the elements creation? Worst case via reflection? So I can at least apply my service on every instance (independent of the query generating it).

Something like this may be possible with a lot of hacking - I definitely wouldn't recommend it (@smitpatel may have ideas here).

[smitpatel]

I really love the idea of the last Select being executed after materialization if it can't be translated to SQL.

That is not true so we can discard the whole concept that EF Core executes a select after materialization.

If I understand correctly then what you want to do is call a certain client method after the results of query are generated. (Hence I agree with @roji that this is not something which fits in EF Core query pipeline for sure). If this method needs to be called only on entity object then https://github.com/dotnet/efcore/issues/15911 may help. If it goes beyond that then you need to use some 3rd party library like OData or AutoMapper (not sure if they support this out of box) to intercept and construct query you want. Especially AutoMapper intercept each query and inject a custom selector on top of it.

Given it is task of intercepting query to augment it to do something which EF Core query pipeline can never translate, I do not believe it belong in EF Core at least the operation AfterExecute.

is there currently at least any chance to hook the elements creation? Worst case via reflection? So I can at least apply my service on every instance (independent of the query generating it).

Not really easy but if you really want to implement your own solution then look into how to intercept query being sent to EF Core to append Select on top of it like how AutoMapper does. This is not recommended user action and more for 3rd party library writers so if you pursue this path, you would need to understand a lot of internals of how query pipeline bootstrap with little to no documentation. For any questions related to this adventure should be directed to forums like stackoverflow.

[angelaki]

Ok, my fault: it doesn't execute the last select statement but just keep it as part of IEnumerable, if it couldn't have been translated. So an ExtensionMethod like AfterExecute would follow just this concept, wouldn't it?

Actually https://github.com/dotnet/efcore/issues/15911 sounds exactly what I am looking for but isn't implemented, yet, right? Wouldn't a OnMaterialized ExtensionMethod just fit this use-case super well?

[smitpatel]

it doesn't execute the last select statement but just keep it as part of IEnumerable, if it couldn't have been translated

That is not true either. The partial client evaluation is altogether different concept from evaluating Select on client side. So what query pipeline currently does is not directly related to what you are asking for. Specifically what you are asking for is to execute Select in enumerable world, which can be achieved easily for any query by calling AsEnumerable before calling Select. Now since you want your service to be composable, you need to implement functionality to add AsEnumerable().Select at the end of query in your app. EF query pipeline doesn't see AsEnumerable operator at top level due to how queryables work in LINQ world, so any such processing is outside the bound of EF. So the extension method certainly doesn't belong in EF Core on the queryable. We have plan to implement #15911 (which is different from the extension method being requested here, though that may be main issue being solved and extension method is posing as XY Problem). Anything more you want to do or cannot wait till #15911 then you need to look into intercepting queries before being passed to EF Core to change it whatever way you need it to.