Proposal: extend the set of things allowed in expression trees

[gafter]

Expression trees support only a subset of the things that could be semantically meaningful. Consider allowing them to support the following constructs:

• The null-coalescing operators such as a?.b, a?[b]
• Expressions involving dynamic
• Invocations that use default or named parameters
• Assignment and other mutation expressions
• A statement-bodied lambda

[olmobrutall]

I think maybe there is a dogfooding problem.

C# team is using the language to build Roslyn so they miss features related to in-memory functional stuff and low-level optimizations. They don’t do too much DB code themselves, so they don’t miss it that much.

[buzzytom]

That seems like a reasonable assumption, I like the dog food analogy 👍

It's not limited to just DB programming either, it anywhere you would convert C# to another form/language. The two cases I can think of:

• Linq -> Sql
• C# -> Shader Languages

Maybe if we could list some other use cases they might see the value.

[olmobrutall]

GraphQL could make IQueryable sexier than ever

[CyrusNajmabadi]

To be honest, at this point, I'm wondering how this can be stagnant for this long. Is it simply that it's just an edge case that few would actually use?

Because there are thousands of requets, with many many many people asking for all sorts of features all over the place :) On top of this, there are only finite resources to make any changes. As such, it will always be the case that some request do not make the cut.

[CyrusNajmabadi]

so they miss features related to in-memory functional stuff and low-level optimizations. ... so they don’t miss it that much.

I think it's more that the C# team has to take a broader view beyond just what they may be hitting themselves. The LDM and product teams have to look at the broad ecosystem out there and think about what provides the highest value given the resources and time available.

So far, this just hasn't reached the bar to warrant the investment. Just like the thousands of other requests that have also not been able to be scheduled. :)

[buzzytom]

Just looking at the current issues list there are 1055 open feature requests. It would be interesting to know how close to the surface they all sit in the sea of noise.

Given the solution is a significant change, maybe it can be split into smaller meaningful pull requests, that are more likely to be accepted. I'm happy to have a go if I can find a small chunk in my spare time, although I've never worked in Roslyn.

[olmobrutall]

Well, if ?. doesn’t work with IQueryable you’ll need to choose between not nullable feature in c# 8 or EntityFramework. That looks high priority to me.

[gafter]

Marking as blocked, as committing to a particular release would probably require a commitment from another team.

[Joe4evr]

dotnet/roslyn#40326 asks for Range support in expressions.

[hez2010]

How is it going now? Seems that it has passed 4+ years since proposed, and Expression now lacks support of many features.

[olmobrutall]

Marking as blocked, as committing to a particular release would probably require a commitment from another team.

Is Roslyn depending on EF commitment? Looks like the issue is in no-man land for eternity :(

[IS4Code]

Not only is there no expression support for features that were added since LINQ, but there is still no support for features that were already available at the time expressions were introduced.

Is really none of these possible without library support?

• Calling methods with ref parameters;
• some simpler statement-bodied lambdas;
• assignment;
• dynamic binding;
• construction of tuples.

A sufficient progress here does not have to satisfy everything. While I would also welcome features that feel "additional" to me, like await support and being able to invoke an actual instance of LambdaExpression from a normal C# expression, finishing stuff that already has proper runtime support should have priority.

[bartdesmet]

Dropping a link to a hobby project prototype of this type of work, including support for statement trees, dynamic, async lambdas, and a lot of the newer language constructs (feature complete up until C# 6.0, partial implementation beyond that point). It comes with a Roslyn fork that supports capturing these language constructs in expression trees by binding to the corresponding runtime library that represents CSharpExpression nodes can be visited using a CSharpExpressionVisitor and that lower/reduce to classic Expression nodes otherwise (so LambdaExpression.Compile just works out-of-the-box as well).

https://github.com/bartdesmet/ExpressionFutures/tree/master/CSharpExpressions

[thomaslevesque]

The lack of the null-conditional operator and pattern matching in expressions is getting more annoying by the day. Almost daily, I find myself wanting to use them in EF Core queries, only to be reminded that it's not supported.

I realize it's probably going to be a lot of work, but I think it should really be a priority. Right now it's like we have two versions of the language. One that is awesome with plenty of shiny new features, and one that is still stuck in 2008. And it's getting worse for every new C# version...

[leandromoh]

For those who also came here looking for expression tree support of null propagating operator ?. and found no progress, I created a project with a visitor that modify expression tree for safe null propagation

[bartdesmet]

There's now a proposal by a member of the Entity Framework team to move expression trees forward. See https://github.com/dotnet/csharplang/discussions/4727

[atrauzzi]

This can't come soon enough. Would really like to be able to use switch expressions in expression trees. I mean, they practically share half their names in common! :laughing:

[aradalvand]

This needs to be prioritized to be honest, this is one of the most disappointing limitations of the language. Still no updates or anything?! It's been a long time since a maintainer commented on this issue, and the issue is 7 years old already, there doesn't seem to me to be any justifiable reason for the fact that it's still not being taken with due seriousness.

[seriouz]

I stumbled over this issue some days ago, too. @MadsTorgersen can you please explain what is the problem of solving this?

[CyrusNajmabadi]

Breaking existing systems that do not expect these nodes.

[hez2010]

Breaking existing systems that do not expect these nodes.

The fact is that the existing systems are waiting for the language change so that they can follow up.

[ViIvanov]

Breaking existing systems that do not expect these nodes.

Why it is not good if existing systems will not understand new expressions (nodes) and will throw exceptions? Looks like it is even possible to create analyzer that will check "the version" of expression and how is it supported by existing system.

If I remember correctly some years ago the reason was in complexity and amount of work. Nice to hear, that it is not the main a reason now 👍

[IS4Code]

If I remember correctly some years ago the reason was in complexity and amount of work. Nice to hear, that it is not the main a reason now

And actually it's still not the reason for most of the features anyway ‒ dynamic expressions, assignment etc. are all already supported, in fact have been supported for 12 years since .NET Framework 4.0. If adding new expression types was not an issue back then, it's not such an issue now. Making use of what is available now is however still a worthwhile step, even if we'd have to wait for things like null-coalescing operators or pattern matching (and even those could be modelled in a way that does not strictly require new expression types).

[hez2010]

Breaking existing systems that do not expect these nodes.

Also it doesn't sound like a reason. We now have source generator, so basically adding any new language feature will introduce new kinds of nodes in the AST and break the existing source generators because they don't recognize the new nodes. This is the same with the expression tree. I don't see why we can't add new nodes to expression tree. If it is for the case that the existing libraries (such as efcore) will get some nodes that are yet able to be unrecognized, they can simply throw NotSupportedException to prevent users from using new expressions until they add support for new nodes. The old input will still work without any breaks. And note that even today efcore still has untranslatable expressions.

[CyrusNajmabadi]

and break the existing source generators because they don't recognize the new nodes. This is the same with the expression tree.

They're explicitly different situations. One started out in a world where it wsa not changing, and effectively had a contract over time that it would not break (and, indeed, breaks happened and had to be rolled back because it was a significant issue in practice). The other started in a world where we said explicitly it would change and change often.

If we did expression trees a new time, we've said we would go the roslyn model. But that's a new system, not an extension to the existing one.

[Eli-Black-Work]

To everyone commenting here: You might also want to check out https://github.com/dotnet/csharplang/discussions/4727, which contains more discussion about how a feature like this might be implemented 🙂

[IS4Code]

and break the existing source generators because they don't recognize the new nodes. This is the same with the expression tree.

They're explicitly different situations. One started out in a world where it wsa not changing, and effectively had a contract over time that it would not break (and, indeed, breaks happened and had to be rolled back because it was a significant issue in practice). The other started in a world where we said explicitly it would change and change often.

If we did expression trees a new time, we've said we would go the roslyn model. But that's a new system, not an extension to the existing one.

There are many solutions that don't cause such issues. Taking use of existing expression types that have been there for 10 years is a good start. Then, if you want to start anew, why not allowing to bind a lambda to the Roslyn model, in addition to Expression<T>?

By the way, what kind of issues did the addition of new expression types in .NET 4.0 cause, and how were they solved?

[bernd5]

The problem is not the existence of new expression types (some where added which are not touched by the compiler). The problem is that it is expected from the compiler to apply the language semantics to the generated tree - not just the syntax. The compiler handles for example implicit conversions, capturing, overload resolution...

For consumers this is quite great because it avoids the need to handle all that... But for the compiler it becomes quite hard.

[aradalvand]

Now almost 8 years since this issue was created, numerous new features have been added to C#, yet expressions are still stuck in C# 3.0 or something. This is simply unacceptable at this point. I fail to see why this is not being prioritized or getting any traction whatsoever, given that it's becoming an increasingly evident pain point. Proposals like #4727 also seem to have been effectively abandoned.

Any updates? cc @CyrusNajmabadi, @tmat, @alrz, @MadsTorgersen

[CyrusNajmabadi]

I fail to see why this is not being prioritized or getting any traction whatsoever

Other things have been more important, and have also not have the significant concerns and constraints present here.

Any updates? cc @CyrusNajmabadi, ..

No. THere have been no updates. Any updates would be posted here as we do all this development in the open. :)

Note: you're welcome to come discuss things more with the team on Discord if you'd like! :)

[Tragetaschen]

More than once I missed the ability to compose expressions or have expression with "holes" to plug in another expression. Especially in the context of Entity Framework I have a lot of cases where queries have common sub expressions, but I cannot extract those into a reusable expression. As a very simplistic example, think

public static IQueryable<T> MySpecialFilter<T>(this IQueryable<T> queryable, Expression<Func<T, bool>> predicate)
{
    return queryable.Where(x => x != null && predicate(x)); // does not compile, because expressions cannot be "called"
}

I always naively assumed that this kind thing should be easily expressible in the tree object model by stuffing predicate at the correct place instead of creating that particular subtree.

[FaustVX]

@Tragetaschen You have to call predicate.Compile()(x) But expression are not meant to be executed.

[olmobrutall]

More than once I missed the ability to compose expressions or have expression with "holes" to plug in another expression. Especially in the context of Entity Framework I have a lot of cases where queries have common sub expressions, but I cannot extract those into a reusable expression. As a very simplistic example, think

public static IQueryable<T> MySpecialFilter<T>(this IQueryable<T> queryable, Expression<Func<T, bool>> predicate)
{
    return queryable.Where(x => x != null && predicate(x)); // does not compile, because expressions cannot be "called"
}

I always naively assumed that this kind thing should be easily expressible in the tree object model by stuffing predicate at the correct place instead of creating that particular subtree.

Hallo @Tragetaschen Maybe this links help: https://github.com/signumsoftware/framework/blob/master/Signum.Utilities/ExpressionTrees/LinqExtensibility.md https://github.com/signumsoftware/framework/blob/2540317460bd5460fcd155876cc94f8ffa1cab81/Signum.Utilities/Extensions/ExpressionExtensions.cs#L48 https://github.com/signumsoftware/framework/blob/2540317460bd5460fcd155876cc94f8ffa1cab81/Signum.Utilities/ExpressionTrees/ExpressionCleaner.cs#L60 https://tomasp.net/blog/linq-expand.aspx/

[svick]

@olmobrutall Or just use LINQKit (NuGet).

[aradalvand]

Having just re-read this thread and also this one, all the while carefully considering the concerns and explanations provided by the likes of @CyrusNajmabadi, @333fred, et al. I must say they are making about as much sense as how many "new" C# syntactic features have had support added for them in expression trees over the last decade or so, which is, of course, a glorious zero.

Let's consider this comment by @333fred — as I think it succinctly and clearly presents the supposed justification:

The significant cost of implementing new nodes does not come from the compiler. It's fairly easy to do in the compiler. Its significant cost comes from every downstream tool now being broken because there is a new node in the tree they were not expecting, and us needing to work with EF and other consumers of expressions to create a way to signal language version support.

(emphasis mine)

Sounds reasonable, right? Except that it should take no more than 5 seconds for anyone remotely familiar with C#/.NET's and the surrounding ecosystem's progression over the last couple of years, to realize two things, which immediately obliterate this reasoning:

Swap the terms "new node" with "new built-in anything" and you'll get an equally sound argument for never adding any feature at any level in any area to the framework/standard library ever; because "what about existing tools that won't immediately support it?", I guess. It is unbelievably unconvincing.

There is literally a fundamental analogy here between any hypothetical new expression nodes, and any new standard constructs in general — with the most notable recent example being what happened with TimeOnly and DateOnly: They got added to the standard library, tools like EF, various serializers, etc. didn't immediately recognize/support them, so any usage in such unsupported contexts resulted in either a runtime exception or unexpected behavior at the time, until support gradually got added. How would new expression nodes be fundamentally ANY different than that?

Notice that according to the same logic as the one that we're being provided here as to why expressions are stuck at the C# of 10 years ago, apparently the addition of TimeOnly and DateOnly would've been a heinous, unthinkably nightmarish idea, because:

The significant cost of implementing [TimeOnly and DateOnly] does not come from the [standard library]. It's fairly easy to do in the [standard library]. Its significant cost comes from every downstream tool now being broken because there is a new [built-in type] in the [framework's standard library] they were not expecting, and us needing to work with EF and other consumers of [user code] to create a way to signal [framework] version support.

It gets even worse, this argument is, in fact, even LESS applicable in the case of expression trees, because all consumers of expression trees (most prominently EF), ALREADY ONLY support a subset of "all possible expression trees". An expression tree consumer not supporting a certain thing at a given time is already the status quo, and as such, fully expected.

Funny. Some people in the other thread (e.g. @olmobrutall here) tried to point out the same thing, heck, even some comments on related Stack Overflow questions made this point, and made more basic logical sense than anyone from the other side here. It's truly astonishing.

[IS4Code]

I don't think the argument here is simply "they cannot be added because that will break existing libraries", it is "how to add them in a way that both preserves the original meaning and interoperates them with existing code".

You could rewrite x.a == 1 && x.b != null into x is { a: 1, b: not null } expecting it to work, but the latter would result in a different expression, even though the change should not affect the outcome.

That being said, there is a myriad of currently unused expression types that are not simply a lowering of other expressions, like DynamicExpression, so those are "safe" because there are no other choices for the representation.

[xamir82]

You could rewrite x.a == 1 && x.b != null into x is { a: 1, b: not null } expecting it to work

But those are still two different expressions (even if they might be functionally the same); so it's only natural that they're represented by two distinct expression node types. In normal code, the compiler might lower one to the other, yes, but that's just an implementation detail of the compiler and fundamentally not relevant to this conversation.

Expressions reflect what's on the "surface", so to speak, just as you can't just replace (x, y) => x + y with (x, y) => Sum(x, y) in an expression tree (where Sum is just int Sum(int x, int y) => x + y) and "expect it to work". Even though if they were evaluated they'd be functionally equivalent, functional equivalence is not what determines expression node types.

[IS4Code]

@xamir82 It is relevant ‒ it is not always apparent whether a lambda is treated as an expression, and someone might easily rewrite it as a part of modernizing the code. Even x is null is different from x == null, but not as much as changing it to a method (unless you are in Unity or somewhere else where == is overloaded for null testing). I don't personally think that it is such a big deal, since you could easily have an analyzer to let you know if this kind of expression is given to a target that might not support it (attributes to express that would be nice altogether), and the focus now should be on .NET 4.0 types anyway where this is not an issue. Moreover, there is Expression.Reduce which should do exactly what is needed to deal with those issues.