Query: Pay your (technical) debt on time

[smitpatel]

This is a grouping of related issues. Feel free to vote (👍) for this issue to indicate that this is an area that you think we should spend time on, but consider also voting for individual issues for things you consider especially important.

---

Creating a general tracking issue for tasks which were deferred in new query pipeline due to time constraint and also as an iteration over the design as we see customer reports we are getting.

This is a meta issue. It will evolve and add more stuff as we discuss things in team meetings (like the one we did about null semantics & command caching). I would prefer this to be assigned to me as the sticky we decided in the planning process. And we will split out smaller tasks out of this and assign to actual implementer. If you are working on something already, please add tracking issue here next to the item.

• Parameter extraction

  • [ ] #19209
  • [ ] #17598

• Preprocessor

  • [ ] Extensibility
  • [x] Aggregation of visitors to avoid traversing whole tree multiple times
  • [ ] Nav expansion improvements
  • [x] Processing of pending selector. When can we flow the selector through. When do we need to apply the selector to get back original shape.
  • [ ] #20291
  • [x] Processing of complex operators like GroupJoin/GroupBy from nav expansion perspective.
  • [x] #20146
  • [x] #21234

• Translation

  • [x] GroupBy aggregate translation improvement
  • [x] Allowing more operators before aggregate
    • [x] Distinct (also introduction of Distinct Column syntax)
    • [x] Where using case block with null as else
    • [x] Ignore/trim OrderBy/ThenBy since it is not useful? (wait for feedback)
  • [x] Keep them enumerable methods or convert to queryable (enumerable methods)
  • [x] Pending collections
  • [x] A different way to store them to avoid disconnect
  • [x] Multi level deep correlation predicate
  • [x] Keep them in sync with outer
  • [x] ColumnExpression.Table
  • [x] Avoid storing actual Table with columnExpression
    • [x] Different way to assign unique table alias
    • [x] Do we ever update the table alias during any visitation?
    • [x] readonly table alias?
    • [x] Hash Code computation
  • [x] Make it readonly after assigning alias?
  • [ ] Block translation of member/methodCall when store type is different from expected
  • [ ] Alternate translation? Eg. ToString
  • [x] Block translation of collection projection when it is not possible to correlate
  • [x] Distinct case
  • [x] GroupBy aggregate case
  • [x] #20332
  • [x] #20164
  • [x] #21321
  • [ ] #21320
  • [x] Ability to bind after client projection
  • [ ] #21280
  • [ ] #21273
  • [ ] #26048

• Postprocessor

  • [ ] Extensibility
  • [ ] Aggregation of visitors to avoid traversing whole tree multiple times

• ShaperCompiler

  • [ ] Aggregation of visitors to avoid traversing whole tree multiple times (pending for InMemory/Cosmos)
  • [ ] #21334
  • [ ] #21335
  • [ ] #21336

• SqlProcessor

  • [x] Aggregation of visitors to avoid traversing whole tree multiple times
  • [x] SqlFunctionExpression nullability
  • [x] If parameter propagates null (if arg is null then func is null)
  • [x] If Function can be nullable (func may never be null even if arg is null)
  • [x] API on DbFunction to configure these facets
  • [x] SearchCondition conversion to relational in order to do conversion for bool columns with value converter
  • [x] Opt out flag to generate shorter SQL. (non-SqlServer kind)
  • [x] NullSemantics after parameter values are known @maumar
  • [x] What should go after relational command caching (visitors which requires parameter values)

• Command execution

  • [ ] Relational command caching which uses lambdas rather than just parameter nullability (partial parameter sniffing) @roji
  • [x] Disconnect one-to-one mapping between client parameters and server parameters

• Whole pipeline

  • [x] Exception messages using core string (referenced in code with TODOs)
  • [x] Cosmos requires Offset-Limit both
  • [x] Move to correct assembly
  • [x] Avoid throwing exception in places other than translation
    • [x] Must do for GroupJoin
  • [x] Evaluate what is being printed for exception when printing non-linq tree
  • [x] XML Docs

[smitpatel]

Consider refactoring in cosmos similar to https://github.com/dotnet/efcore/pull/21181

[smitpatel]

21700 causes slight regression and spike in memory usage just to compute hash code. Which leads into another rabbit hole that projection is mainly comprised of column expressions coming from tables and when computing their hash code we may go inside another select expression.

Ideally this is not necessary

• The hash code computation of SelectExpression will figure out mismatch Tables through SelectExpression.Tables
• If graph is consistent then a particular column expression would only come from a particular table. I tried to apply solution that ColumnExpression can be compared with just reference but it caused test failure. A solution where column expression are created from a fixed place of reference could resolve the issue as no 2 similar column expression instance would be generated. Things to keep in mind for that is all the places which can create column expression should not create one if there is already generated one. MakeNullable would be a bottleneck.