Announcing the Plan for EF Core 6.0 (Updated)

[ajcvickers]

May update

In January we announced the plan for EF Core 6.0. Since then we have learned more and gathered more data from customers. These inputs have lead to some updates to the plan. This includes punting some smaller enhancements to the next release, as well as scoping of some the bigger items.

Scoped into 6.0:

The most notable changes scoped into EF Core 6.0 are:

• We have been doing extensive customer development around the Cosmos provider. This has resulted in the following enhancements being pulled into EF Core 6.0:
  • Azure Cosmos DB provider should default to implicit ownership
  • Find/FindAsync performs SQL API query when entity has embedded entities
  • Set partition key on join entity type by convention
  • Optimize more queries that could use ReadItem
  • Detect partition key filters in more queries
  • Translate subquery in filter condition
  • FromSql support
  • Allow to specify consistency level for CUD operations
  • Configure TTL per entity/entity type/collection
  • API to configure container facets (throughput, size, partition key, etc.)
  • Diagnostic events including statistics (query cost, activity id)
  • Support aggregate operators
  • Distinct operator in queries
  • Add translators for member/methods which map to built-in functions
  • Add basic support for collections and dictionaries of primitive types

The most notable changes scoped out of EF Core 6.0 are:

• We have improved test infrastructure and added new tests to support the work done for EF Core 6. Additional improvements in this area have been scoped out of the EF Core 6.0 release.
• Migrations bundles are still in the plan, but most of the other Migrations features have been cut due to resource constraints.
• Raw SQL queries for primitive and unmapped types has been cut from 6.0 due to resourcing constraints and priority adjustments.
• The following miscellaneous issues were cut from the 6.0 release due to resource constraints:
  • Complete full/free-text search on SQLite and SQL Server
  • SQL Server spatial indexes
  • Use the new batching API from ADO.NET

---

Original post

Today we are excited to share with you the plan for Entity Framework Core 6.0. This issue contains a quick summary of the plan and acts as a place for you to leave feedback.

This plan brings together input from many stakeholders and outlines where and how we intend to invest for the Entity Framework (EF Core) 6.0 release.

This plan is not set-in-stone and will evolve as we work on the release based on what we learn. This learning includes feedback from people like you, so please let us know what you think!

IMPORTANT This plan is not a commitment. It is a starting point that will evolve as we learn more. Some things not currently planned for 6.0 may get pulled in. Some things currently planned for 6.0 may get punted out.

General information

EF Core 6.0 is the next release after EF Core 5.0 and is currently scheduled for November 2021 at the same time as .NET 6. EF Core 6.0 will align with .NET 6 as a long-term support (LTS) release.

EF Core 6.0 will likely target .NET 6 when released. It is unlikely to support any .NET Standard version. It will not run on .NET Framework. See the future of .NET Standard for more information.

Themes

Highly requested features

As always, a major input into the planning process comes from the voting (👍) for features on GitHub. For EF Core 6.0 we plan to work on the following highly requested features:

• SQL Server temporal tables
  • Allow temporal tables to be created via Migrations, as well as allowing access to historical data through LINQ queries.
• JSON columns
  • Introduce common patterns for JSON support that can be implemented by any database provider.
  • JSON column support will be implemented for SQL Server and SQLite. (Note that the PostgreSQL and MySQL providers already support JSON columns.)
• ColumnAttribute.Order
  • Allow arbitrary ordering of columns when creating a table with Migrations or EnsureCreated.

Performance

While EF Core is generally faster than EF6, there are still areas where significant improvements in performance are possible. We plan to tackle several of these areas in EF Core 6.0, while also improving our perf infrastructure and testing.

• Performance infrastructure and new tests
  • Improve the infrastructure for performance tests as well as adding new tests and fixing low-hanging fruit.
• Compiled models
  • Compiled models will improve startup performance, as well as having generally better performance when accessing the model.
• TechEmpower Fortunes
  • We plan to match Dapper performance on the TechEmpower Fortunes benchmark. (This is a significant challenge which will likely not be fully achieved. Nevertheless, we will get as close as we can.)
• Linker/AOT
  • We will continue investigating in making EF Core work better with linkers and AOT. We do not expect to fully close the gap in the 6.0 timeframe, but we hope to make significant progress.

Migrations and deployment

Following on from the investigations done for EF Core 5.0, we plan to introduce improved support for managing migrations and deploying databases. This includes two major areas:

• Migrations bundles
  • Migrations bundles will provide a simple, robust mechanism for deploying EF Core migrations.
• Managing migrations
  • Wwe plan to improve the tools and project/assembly management for EF Core migrations.

Improve existing features and fix bugs

• EF6 query parity
  • We plan to close the query gap to EF6 and make supported EF Core queries a true superset of supported EF6 queries.
• Value objects
  • We plan to introduce a better experience focused on the needs of value objects in domain-driven design.
  • This approach will be based on value converters rather than owned entities which have proved not to be a good fit.
• Cosmos database provider
  • We are actively gathering feedback on which improvements to make to the Cosmos provider in EF Core 6.0; please make sure to vote (👍) for the Cosmos features that you need.
• Expose model building conventions to applications
  • Model building conventions are currently controlled by the database provider. In EF Core 6.0, we intend to allow applications to hook into and change these conventions.
• Zero bug balance (ZBB)
  • We plan to fix all outstanding non-blocked bugs during the EF Core 6.0 timeframe.
• Miscellaneous smaller features
  • Split query for non-navigation collections
  • Detect simple join tables in reverse engineering and create many-to-many relationships
  • Complete full/free-text search on SQLite and SQL Server
  • SQL Server Spatial Indexes
  • Mechanism/API to specify a default conversion for any property of a given type in the model
  • Use the new batching API from ADO.NET

.NET integration

The EF Core team also works on several related but independent .NET Data technologies. In particular, we plan to work on:

• Enhancements to System.Data
  • Implementation of the new batching API
  • Continued work with other .NET teams and the community to understand and evolve ADO.NET
  • Standardize on DiagnosticSource for tracing in System.Data.* components
• Enhancements to Microsoft.Data.Sqlite
  • Connection pooling
  • Prepared statements
• Nullable reference types
  • We will annotate the EF Core code to use nullable reference types.

Experiments and investigations

The EF team is planning to invest time during the EF Core 6.0 timeframe experimenting and investigating in two areas. This is a learning process and as such no concrete deliverables are planned for the 6.0 release.

• SqlServer.Core
  • An experiment in collaboration with the community to determine what potential there is modern .NET features in a highly performant SQL Server driver.
• GraphQL
  • We plan to investigate the space and collaborate with the community to find ways to improve the experience of using GraphQL with .NET.

Find out more and give feedback

This post is a brief summary of the full EF Core 6.0 Plan. Please see the full plan for more information.

Your feedback on planning is important. The best way to indicate the importance of an issue is to vote (👍) for that issue on GitHub. This data will then feed into the planning process for the next release.

In addition, please comment on this issue if you believe we are missing something that is critical for EF Core 6.0, or are focusing on the wrong areas.

[ErikEJ]

Great and ambitious plan, would be fantastic if you could do:

Detect simple join tables in reverse engineering and create many-to-many relationships

early so I can get it in EF Core Power Tools asap. 😎

[Tiberriver256]

JSON Columns has a lot of potential. Are you planning on making them queryable?

*EDIT: Corrected 'JSON Tables' to 'JSON Columns'

[ErikEJ]

@Tiberriver256 What is "JSON Tables" - CosmosDB ??

[Tiberriver256]

@ErikEJ - Typo meant it to be 'JSON Columns'

[aredfox]

To be sure, does this tie into SQLServer JSON storage format? As in: https://docs.microsoft.com/en-us/sql/relational-databases/json/store-json-documents-in-sql-tables?view=sql-server-ver15 How would that syntax look like, just Linq2EF like now?

[domagojmedo]

Currently EF is at 157k RPS and goal is to get as close to 198k as possible? If I read fortunes results right

[Pilchie]

:eyes:

[Wraith2]

SqlServer.Core An experiment in collaboration with the community to determine what potential there is modern .NET features in a highly performant SQL Server driver.

I have pondered whether rewriting the driver from scratch using System.IO.Pipelines would be beneficial. As an individual working on Sql in my spare time it's not a project that is worth starting, I'd never finish. As a group it might stand a chance of getting somewhere.

[ajcvickers]

@Wraith2 Your name has been mentioned as we discussed this. We're certainly hoping you will get involved! :-)

[ajcvickers]

@aredfox Yes, but I'm not sure what it looks like to query. @roji?

@domagojmedo Not sure which round is currently published and whether we are starting from that, or from where we are internally. @roji?

[daniel-frank]

Any plan to support MongoDB?

[davidroth]

Great roadmap! I especially love that you are trying to eliminate bugs and to get query parity with EF6 💯 And of course performance is always appreciated 🚀 The only think that makes me sad is that inheritance for owned types https://github.com/dotnet/efcore/issues/9630 is not in the list. Owned collections are great in the DDD aggregate-children scenario. Not being able to map polymorphic types makes the feature useless in many scenarios.

[ilmax]

Great roadmap, I love it! Any plan to give some love to global query filters? Like the ability to specify and combine multiple global query filters and turn one/some of them off on demand

[mike7ang1rdz]

I'd have issues when my entities have several levels of inheritance and when using methods in the entity

[roji]

Currently EF is at 157k RPS and goal is to get as close to 198k as possible? If I read fortunes results right

Yes, that's right.

Note that you're looking at the TE round 19 results. While this is the latest round, results are from May, so before .NET/EF Core 5.0 were released (round 20 runs are now in progress). This is an up-to-date view of the latest runs.

[roji]

@aredfox re JSON support, we don't have all the specifics yet and need to investigate how JSON support looks like across databases (SQL Server, PostgreSQL, MySQL, Sqlite...). For SQL Server specifically, the idea would indeed to translate LINQ queries to the various JSON functions (JSON_VALUE, JSON_QUERY). Re storage, unless I'm mistaken, SQL Server doesn't have a JSON storage type (unlike PG and some other databases), so storage is typically just done in nvarchar columns. As the SQL Server docs mention, it's possible to decompose your JSON document to a relational schema; we may be able to support that via the new value object / value converters feature also planned for 6.0, but it's really too early to tell.

[Zoxive]

I had a peak through some of the performance backlogs they seem pretty ambitious. I was mostly looking because I was curious if anything mentioned source generators (https://devblogs.microsoft.com/dotnet/new-c-source-generator-samples/) but failed to see it anywhere. Has any thought gone into source generators to convert some of the generated code into compiled code?

[roji]

@Zoxive we may look into source generators for the compiled model feature (not definite).

Other than that, EF Core performs the bulk of its code generation during runtime, using LINQ expression trees (e.g. materialization logic for reading entities from the database); this is quite different from source generators, which works at build-time, so there's less potential there. We do have some ideas in that direction as well, but they're probably beyond the scope of 6.0.

[ajcvickers]

@mike7ang1rdz Please file a new issue and attach a small, runnable project or post a small, runnable code listing that reproduces what you are seeing so that we can investigate.

[kewinbrand]

what about migrations that support async operations?

[erikrenaud]

Great stuff, would love to see more cosmosdb love and issue #9630.

As for graphql, i had high hopes for odata a while back. We need proper DotNet Core support for server side GraphQL (and OData) as well as the client side. Also. this love has to trickle back to other systems at Microsoft (i.e. Power[BI, Flow, Apps] can connect to an OData or GraphQL source).

[khalidabuhakmeh]

I would love to see JSON query support for SQL Server and the ability to project (SQL Server-side) to a JSON object. I could see this bypassing the need for serialization/deserialization to a C# model.

[expcat]

For GraphQL, does the team consider assisting GraphQL.EntityFramework to implement SimonCropp/GraphQL.EntityFramework#145 ?

[ajcvickers]

@kewinbrand MigrateAsync already exists. Is there something else you are looking for?

[ajcvickers]

@khalidabuhakmeh JSON query support is included here. Projecting to JSON is interesting and inspired me to file #23920

[ajcvickers]

@expcat We are still investigating the GraphQL space. If we believe that contributing to an open source project is the best way to move the ecosystem forward, then we will do that.

[kewinbrand]

@ajcvickers what I meant was support async operations in Up/Down methods in Migrations

[ajcvickers]

@kewinbrand The Up and Down methods should not be doing database I/O.

[kewinbrand]

@ajcvickers I see migrations are not designed to access database. But there are cases that I need external data (like making a call to another API, read from distributed cache, etc) to transform the data some way

[erikrenaud]

Great roadmap! I especially love that you are trying to eliminate bugs and to get query parity with EF6 💯 And of course performance is always appreciated 🚀 The only think that makes me sad is that inheritance for owned types #9630 is not in the list. Owned collections are great in the DDD aggregate-children scenario. Not being able to map polymorphic types makes the feature useless in many scenarios.

I'd love to see this feature #9630 make it for 6.0 also... When using CosmosDB, without polymorphic owned types, it becomes diffucult to use the technology correctly.

[sam-wheat]

This is still in the moonshots category. I bring it up again to keep you guys thinking about disruptive ways to make DBMSs better in years to come.

In the year since I first wrote the post above dotnet has only gotten better. Blazer, which runs c# in the browser was once widely ridiculed however many now consider it a viable alternative to javascript.

In the same spirit as Blazer I hope one day to run dotnet in a dbms where my query can be executed without first being translated to SQL or some variant of it. By todays standards its a wild idea. But cars and now electric cars were once wild ideas too.

[naruto1227]

Any plan to support MongoDB?

[ajcvickers]

@zhanghaiboshiwo It's not currently in our plan, but Mongo's strong showing in the recent EF survey means that we are thinking about how best to support it.

[alexmurari]

@ajcvickers Any plans on flexible navigation properties, like this issue?

[ajcvickers]

@alexmurari That issue is in the Backlog milestone. This means that it is not planned for EF Core 6.0.

[alexmurari]

@ajcvickers Yeah, I asked because there are several other issues with the same premise: make navigation properties types more relaxed in relation to it's backing field, not just that issue I mentioned. IIRC there's some other issue around about IImmutableList<T> navs and List<T> backing field or something like that but couldn't find it. Just wanted to know if there's any plan on that direction for 6.0, not just exactly the mentioned issue.

[IgorMenshikov]

Any plans to add query hints to EF Core? I am working on an application with SQL Server as database. Database is quite big and SQL Server started to use wrong execution plans (it uses SCAN instead of SEEK). Only way I found to win it is to use query hints (like WITH(FORCESEEK)).

EF Core does not allow that, so I had to replace Linq on direct SQL queries just because of that.

[ErikEJ]

@IgorMenshikov interceptors?

[IgorMenshikov]

@ErikEJ I know about them but it is very "dirty" way because it requires to work with raw SQL, parse them and change.

I have many queries and idea to hardcode logic at interceptors looks terrible.

Some standard way to use query hints much better.

[ajcvickers]

@IgorMenshikov You can use query tags to identity the queries you need to add hints to, as shown in the docs: https://docs.microsoft.com/en-us/ef/core/logging-events-diagnostics/interceptors#example-command-interception-to-add-query-hints

Full support for query hints is tracked by #6717. Please upvote that issue.

[IgorMenshikov]

@ajcvickers I know that way but as I wrote it is a very dirty way because it requires to manual change raw SQL (and often parse it) with code like

if (command.CommandText.StartsWith("-- Use hint: robust plan", StringComparison.Ordinal))
        {
            command.CommandText += " OPTION (ROBUST PLAN)";
        }

Some hints must be at JOIN part like

FROM Main
JOIN Another WITH(FORCESEEK) ON Another.ID = Main.ID

So, you need to parse SQL and find a place to put hints. Terrible idea.

With my database, SQL Server quire often started to use "index scan" even to find 100s rows at 10M table. I have not found any way SQL Server use seek automatically (rebuilt indexes, stat). Only query hints help.

Interceptors can be used for a single "hack" but it is not something I accept to use on regular basic for many queries.

[WijanRuiz]

Appart from SQL Server are any plans on improve ADO.NET for any other providers like PostgreSQL or MySQL?

[roji]

@WijanRuiz the EF plan doesn't include plans to improve the SQL Server ADO.NET provider (Microsoft.Data.SqlClient) - that is tracked on https://github.com/dotnet/sqlclient. The PostgreSQL and MySQL ADO.NET providers also aren't included in the EF Core plan since they're maintained by other people (outside of Microsoft), so you need to go to their respective repos.

[ajcvickers]

@WijanRuiz Our experience has been that the community-driven ADO.NET providers for PostgreSQL and MySQL are pretty good, both in terms of functionality and perf. Is there anything specific you are looking for?

[WijanRuiz]

@WijanRuiz Our experience has been that the community-driven ADO.NET providers for PostgreSQL and MySQL are pretty good, both in terms of functionality and perf. Is there anything specific you are looking for?

Just asking cause I saw the plan to improve the Microsoft.Data.SqlClient and I didn't know if you want to improve it from the root including ADO.NET, or just the SqlClient for Sql Server. But, it was just a question, I am really satisfied with the perf right now. Thank you.

[ajcvickers]

@WijanRuiz There aren't any plans for the EF team to directly improve Microsoft.Data.SqlClient. We do have plans to experiment with a different approach than Microsoft.Data.SqlClient (SqlServer.Core/Woodstar), but that plan is not intended to improve or replace Microsoft.Data.SqlClient.

[denmitchell]

@ajcvickers, I am very excited about the upcoming support for SQL Server Temporal Tables. Just in case it helps, I have repo with code that generates Temporal Tables via Migrations (EDennis.MigrationsExtensions). My team has been using it for a couple years now. My library may make more assumptions about model structure than yours would, but there might be something useful in the code for you.

[Morgs007]

Great and ambitious plan, would be fantastic if you could do:

Detect simple join tables in reverse engineering and create many-to-many relationships

early so I can get it in EF Core Power Tools asap. 😎

This should be top priority as at the moment, simplest navigation simply fails...

[ErikEJ]

@Morgs007 Nothing should "fail" - if that is the case, please create a new issue with repro details.

[Morgs007]

@Morgs007 Nothing should "fail" - if that is the case, please create a new issue with repro details.

What I meant is that porting from EF to EF Core, most navigations are now errors and requires a complete rewrite to make it work. Which I guess will require another run of rewrite after the Many-to-many eventually become available... I never knew .NET would have such kinds of shortfalls, they framework has always been 100% complete, now, so much is missing