Plan for Entity Framework Core 8 (EF8)

[ajcvickers]

Today we are excited to share with you the plan for Entity Framework Core 8. This issue contains a quick summary of the plan and acts as a place for you to leave feedback.

This plan brings together input from many stakeholders and outlines where and how we intend to invest in Entity Framework Core 8 (EF Core 8; EF8)

IMPORTANT This plan is not a commitment; it will evolve as we continue to learn throughout the release. Some things not currently planned for EF8 may get pulled in. Some things currently planned for EF8 may get punted out.

General information

EF Core 8 is the next release after EF Core 7 and is scheduled to ship in November 2023, at the same time as .NET 8. EF8 currently targets .NET 6. This will likely be updated to .NET 8 as we near the release date. EF8 will align with .NET 8 as a long-term support (LTS) release. See the .NET support policy for more information.

Themes

Large investments for EF8 and data access in .NET 8 fall under the following themes:

Highly requested features

• JSON columns Build on EF7 JSON support to further power the document/relational hybrid pattern.
• Value objects Applications can use DDD-style value objects in EF models.
• SQL queries for unmapped types Applications can execute more types of SQL query without dropping down to ADO.NET or using third-party libraries.

Cloud native and devices

• AOT and trimming with EF Core Small, fast-starting EF Core applications with no dynamic code generation.
• AOT and trimming for ADO.NET Low-level data access can be used in cloud native applications.

Performance

• Woodstar Fast, fully managed access to SQL Server and Azure SQL for .NET applications.

Visual Tooling

• First-class T4 templates in Visual Studio Leverage T4 templating across multiple areas in Visual Studio.
• EF Core Database First in Visual Studio Out-of-the-box Database First tooling in Visual Studio.

Developer experience

• Make EF Core better Improve the developer experience be making many small improvements to EF Core

Find out more and give feedback

This post is a brief summary of the full EF8 plan. Please see the full plan for more information.

Your feedback on planning is important. The best way to indicate the importance of an issue is to vote (👍) for that issue on GitHub. This data will then feed into the planning process for the next release.

In addition, please comment on this issue if you believe we are missing something that is critical for EF8, or are focusing on the wrong areas.

[ErikEJ]

When will you open source Woodstar? Or at least share the code.

[ajcvickers]

@ErikEJ As soon as possible. We're not holding it back for any particular reason, and I imagine @roji will share something pretty soon after he starts working on it.

[ErikEJ]

Great. For some reason I was under the impression that @smitpatel had done a lot already..

[roji]

@ErikEJ yeah, there's already some considerable done, but not yet done... Once we have something that works and is in a state to make public, we'll definitely do that.

[davidroth]

To increase usefulness of DDD aggregate mapping via owned (collection) types, the owned inheritance issue would be really important: https://github.com/dotnet/efcore/issues/9630 😀

But yeah, maybe next time 😅🤞

But beside from that: Great plan as always 👍🚀

[shadzhipopov]

What about recursive queries? In almost every project (if not every) I had to write some recursive CTEs, and it would be really nice if we can move this SQL to code. In SQL server, postgre and oracle, the SQL is almost identical, and I do not expect that every db provider will support them, but I believe this is somewhat missing piece...

Merry Christmas to all

[ajcvickers]

@shadzhipopov Make sure to vote (👍) for #2865.

[janstrube]

I would like to see support for Table Value Parameters (TVP). This can increase performance and simplify business logic significantly when you have to deal with larger client side collections, yet EF has very limited support. The work around using bulk copy inserts into a temp table are sub-optimal.

[ilmax]

Any chance to support global query filters with compiled models in the 8 timeframe?

[Misiu]

Please consider read-only entities (https://github.com/dotnet/efcore/issues/7586).

[ajcvickers]

@janstrube Please vote (👍) for #19016 and/or #13239. It's unlikely these will make it into 8.

@ilmax Please vote for #24897. This might make 8 based on some of the AOT work that is going on--see #29924.

@Misiu Please vote for #7586. It's unlikely this will make it into 8.

[codyburkard]

I would love to see support for managed identities in the CosmosDB provider. For security reasons managed identities are used extensively nowadays, but some apps end up being non compliant to security best practices because they are not supported by CosmosDB EF.

It would be great to see this prioritized!

[ajcvickers]

@codyburkard Is this different to #26491?

[YohanSciubukgian]

Could you consider extending support for more advance CosmosDB SQL API queries ?

[ajcvickers]

@YohanSciubukgian You'll need to be more specific. I suggest you look through issues labeled with "area-cosmos" and "area-query" and vote (👍) for those that you need, or file new, specific, issues for query constructs that you need and are not covered by existing issues.

[zejji]

Since improved JSON/hybrid-relational support is one of the important goals of EF Core 8, I'm just including a cross-reference to #25272, since it means it is not currently feasible to map multiple CLR types to a single database column efficiently. For us, this has limited the utility of the JSON support.

Worth voting for if it affects others...

[bachratyg]

It would make shared/library code more foolproof to use if the context could be configured from the app service provider similar to how IConfigureOptions works. Currently I have to provide multiple extension methods that should be called on the service collection (to add core services), AddDbContext (to add interceptors), OnModelCreating (to add model classes that would show up in migrations) etc.

[ajcvickers]

@bachratyg Can you file an issue for this and include considerably more detail. Ideally, include what the configuration from the service provider would look like, and how the model configuration relates to the DbContext type being configured and the DbSets it exposes.

[bachratyg]

Full proposal here: #30061

[YohanSciubukgian]

@YohanSciubukgian You'll need to be more specific. I suggest you look through issues labeled with "area-cosmos" and "area-query" and vote (👍) for those that you need, or file new, specific, issues for query constructs that you need and are not covered by existing issues.

More or less all the issues punted-for-7 & area cosmos. I have already thumbs up all issues I was interested in. https://github.com/dotnet/efcore/issues?q=is%3Aopen+is%3Aissue+label%3Apunted-for-7.0+label%3Aarea-cosmos

[MgSam]

I'm super excited about the plans you guys have here. A lot of this stuff has been on my wishlist for literally a decade.

WoodStar- would love to hear more about this and offer feedback. ADO.NET is an ancient, clunky beast and I am ecstatic it's finally getting re-written. I think you should consider offering Dapper's functionality out-of-the-box. In addition, would love to see SqlBulkCopy get some love. In an ideal world it would be able to work with DataFrame- but that would require DataFrame actually getting finished. Any chance your team can adopt it?

T4 Tooling- this is super-exciting to hear that you guys are going to improve the T4 tooling. Can you share more details around this? Will it be your team doing this or another team? Some features I'd like to see:

1. In-box syntax highlighting support
2. Runtime T4 templates have generated .cs files which contain comments indicating the filepath of the T4 file they came from- this file path is absolute, which causes a lot of merge conflicts when different team members have repos checked out in different folders. The filepath should either be eliminated or made relative to the sln file
3. Razor syntax being supported in T4 would be amazing.

Database-first- I think it's fantastic you're going to ship this out-of-the-box. I've always advocated that for large enterprises, code-first is often a non-starter.

[roji]

@MgSam

WoodStar- would love to hear more about this and offer feedback. ADO.NET is an ancient, clunky beast and I am ecstatic it's finally getting re-written.

WoodStar would be about providing a fast, modern driver for SQL Server and SQL Azure, not about rewriting ADO.NET.

[ErikEJ]

@MgSam I have published a very popular extension for reverse engineering - EF Core Power Tools - since 2017...

[MgSam]

@MgSam

WoodStar- would love to hear more about this and offer feedback. ADO.NET is an ancient, clunky beast and I am ecstatic it's finally getting re-written.

WoodStar would be about providing a fast, modern driver for SQL Server and SQL Azure, not about rewriting ADO.NET.

Isn't that just semantics though? If EF starts using WoodStar and we can finally stop using some of these core types in ADO.NET that haven't been touched in 15 years.

@MgSam I have published a very popular extension for reverse engineering - EF Core Power Tools - since 2017...

Yes, I think it's great! But I think out-of-the-box is critical for widespread adoption of any feature. I think most devs don't install any extensions, or worse have corporate policies actively banning them.

[ErikEJ]

@MgSam I hope woodstar will just eventually be a better ADO.NET driver for Azure SQL / SQL Server...

[roji]

WoodStar would be about providing a fast, modern driver for SQL Server and SQL Azure, not about rewriting ADO.NET.

Isn't that just semantics though?

It's definitely not semantics - replacing ADO.NET means designing a new database abstraction API for .NET, which would need to make sense for all databases; the effort for Woodstar really is to provide a faster/more modern SQL Server driver. It's true that we're experimenting with non-ADO.NET API surfaces, but ultimately the driver will definitely be accessible via ADO.NET, otherwise it would be incompatible with all existing upper layers (EF, Dapper...).

If EF starts using WoodStar and we can finally stop using some of these core types in ADO.NET that haven't been touched in 15 years.

I'm not following; you can already use EF today with SqlClient and not have any contact with ADO.NET. If EF supports Woodstar in the future, it would look more or less identical to what EF looks like for current users of SQL Server. The advantage would be in performance, bug fixing and specific features, not in a new API surface.

[MgSam]

I'm not following; you can already use EF today with SqlClient and not have any contact with ADO.NET. If EF supports Woodstar in the future, it would look more or less identical to what EF looks like for current users of SQL Server. The advantage would be in performance, bug fixing and specific features, not in a new API surface.

Everything in EF ultimately uses SqlConnection/DbConnection when it comes down to it, no? Which is part of ADO.NET and is limited by cruft and design decisions made 15 years ago. My hope was that this project would be an investment for more modern types at the base of the db stack. SqlBulkCopy being another huge one. If that's not the case, that's a shame but I'd still be interested in learning more.

I've not yet experimented with the more recent work you guys did around bulk operation support but traditionally for the whole lifetime of Entity Framework / EF Core, if you needed to insert large amounts of data you had to use SqlBulkCopy or else suffer massive performance penalties. So EF + SqlBulkCopy has always been the reality of working with data for me in .NET. That said, SqlBulkCopy has always been a very broad hammer, it works with DataTables, has no generic object support, doesn't directly support more advanced insertion strategies like Merge. So I've always ended up wrapping SqlBulkCopy to provide additional functionality.

[roji]

Everything in EF ultimately uses SqlConnection/DbConnection when it comes down to it, no? Which is part of ADO.NET and is limited by cruft and design decisions made 15 years ago. My hope was that this project would be an investment for more modern types at the base of the db stack.

Yes, everything in EF ultimately uses ADO.net under the hood, but that doesn't really matter to anyone using EF. In other words, if we had a super nice ADO.NET replacement tomorrow, EF would still look exactly the same as it's layered on top and has its own API.

So if you're using ADO.NET directly (without EF), then it's certainly understandable that you'd want something newer to replace it. But otherwise you shouldn't really care what ADO.NET looks like.

SqlBulkCopy specifically is not part of ADO.NET at all - it's a completely SqlClient-specific API, specific to SQL Server. Other database drivers have bulk import/export APIs which look completely different.

We do intend to add a bulk import to EF, whose SQL Server implementation would be layered on top of SqlBulkCopy (see #27333); that would allow you to use bulk-insert regular entities but with SqlBulkCopy performance.

[...] it works with DataTables [...]

You can use SqlBulkCopy without DataTables - IIRC it can be used with any DbDataReader, which can be an abstraction for wrapping anything (and in that sense can serve a similar function as data frames, in this very specific sense).

[...] doesn't directly support more advanced insertion strategies like Merge [...]

That's very likely to be a limitation of SQL Server itself; bulk copy features in database are usually about inserting bulk data into a table in the fastest way possible; MERGE is something very different. A typical pattern here is to efficiently bulk insert via SqlBulkCopy into a temporary table, and then MERGE that into your destination.

[programcsharp]

Coming here to ping second-level caching as an important feature. I commented with more detail on the backlogged specific 2lc issue here: https://github.com/dotnet/efcore/issues/27735#issuecomment-1443538978

This is the one big issue keeping us (and probably many others) from considering EF as an option. After experiencing the benefits good caching when scaling up, it becomes table stakes when choosing an ORM. Output caching is what you want for public facing apps like social media, but for SaaS apps that are mostly CRUD that doesn't work -- you really need second level caching to perform well. I check back every couple of years to see if EF is considering it, then have to go back to our hand built ORM instead because caching is so important :-/

[roji]

@programcsharp sure thing. I'm still not convinced that caching should be an internal concern of the ORM - as opposed to a layer above it or a plugin via interception - but that conversation is ongoing in #27735. Let's continue it there.

[arthurlomakin11]

What about recursive queries? In almost every project (if not every) I had to write some recursive CTEs, and it would be really nice if we can move this SQL to code. In SQL server, postgre and oracle, the SQL is almost identical, and I do not expect that every db provider will support them, but I believe this is somewhat missing piece...

Merry Christmas to all

You can use linq2db library with EF core for CTE queries. And it's concept can be taken for EF Core itself later

[abratv]

@roji is multi tenancy being considered for woodstar? We sharded the database/customer

[roji]

@abratvisma not at this point. Woodstar is currently a high-perf oriented experiment rather than something targeting full, mainstream usage right away.

[marchy]

To increase usefulness of DDD aggregate mapping via owned (collection) types, the owned inheritance issue would be really important: #9630 😀

But yeah, maybe next time 😅🤞

But beside from that: Great plan as always 👍🚀

Fully agree here, inheritance is more important than ever for modelling domains, with modern type hierarchies being defined for all sorts of domain concepts that have natural variances (ie: Swift's associative enums which generate class hierarchies, Kotlin's data classes, type unions / discriminating unions proposed for C# V-Next etc.). Owned types are supposed to be transparent pass-throughs, but this limitation places a severe restriction on what they can be used for – needing to fall back to manual parsing (ie: JSON strings) for any structure that requires more complexity.

[wisamidris7]

Make Merge Migration Command

Like Merge-Migration 1 4

This Mean From 1 4 4 Migrations Will Be Merged To Be Easy To Me Developer To Also The Table In Database EFMigrations

Make In New Mergred Migration builder.RemoveMigration("??????");

Cause I Go Manually In Databases Cause I Have Multitenant App So This Will Be So So Aesome Update For EF8

[wisamidris7]

builder.RemoveMigration("????"); // This Mean Remove The Row Migration From EFMigrations Table
builder.CreateTable(????);

This Will Be Cool

[ajcvickers]

@wisamidris7 This is tracked by #2174. Please vote (👍) for that issue.

[wisamidris7]

@wisamidris7 This is tracked by https://github.com/dotnet/efcore/issues/2174. Please vote (👍) for that issue.

This Will Be Very Very Nice Feature

[wisamidris7]

You Can Make It Without Voting Cause My App Has 74 Migration This Just For Now And This To To Bad And I Have Many Clients

[MichelZ]

Yeah, giving migrations some love would be very nice

[brunom]

Please use JSON to avoid the cartesian explosion of including related data.

[roji]

@brunom that's #30958, which won't be happening for 8.0. But please up-vote it to show your interest.

[ErikEJ]

@roji My semi-annual update: Any news and or timelines to share regarding Woodstar?

[roji]

@ErikEJ not yet, promise to post something in a month or two once we're past the intensive .NET feature-work period.

[ErikEJ]

@roji Cool. Highly appreciated.

[wisamidris7]

Also int.Parse() Feature Will Be Nice

[gsGabriel]

The EF is based in UnitOfWork and repository pattern and this is done through the DbContext class. But if I want to expose this implementation through an interface to other layers, we lose some features such as the CompiledQueries or ContextFactory because a concrete class of DbContext is required. Perhaps the API, instead of requiring a concrete DbContext class, would require an IDbContext interface so that when we expose the interface that implements the Context, it can take advantage of these features.Would it be nice to have interfaces instead of concrete classes, or do you have a specific reason for that?

Another thing I would like to know, thinking about crud systems with pagination pages, many times we need to make two queries to assemble the pagination structure. A query to get the total and another to get the paginated list, however, the DbContext is not thread-safe and we cannot do both operations at the same time. Wouldn't it be possible to create a DBContext scope that would allow this concurrent execution?

[ajcvickers]

@gsGabriel Re interfaces, we recommend you create your own interfaces. See discussion here: #16470. Re concurrent execution, we could, but it would almost always be the wrong thing to use.

[gsGabriel]

@ajcvickers I agree, but many times we need to get small pieces of data to compose an aggregate, I think that these multiple pieces could happen asynchronously.

What about queries with multiple result sets? could we have something in the future? so we could compose these queries and return them in one go. I would still prefer to allow concurrent executions but it would help in most scenarios

[ajcvickers]

@gsGabriel Multiple queries in a single round-trip is tracked by #10879.