search

dotnet / efcore

EF Core is a modern object-database mapper for .NET. It supports LINQ queries, change tracking, updates, and schema migrations.
https://docs.microsoft.com/ef/
MIT License
13.75k stars 3.18k forks source link

Reduce EF Core application startup time via compiled models #1906

Closed mikary closed 3 years ago

mikary commented 9 years ago

This an "epic" issue for the theme of generating a compiled model with fast loading and access times. Specific pieces of work will be tracked by linked issues.

Proposed for 6.0

Used by calling new tool commands (all parameters are optional)

dotnet ef dbcontext optimize -c MyContext -o MyFolder -n My.Namespace
Optimize-DbContext -Context MyContext -OutputDir MyFolder -Namespace My.Namespace

Backlog

elksson commented 4 years ago

@jespersh what would be the suggestion when my azure function app uses shared EFModel that is used in many of our other applications. This model has many tables. How can I re-use the same EF Model but only compile the tables which are required?

jespersh commented 4 years ago

@elksson I think you need to rethink how you're doing Azure Functions. If you don't want to split up, the application, then might I suggest you split up the dbcontext into multiple to your specific function needs?

elksson commented 4 years ago

@jespersh i generally would agree with your comment except that we have some functions which would then require to do joins from more than 1 context. And while each context would likely be in the same database just a different schema. There is not a way to make EF generate a cross context dB query when those queries are on the same dB. This means that the query will have to pull back many values and filter further in memory. Becoming highly inefficient than just doing a multi-schema db query.

Are you aware of any solutions to this problem?

StrangeWill commented 4 years ago

Yeah splitting up the context is a non-solution, that's basically making our solutions worse because the framework won't provide a viable solution for a pretty common problem-space.

If you work on any large ERP, God help you and your 45+ second startup times all coming down to EF Core. Even stuff like Doctrine allows us to have a caching layer that we can offload to Redis (while not ideal because I'd still have a massive warm-up time after first deploy after a schema change, it's better than this).

marcelgoldstein commented 4 years ago

We also have a long startup time for our application (wpf fat-client) duo to the warmup in EntityFrameworkCore. Subscribed to this issue a long time ago in hope of a solution coming by, please prioritize.

eostarman commented 4 years ago

We have a model that contains 1,149 entities (with 16,607 properties). It has 3,298 relationships. And it takes 2:24 minutes to get a result back from the first query (I instantiate the context, then use it in a small console mode app). I can provide the model (privately) if that'll help track what's going on.

mimranfaruqi commented 4 years ago

This is a very old opened issue. More than 5 years now!!! I don't know if I should switch to something else as an alternative to EF Core. My tables does not have enough data, yet query takes 26 seconds. Only solution is to get things separately in different queries but that would be problematic when tables would be huge.

roji commented 4 years ago

@mimranfaruqi this feature (compiled models) is about speeding up application startup only. If your query is always running slowly, this issue will not help you - please open a new issue with a runnable sample demonstrating the problem.

GrimaceOfDespair commented 4 years ago

This issue has a lot of history around it and five years of discussion in the team. [...] It's an involved and complex change with many considerations.

I appreciate the complexity of something like this, but this is biting applications out there for quite some time.

Cutting up the application in separate contexts doesn't feels like an actual solution - but it's the only work-around that exists. One that could actually drive users to other places. It's the choice between investing in something you don't want to do versus moving to an alternative that works as expected.

The absence of any decent workaround is a tough sell for anyone trying to advocate EF core in an enterprise.

I totally understand there's priorities, but this feels like we get a muscle car with only 2 gears: we can show it to our friends, but forget about taking it on the highway.

Kashif-datamagnetics commented 4 years ago

Any update on this? Will this be added in ef core 5.0? Thanks

AndriySvyryd commented 4 years ago

@Kashif-datamagnetics This issue is in the backlog milestone, meaning that it hasn't been planned for a specific release and work hasn't been started.

failwyn commented 4 years ago

This issue has been in the backlog for 5 years and makes the product neigh unusable for large production applications! It also extends development times by making you wait 3 minutes every time you start the application for debugging...

This issue has officially become a joke...

PeteX commented 4 years ago

This issue makes EF Core very difficult to use with "serverless" hosting solutions. If the application is going to have occasional cold starts, it's vital that it starts quickly. The startup time for ASP.NET Core generally has come down a lot with recent releases, which is fantastic, but now we're finding that the delay comes when we make the first EF query. We want to move as many of our workloads as possible to serverless because of the economics, and to minimise our contribution to climate change (underutilised servers mean wasted power).

I realise there are lots of priorities for the team, so I hope this doesn't seem demanding, I just wanted to point out that this is a problem with serverless as this doesn't seem to have come up in the discussions before.

rezabay commented 4 years ago

The long term solution to solve this issue is to use Source Generator link. Regarding using EF core in microservices, I think it would be better to use something like Dapper rather than a complex ORM.

AndriySvyryd commented 4 years ago

@rezabayesteh Source generators don't allow executing code to build the model, see https://github.com/dotnet/roslyn/issues/45060

rezabay commented 4 years ago

@AndriySvyryd Thank you I got the limitation now.

stevendarby commented 4 years ago

Apologies if this is obvious, but I didn’t see it mentioned already.

I found that Debug level logging significantly slowed down initial model creation because EF logs a lot during model creation (at least for my particular model). Lowering this to Information made a significant improvement while still retaining the logging of SQL statements (which I find essential during development).

So while this obviously isn’t a complete solution for the problem, if you happen to develop with Debug level logging (perhaps unthinkingly) try lowering it to make the frequent restarts during development slightly less painful.

GrimaceOfDespair commented 3 years ago

I really start to wonder whether my bet on EF Core was the wrong one - a bet based on MS backing and the trust dotnetcore was able to build.

I realize that the library came a long way, from being a Hello World ORM for several versions, to being an actual alternative to NHibernate - though ever far from feature par. Now, I have reached back to this ticket in the past two years to check whether there was any way to get past that initial performance hit. There is just none. NONE!

For over 5 years, this has been considered an acceptable annoyance among other backlog items. Now it ended up on the 6.0.0 backlog and I can only hope that this time, it will be picked up. But even if it is, I guess we'll not have anything concrete in our hands within the next year. How on earth is all this compatible with being a serious ORM contender?

So, all ranting aside, does the current status give any perspective on a solution and what would be the timeframe? 'Cause I'm at the brink of switching to NHibernate...

StrangeWill commented 3 years ago

I mean the hard reality of this is: we write a lot of apps, and we only ran into this for some of the worst platforms we've worked with (and even then we found workarounds).

A single monolith managing over a thousand tables is questionable architecture at best, and yeah our entire frustration was "we're plugging EF Core into inflexible/poorly designed ERP systems", we worked around it, it was generally fine. If you're a developer of one of these platforms, I get it, you're stuck with it, sorry.

I completely understand why this isn't a priority, we haven't run into this problem since, so I can't really try to guilt-trip the EF Core team for not focusing on this.

The ridiculous performance benefit we get from EF Core outweighs going back to any older-school ORMs, this issue only bites us on a single-digit number of projects at this point, it's rough if you're one of those projects, and maybe they'll actually fix this in v6, but I also understand why it's a much lower priority vs. features they've been adding that we do use every day.

GrimaceOfDespair commented 3 years ago

Ok, now I'm truly embarrassed. Your remark about thousands of tables triggered me to do another profiling session - given that I'm just handling at most hundred of them. And what do you know ... the hotspots were not in EF code.

I investigated this previously, but apparently never in the right way. I feel stupid enough not to post this, but it's only fair. The hit for those hundred tables was actually very acceptable. :facepalm:

I guess I can only say thank you for taking the time to respond to my rant the way you did :roll_eyes:

ajcvickers commented 3 years ago

@GrimaceOfDespair Glad you were able to get unblocked. With regard to:

alternative to NHibernate - though ever far from feature par.

It would be great to know which features of NHibernate (that you use when you use NHibernate) are not available in EF Core?

GrimaceOfDespair commented 3 years ago

It would be great to know which features of NHibernate (that you use when you use NHibernate) are not available in EF Core?

It's been a while since I've used NH, but I've always kept an eye open for EF on the sideline. Even in the days of edmx pain (who has not cursed on out of sync edmx?), I really tried to use EF. But for a long time, I've felt NH to be superior, albeit at the cost of complexity.

So, to answer your question, I have to retrofit what I still know.

NHibernate is particularly known for its ability to map to about any legacy db. I think that's mostly due to its flexible collection mapping. Especially when you have a relation with an attribute, like with Indexed collections or have to deal with ordering (bag) or dictionaries (map), it really shines. NH Collections allow you to not worry about persistence where you don't want to.

In that context, I've also experienced a lot of pain with the EF first level cache. Maybe it has changed since I last tried, but I was unable to clear the session context of all entities. This made it particularly hard to manipulate - again - incoming collections that had to be updated base on some element criteria.

And then there's second level cache, coming out of the box. Yes, you can abuse this to death, but if applied right, that's one more piece of cache that you don't have to write yourself - with out-of-the-box invalidation!

These are the things that come to mind without attacking my memory too much.

ilmax commented 3 years ago

Hey @ajcvickers are you aware of this one? It's a comparison mostly in terms of DDD capabilities, but it's a great start /cc @vkhorikov

vkhorikov commented 3 years ago

@ilmax Thanks for tagging me. @ajcvickers that course is the collection of pain points that myself and my teams have been experiencing when trying to work with EF Core. They are the primary reason why I push for NHibernate in all of my projects.

pantonis commented 3 years ago

Are you planning to add support for GlobalQueryfilters queries? It is a very powerful feature used by most modern systems.

AndriySvyryd commented 3 years ago

@pantonis We'll try to add support for at least some. Tracked by https://github.com/dotnet/efcore/issues/24897

JanKotschenreuther commented 3 years ago

I have just read the Blog-Post https://devblogs.microsoft.com/dotnet/announcing-entity-framework-core-6-0-preview-5-compiled-models/# which mentions that "Lazy loading proxies" and "Change tracking proxies" are not supported.

Are there any plans to improve the startup/initialization with lazy loading and change tracking enabled or is this the final state? Those features are pretty important to us.

We got a model with 843 entity types (656 tables and 187 views), 12290 properties and 1137 foreign keys. Sorry, i cannot share the model. We are using EF Core 5.0.5. Instanciating the model the first time takes just 2 seconds or less.

I have been observing 90 seconds of initialization on the DbContext when reaching the first LINQ-Query. Disabling lazy loading reduced initialization to 60 seconds. So just adding the lazy loading proxies added 30 seconds.

At the moment I am observing other timings for the same code. The model has increased with more tables and views. But at the moment I am observing 45 seconds for initialization with lazy loading. Without lazy loading it only took around 7 seconds. This observation is a bit weird and i cannot serve any information how this could have improved, because the code did not change and we did not try out previews. Maybe due to some updates, I am not sure. However our production servers still take 90 seconds.

Executing the same query a second time results in usual nice query times of half a second.

We are using dependency injection for DbContext in our ASP.NET Core 5 application which makes a lot of use of lazy loading and change tracking. We were able to observe the same slow initialization timings in simple console applications.

Making use of DbContext-Pooling to decrease instanciating time is nice, but 2 seconds do not bother us as much compared to 45 or 90 seconds for initialization, the initialization is the real show-stopper here.

AndriySvyryd commented 3 years ago

@JanKotschenreuther See https://github.com/dotnet/efcore/issues/20135 and https://github.com/dotnet/efcore/issues/24902

ZvonimirMatic commented 3 years ago

I can't get compiled models working when I'm using ValueConverters. I have the following value converter:

public class TrimConverter : ValueConverter<string, string>
{
    public TrimConverter() : base(x => x.Trim(), x => x.Trim()) { }
}

When I'm not using the converter, running dotnet ef dbcontext optimize ... works as expected. When I'm using the converter, I get the following error: The property 'MyEntity.MyProperty' has a value converter configured. Use 'HasConversion' to configure the value converter type.

I do not understand what is wrong from the message. Is this somehow related to #24896?

AndriySvyryd commented 3 years ago

@ZvonimirMatic How do you configure the property to use the converter?

It should be something like propertyBuilder.HasConversion(typeof(TrimConverter), null)

ZvonimirMatic commented 3 years ago

@AndriySvyryd

I configured it using propertyBuilder.HasConversion(new TrimConverter());. After seeing your reply, I tried changing it to propertyBuilder.HasConversion(typeof(TrimConverter), default(ValueComparer)); (when I tried using null instead of default(ValueComparer), the method call was ambiguous).

Now I get the following error: The property 'MyEntity.MyProperty' is of type 'string' which is not supported by the current database provider. Either change the property CLR type, or ignore the property using the '[NotMapped]' attribute or by using 'EntityTypeBuilder.Ignore' in 'OnModelCreating'.. I tried doing that for multiple properties, but I always get the same error ("The property '' is of type '' which is not supported by the current database provider. ...".

AndriySvyryd commented 3 years ago

@ZvonimirMatic Could you file a new issue with a small repro project so we can investigate?

ZvonimirMatic commented 3 years ago

@AndriySvyryd Sure, no problem. Do you want me to create a repository with the repro project or do you want me to just copy and paste file contents in the issue description?

AndriySvyryd commented 3 years ago

@ZvonimirMatic You can paste them if they are small enough

ZvonimirMatic commented 3 years ago

@AndriySvyryd Filed the issue #25187.

julielerman commented 3 years ago

Recommendation (shall I make it an issue?): require output directory. All of those generated files landing in the main project folder is an easy mistake to make and a hard one to clean up after.

ajcvickers commented 3 years ago

@julielerman Covered by #25059

Luigi6821 commented 3 years ago

Hi, about the topic I would like to know if we can expect on next EF Core releases a feature that allows to make persistent the finalized model. This feature could be used to speedup the startup process like in the following example code:

IModel model;

if (!FirstStartup()) { modelBuilder.Entity() // Model definitions... model = modelBuilder.FinalizeModel(); SerializeModelForUsingOnNextStartup(model) // This serialize the model and make it persistent for next startup load } else model = DeserializeModelFromFirstStartup();

optionsBuilder.UseModel(model)

Thanks in advance Luigi

manigandham commented 3 years ago

@Luigi6821 There's a blog post showing how compiled models will work with EF Core 6.0: https://devblogs.microsoft.com/dotnet/announcing-entity-framework-core-6-0-preview-5-compiled-models/

Luigi6821 commented 3 years ago

@Luigi6821 There's a blog post showing how compiled models will work with EF Core 6.0: https://devblogs.microsoft.com/dotnet/announcing-entity-framework-core-6-0-preview-5-compiled-models/

The solution proposed, for what I understood, is not usable in some scenarios like the one I am using. The mapped entities and properties could be mapped differently depending on db schema version to which context is connecting to. Creating a static model version at compile time is not usable when I create app for different schema version. Maybe I didn't fully understand the proposed solution. Can you suggest me what I can do in the scenarios I described? Thanks id advance

roji commented 3 years ago

@Luigi6821 please open a new issue with a precise description of what you're trying to do. EF Core's model does not rely on your database schema, only on your code; it's is normally built during application startup without connecting to the database.

ZvonimirMatic commented 3 years ago

@AndriySvyryd @roji

I tried upgrading to rc1, but now I get this error which is somehow connected to EFCore.NamingConventions package (that's why i'm mentioning @roji). I'm using preview 3 version since it's the latest one.

So far I was using preview 7 version of EF Core and it was working perfectly, except the value converters which is supposed to be fixed with rc1. I upgraded all the EF Core packages to rc1 and tried to run the same optimize command as I was using before (without changing anything in the model) and I got this error:

Method 'GetServiceProviderHashCode' in type 'ExtensionInfo' from assembly 'EFCore.NamingConventions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7' does not have an implementation.

This is the whole log:

Build started...

Build succeeded.
System.TypeLoadException: Method 'GetServiceProviderHashCode' in type 'ExtensionInfo' from assembly 'EFCore.NamingConventions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7' does not have an implementation.
   at EFCore.NamingConventions.Internal.NamingConventionsOptionsExtension.get_Info()
   at Microsoft.EntityFrameworkCore.DbContextOptions.GetHashCode()
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd[TArg](TKey key, Func`3 valueFactory, TArg factoryArgument)
   at Microsoft.EntityFrameworkCore.Internal.ServiceProviderCache.GetOrAdd(IDbContextOptions options, Boolean providerRequired)
   at Microsoft.EntityFrameworkCore.DbContext..ctor(DbContextOptions options)
   at InCore.Domain.EFCore.InCoreContext..ctor(DbContextOptions options) in C:\projects\incubis-software\incubis\backend\Libraries\InCore\InCore.Domain\EFCore\InCoreContext.cs:line 37
   at Insurance.Domain.InsuranceContext..ctor(DbContextOptions`1 options) in C:\projects\incubis-software\incubis\backend\Domains\Insurance\Insurance.Domain\InsuranceContext.cs:line 15
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass2_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetServiceOrCreateInstance(IServiceProvider provider, Type type)
   at Microsoft.EntityFrameworkCore.Design.Internal.DbContextOperations.<>c__DisplayClass21_3.<FindContextTypes>b__11()
   at Microsoft.EntityFrameworkCore.Design.Internal.DbContextOperations.CreateContext(Func`1 factory)
   at Microsoft.EntityFrameworkCore.Design.Internal.DbContextOperations.CreateContext(String contextType)
   at Microsoft.EntityFrameworkCore.Design.Internal.DbContextOperations.Optimize(String outputDir, String modelNamespace, String contextType)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OptimizeContextImpl(String outputDir, String modelNamespace, String contextType)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OptimizeContext.<>c__DisplayClass0_0.<.ctor>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.Execute(Action action)
Method 'GetServiceProviderHashCode' in type 'ExtensionInfo' from assembly 'EFCore.NamingConventions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7' does not have an implementation.

Do you have an idea where the issue might be?

ErikEJ commented 3 years ago

@ZvonimirMatic This: https://github.com/dotnet/efcore/issues/26022

ZvonimirMatic commented 3 years ago

@ZvonimirMatic This: #26022

@ErikEJ Thanks!