Compiled model performance trends

[ajcvickers]

In investigating the compiled model changes in EF9, I noticed that things were taking quite a lot longer than expected, so I did some analysis.

The model here is the one used in the samples: https://github.com/dotnet/EntityFramework.Docs/tree/main/samples/core/Miscellaneous/CompiledModels

The first issue is that, for this sample model with 449 entity types, 6390 properties, and 720 relationships, the startup time is now worse when using a compiled model.

EF version	Normal (s)	Compiled model (s)
EF Core 6	3.71	0.93
EF Core 7	4.07	1.04
EF Core 8	4.52	3.37
EF Core 9	4.48	5.24

By startup time, I mean wall clock time to run the following:

using (var context = new BlogsContext())
{
    entityCount = context.Model.GetEntityTypes().Count();
}

My suspicion that a lot of this is assembly loading and/or JIT, based on the increase in DLL size across releases:

EF version	Normal (MB)	Compiled model (MB)
EF Core 6	0.9	2
EF Core 7	0.9	2
EF Core 8	0.9	8
EF Core 9	0.9	20

Related to all this is that the time to run dotnet ef dbcontext optimize on this same model has increased very dramatically in EF9.

EF version	Time to run dbcontext optimize (s)
EF Core 6	11
EF Core 7	11
EF Core 8	40
EF Core 9	107

[roji]

Note: see profiling session done by @muzopraha in #33495:

[stevendarby]

This also shows regression for non-compiled models. Is a separate issue needed to track that?

[roji]

@stevendarby yeah, that's true - we'll discuss this in the team as well.

[ajcvickers]

@stevendarby @roji We discussed the regression in 8.0 at the time, and it was considered acceptable because if people had slow model building performance then they could use compiled models! Oh, the irony.

[stevendarby]

@ajcvickers Irony aside, it's also not really true in all cases - for example, if you use query filters?

[ajcvickers]

@stevendarby Agreed.

[iainnicol-hymans]

Is it a reasonable workaround, for now, to compile the model with EF Core 7, but then upgrade packages and run with v8 or v9? It doesn't crash and burn immediately, but I'd like to know if that's a completely unsupported mix. Thanks.

Not sure how surprising this is, but when I encountered this issue, I discovered that AOT doesn't really improve the model init time of compiled models. (Leaving aside the fact queries won't run on AOT).

[AndriySvyryd]

Is it a reasonable workaround, for now, to compile the model with EF Core 7, but then upgrade packages and run with v8 or v9? It doesn't crash and burn immediately, but I'd like to know if that's a completely unsupported mix. Thanks.

No, it's completely unsupported. A "better" workaround would be to use some post-processing script to remove the slow calls added in the newer code. Most of them will be just computed lazily when not using NativeAOT.

[AndriySvyryd]

This is the proposed action plan to deal with this regression:

• [ ] By default generate a compiled model equivalent to what EF Core 6 would generate, with CLI switches that would add more code to the compiled model (For NativeAOT fully eager generation is required)
• [ ] #31370
• [ ] Look for opportunities to generate less code by leveraging defaults (in particular, for provider-specific configuration)
• [ ] Fully qualify (including the return type) the generated lambdas #33773
• [ ] "Lift" reused lambdas, comparers and type mappings to a common location
• [ ] By default, don't use the backing field for non-virtual auto-props
• [ ] Inline property accessors to avoid lambdas with closures (this would require new API on InternalEntityEntry)
• [ ] Don't generate lambda expressions for NativeAOT (#32717)

[TonyValenti]

Is this going to be taken care of for .NET 9.0? This has become almost intolerable for our app. Our app has 100 tables which have a total of 6500 columns.

It is a short-lived application but the initial connection to the database takes 1-5 minutes depending on the hardware specs. That initial connection takes more time than the whole rest of the app.

I am really hopeful this can be given high priority.

[ramonesz297]

@TonyValenti As a workaround you can use something like this:

function CompiledModelFix {
    $file = '.\PathToDbContextProject\CompiledModels\{MyDbContext}ModelBuilder.cs'
    $content = Get-Content $file -Raw

    # Remove specific line
    $content = $content -replace 'AddRuntimeAnnotation\("Relational:RelationalModel", CreateRelationalModel\(\)\);', ''

    # Remove specific method
    $content = $content -replace 'private IRelationalModel CreateRelationalModel\s*\(\)\s*\{[\s\S]*', '} }'

    while ($true) {
        try {
            # Attempt to write to the file with UTF-8 encoding and BOM
            Set-Content -Path $file -Value $content -Force -Encoding UTF8 -ErrorAction Stop

            Write-Host -BackgroundColor Yellow "Fix was applied..."
            break  # Exit the loop if writing is successful
        }
        catch {
            # If writing fails due to file being used by another process, wait and retry
            Start-Sleep -Seconds 1
        }
    }
}

It`s a part of ps1 script (for adding\removing migrations), that removes slow code in CompiledModel.

[TonyValenti]

My leadership would consider that a brittle approach and would not allow us to use that.

I'm really hopeful this gets significant investment.

[TonyValenti]

Hi @AndriySvyryd - Do you know when dev will start on this? Is there anything I can do to help this get prioritized?

I really, really, really want to make sure this gets resolved for 9.0.

[AndriySvyryd]

@TonyValenti I can't tell exactly when or what, but something will be done about this for 9.0