EF Migrations assembly build time went from ~30 seconds to ~4 minutes after installing VS 16.8

[augustoproiete]

I have a solution with projects targeting netstandard2.1 and netcoreapp3.1 that takes about ~40 seconds to run a full rebuild on Visual Studio 16.7 and prior versions.

After installing Visual Studio 16.8 - and without making any changes to any of the projects, the build time now increased to about 4 minutes :eyes: but only when building via Visual Studio... If I run dotnet build on the solution, I get back to the ~40 seconds to run.

I was able to narrow it down to the assembly that contains the EF database migrations. If I exclude this project from the build, it finishes in less than 10 seconds (which means the EF migrations assembly takes ~30 seconds on VS 16.7 and prior).

VBCSCompiler appears to be hanging for a long time, but eventually the build finishes.

Is this a known issue? I was able to find some comments about analyzers slowing the build (i.e. analyzers running on the EF migrations generated code) could that be the case, and what is the workaround (if any)?

---

DatabaseModels.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>netstandard2.1;netcoreapp3.1</TargetFrameworks>
    <RuntimeIdentifiers>win-x64</RuntimeIdentifiers>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.EntityFrameworkCore" Version="3.1.9" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="3.1.9">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.EntityFrameworkCore.Relational" Version="3.1.9" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Proxies" Version="3.1.9" />
    <PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="3.1.4" />
  </ItemGroup>
</Project>

---

DatabaseMigrations.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFrameworks>netstandard2.1;netcoreapp3.1</TargetFrameworks>
    <RuntimeIdentifiers>win-x64</RuntimeIdentifiers>
  </PropertyGroup>

  <ItemGroup>
    <ProjectReference Include="..\DatabaseModels\DatabaseModels.csproj" />
  </ItemGroup>

</Project>

---

global.json

{
  "sdk": {
    "allowPrerelease": false,
    "version": "3.1.100",
    "rollForward": "latestFeature"
  }
}

---

EF Core version: 3.1.9 Database provider: Microsoft.EntityFrameworkCore.NpgSql 3.1.4 Target framework: .NET Core 3.1 Operating system: Windows 10 IDE: Visual Studio 2019 16.8

[ajcvickers]

@roji Analyzers?

[roji]

Odd... if this were an analyzer issue, I wouldn't expect it to build any differently between VS and the command-line... All the analyzer-related perf issues unsurprisingly affected both.

Can you please try on VS on another machine, to rule out some VS state issue? If you're seeing the same thing on another machine, can you please put together a code sample that shows this?

[augustoproiete]

@roji I can reliably reproduce in other machines. What would be a good email address to share a repro project privately?

[roji]

You can send it to the address on my github profile - any cleanup to make the code repro minimal would be greatly appreciated!

[augustoproiete]

@roji Perfect, thanks. I've just sent you and invite to join a private repo with a minimal project that contains a small number of (anonymized) EF migrations that I believe can reproduce the problem on a smaller scale.

dotnet build (~11 seconds)

C:\repro-dotnet-efcore-issue23291>git clean -fxd

C:\repro-dotnet-efcore-issue23291>dotnet build my-app.sln
Microsoft (R) Build Engine version 16.7.1+52cd83677 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  Restored C:\repro-dotnet-efcore-issue23291\src\MyApp.DataModels.Migrations\MyApp.DataModels.Migrations.csproj (in 618 ms).
  2020-11-12_214144 - BeforeBuild. TFs: netstandard2.1;netcoreapp3.1, TF: netcoreapp3.1
  2020-11-12_214144 - BeforeBuild. TFs: netstandard2.1;netcoreapp3.1, TF: netstandard2.1
  MyApp.DataModels.Migrations -> C:\repro-dotnet-efcore-issue23291\src\MyApp.DataModels.Migrations\bin\Debug\netcoreapp3.1\MyApp.DataModels.Migrations.dll
  2020-11-12_214153 - AfterBuild. TFs: netstandard2.1;netcoreapp3.1, TF: netcoreapp3.1
  MyApp.DataModels.Migrations -> C:\repro-dotnet-efcore-issue23291\src\MyApp.DataModels.Migrations\bin\Debug\netstandard2.1\MyApp.DataModels.Migrations.dll
  2020-11-12_214155 - AfterBuild. TFs: netstandard2.1;netcoreapp3.1, TF: netstandard2.1
  2020-11-12_214155 - AfterBuild. TFs: netstandard2.1;netcoreapp3.1, TF:

Visual Studio (~50 seconds)

1>------ Rebuild All started: Project: MyApp.DataModels.Migrations, Configuration: Debug Any CPU ------
1>2020-11-12_214542 - BeforeBuild. TFs: netstandard2.1;netcoreapp3.1, TF: netstandard2.1
1>2020-11-12_214542 - BeforeBuild. TFs: netstandard2.1;netcoreapp3.1, TF: netcoreapp3.1
1>MyApp.DataModels.Migrations -> C:\repro-dotnet-efcore-issue23291\src\MyApp.DataModels.Migrations\bin\Debug\netcoreapp3.1\MyApp.DataModels.Migrations.dll
1>2020-11-12_214627 - AfterBuild. TFs: netstandard2.1;netcoreapp3.1, TF: netcoreapp3.1
1>MyApp.DataModels.Migrations -> C:\repro-dotnet-efcore-issue23291\src\MyApp.DataModels.Migrations\bin\Debug\netstandard2.1\MyApp.DataModels.Migrations.dll
1>2020-11-12_214632 - AfterBuild. TFs: netstandard2.1;netcoreapp3.1, TF: netstandard2.1
1>2020-11-12_214632 - AfterBuild. TFs: netstandard2.1;netcoreapp3.1, TF:
========== Rebuild All: 1 succeeded, 0 failed, 0 skipped ==========

[joakimriedel]

On a side note, I have found that adding the following to an .editorconfig file in the Migrations folder does wonder with Visual Studio responsivity (both IDE and build) when there are many migrations. Perhaps it helps in your case as well?

# All files
# Sets generated code for all migrations
[*]
generated_code = true

[indifit]

I have a development project with 226 Migrations at current count and can confirm that since updating to VS 16.8 my build time has dramatically increased too.

[roji]

I can confirm this repros for me as well. Some notes:

• Visual Studio 16.7.8 does not have this issue, 16.9.0 preview1 does have it (I didn't test 16.8.0 but it's reported by users above).
• The repro uses EF Core 3.1 on netcoreapp3.1 with a 3.1 SDK, so this is not a 5.0-related regression.
• The repro includes around 60 migrations, each with a context snapshot weighing around 600K (full of fluent APIs). We've indeed seen cases of severe build time regressions when analyzers mistakenly processed big migration snapshots; my guess would be that something changed between VS 16.7 and VS 16.8, which incorrectly causes analyzers to process these files, even though they have // <auto-generated /> at the start (the EF analyzer opts out of generated files).

@augustoproiete just to confirm, is it OK if I share your repro code internally within Microsoft as needed?

[augustoproiete]

@joakimriedel Thanks for the tip! I tested it on my project, and it helped to bring the build time down a little bit (~40 seconds out of ~4.5 minutes), so it seems that the .editorconfig with the generated_code marker had an effect on some analyzers, at least.

[augustoproiete]

@roji Thanks for looking into this so quickly and I'm glad you can reproduce the issue. Yes, it's OK to share the repro code within Microsoft.

[roji]

@augustoproiete thank you! We'll post back here as soon as we have something.

[roji]

Probable dupe: https://developercommunity.visualstudio.com/content/problem/1253845/visual-studio-2019-version-1680-very-slow-to-build.html

[rosdi]

We are having this issue across our entire team of 5 developers, so this is not a machine specific issue. Build time that normally takes about 30-40 seconds now went up to 20 minutes!

A single edit in any file (that will cause the project to be recompiled), will force us to wait for ~20 minutes before it is finished.

I added <RunAnalyzersDuringBuild>false</RunAnalyzersDuringBuild> in my .csproj file but that does not make a difference.

Disabling Resharper also does not make a difference.

Here is my csproj file:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <WarningsAsErrors>NU1605;CS0108;CS1572;CS1998</WarningsAsErrors>
    <RunAnalyzersDuringBuild>false</RunAnalyzersDuringBuild>
  </PropertyGroup>

  <ItemGroup>
    <FrameworkReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="JetBrains.Annotations" Version="2020.1.0" />
    <PackageReference Include="MediatR" Version="9.0.0" />
    <PackageReference Include="Microsoft.AspNetCore.Identity.EntityFrameworkCore" Version="3.1.9" />
    <PackageReference Include="Microsoft.Data.SqlClient" Version="2.0.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore" Version="3.1.9" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="3.1.9" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="3.1.9">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="3.1.9" />
    <PackageReference Include="Newtonsoft.Json" Version="12.0.3" />
  </ItemGroup>

</Project>

I am using Visual Studio 16.8.1

[roji]

@rosdi and others, we're well aware of this issue, have reproduced it successfully, and it's under investigation.

[ghosttie]

To give you an idea of the severity of the issue, our project has 1224 migrations, and build time went from 10 minutes to 3 hours

[clement911]

We are experiencing the same issue, making it completely unworkable. While the fix is being worked on, is there a way to disable code analysis? joakimriedel 's editoconfig thing didn't work for us.

We checked these off but even that seem to have no impact.

[berets76]

We are experiencing the same issue, but we have no Migrations, it's a "simple" blazor server project.

Analyzers disabled, didn't solve the problem

Version 16.8.1

[YuriyFront]

The same issue. Do upgrade today and now project build time increased 10 times + CPU usage is 100% + memory around 15GB

[roji]

This has been identified as a compiler issue, with the fix already merged for the next patch: https://github.com/dotnet/roslyn/pull/48897.

[ErikEJ]

@roji Would be nice to know when to expect the fix to be in VS.

[roji]

~According to the milestone the issue has been merged for VS 16.9 Preview 2 (the next one), but I don't have any precise info on when that's coming out...~

Sorry, we've obviously discussing the 16.8 backport. I don't have precise info on that either unfortunately...

[ErikEJ]

Yes, I mean 16.8 - seems reasonable to include it, I guess?

[rosdi]

Yeah... we need this to be in 16.8.2 ASAP. Because as it currently stands, 16.8 or 16.8.1 is unusable for us.

[roji]

The backport (https://github.com/dotnet/roslyn/pull/49419) explicitly says it was merged into 16.8, so yes - this should be available in the next VS 16.8 patch version.

[augustoproiete]

Awesome news! Thank you @roji and team!

[indifit]

If you’re in a position where you won’t need to go back to any previous DB state, I found that following the steps in this article ( https://weblog.west-wind.com/posts/2016/jan/13/resetting-entity-framework-migrations-to-a-clean-slate) and consolidating my migrations really helped reduce the compile time.

On Sun, 15 Nov 2020 at 13:10, ghosttie notifications@github.com wrote:

To give you an idea of the severity of the issue, our project has 1224 migrations, and build time went from 10 minutes to 3 hours

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dotnet/efcore/issues/23291#issuecomment-727567498, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYV6TUZ65S3N2Z53HXILLTSP7HM7ANCNFSM4TTWFH2A .

-- Simon Pickersgill

[augustoproiete]

For watchers on this issue, Visual Studio 2019 version 16.8.2 is out and it fixes the increased build time :tada: :sweat_smile: https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes#16.8.2

(or at least it fixed it for me)

[berets76]

For me 16.8.2 doesn't fix that issue, my simple blazor server project (no migrations) takes 3 minutes to build, with 16.7.8 it took few seconds...

[roji]

@berets76 if you're still experiencing the issue, can you please submit a minimal program that reproduces it with VS 16.8.2?

[rsuk]

Huge thanks to @roji and the EF Core team for getting this nailed so quickly. Can confirm that in our EF Core project with a couple of hundred migrations the build time is back to normal with 16.8.2 (it was not completing at all in 16.8.1 and 16.8.2).

[roji]

@rsuk FYI we didn't really do anything, this was handled on the VS side :)

[Mizuwokiru]

Project is still built slowly using dotnet build command if I have many migration files (66 MB). I don't have this issue on the machine without .NET 5 installed.

[roji]

@Mizuwokiru can you please open a new issue with a repro project so we can investigate?

[berets76]

@roji I found the issue for my case: I've no migrations, but I scaffolded an existing MS SQL database structure, and the problem is the OnModelCreating method (I've160 tables).

I simply delete OnModelCreating method content within context class, and my build time returns to few seconds instead of 4 minutes!

Obviously it's not a workaround because without fluent definition my code can't run.

[roji]

@berets76 have you upgraded to Visual Studio 16.8.2 which contains the fix, as suggested above? If you've upgraded and are still seeing a 4-minute build time, can you open a new issue with a minimal code sample, so I can investigate? If you can't share the code publicly, you can also share it privately with me via the email on my github profile.

[berets76]

@roji I wrote to you privately

[roji]

For anyone still running into this on the command-line, note that you'll have to wait until dotnet SDK 5.0.101 is released - which should be in around a week. SDK 5.0.100 uses version 16.8.0 of the compiler, which has the issue.

[WinInsider]

To speed up migrations build/sql script generation, is there a way to collapse all the historic migrations into one file? We have over 300 migrations accumulating over 2 years, is the a guideline for such operation?

[clement911]

As far as I know there is no automated way: see https://github.com/dotnet/efcore/issues/2174

There is a manual way but it seems quite error prone: https://docs.microsoft.com/en-us/ef/core/managing-schemas/migrations/managing?tabs=dotnet-core-cli#resetting-all-migrations

[MichelZ]

I've created an MSBuild task that tries to solve this automatically. All you have to do is install the task. (Not the historic migration collapse, but it decreases build times significantly by excluding most *.Designer.cs files on compilation)

It automatically moves these attributes from the *.Designer.cs file to the main code file:

[DbContext...
[Migration...

It sets the *.Designer.cs files to not compile: <DefaultItemExcludes>$(DefaultItemExcludes);**\*.Designer.cs</DefaultItemExcludes>

It then looks for the latest N(configurable count) of *.Designer.cs files and specifically enables them for compilation by <Compile Include="xxx" />

Install: dotnet add package MSBuild.EntityFrameworkCore.RemoveDesignerCompilation --prerelease

(You have to use --prerelease until there is a stable version published. I'd like to gather some pre-release feedback first)

NuGet

GitHub