`dotnet ef migrations` Invalidates SLN Cache, Resulting a Full Recompile

[Mike-E-angelo]

Ask a question

This issue started with #25524 which led to #9716, which suggested using the corresponding dotnet ef commands. I was happy with using this workaround and closed the issue.

Unfortunately, it appears the workaround also suffers from an issue, which is that it seems to invalidate the SLN/build cache when it runs.

The result of this is a full rebuild of the solution the next time this is attempted in Visual Studio. For larger solutions this takes time and feels disruptive. I made mention of this in https://github.com/dotnet/efcore/issues/9716#issuecomment-911914458 but did not hear anything back so ensuring this is captured here.

I did some searching in the issues as well and no results showed up. However, I have not had too much luck with it or GitHub search so this may be already captured. Please let me know if so and I will close the issue.

You know, standard procedure. 😆

Include your code

First, I run dotnet clean and SHIFT-CTRL-B on my SLN:

========== Build: 130 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========

SHIFT-CTRL-B again to be sure:

========== Build: 0 succeeded, 0 failed, 130 up-to-date, 0 skipped ==========

I then run the following commands:

dotnet ef database drop -f -v --project Starbeam.Model.Schema --startup-project Starbeam.Model.Schema
dotnet ef migrations remove -v --project Starbeam.Model.Schema --startup-project Starbeam.Model.Schema
dotnet ef migrations add Initial -v --project Starbeam.Model.Schema --startup-project Starbeam.Model.Schema

Finally, after that is complete, I run SHIFT-CTRL-B once more in Visual Studio:

========== Build: 129 succeeded, 0 failed, 1 up-to-date, 0 skipped ==========

FWIW this is with 6.0.0-rc.2.21452.6 tools.

Include stack traces

NA

Include verbose output

NA

Include provider and version information

EF Core version: 6.0.0-rc.2.21452.6 Database provider: Microsoft.EntityFrameworkCore.SqlServer Target framework: net6.0 Operating system: Windows 10 IDE: Visual Studio 2022 Preview 3.1

[davidroth]

Is the project Starbeam.Model.Schema explicitly or implicitly referenced by the other 129 projects? If so, it makes sense, because with migrations add you are adding additional types to the project which requires recompilation of the downstream projects.

[ajcvickers]

@Mike-E-angelo In the section "I then run the following commands:" if you replace all three commands with a single "dotnet build" do you get the same behavior?

[Mike-E-angelo]

That is exactly the expectation I have as well @davidroth, and this is what occurs when using PMC. That is, when using PMC, all the dependent projects are built. Of course, this is not necessary/desired and is captured in #9716. When this does occur in PMC, about 40 projects are built in my case, and they are done with each command listed above (which is why I reported #25524).

In contrast, after using a dotnet ef command, all 130 projects within the solution are built when CTRL-SHIFT-B is done in Visual Studio.

@ajcvickers I appreciate you troubleshooting this with me. I did look into that for you, and it appears that donet build and the SLN are synchronized as expected. This is what I see after running dotnet build:

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:16.68

And then the SLN:

========== Build: 0 succeeded, 0 failed, 130 up-to-date, 0 skipped ==========

Build time 00:00:00.967
Build ended at 9/7/2021 7:51:35 AM

[Mike-E-angelo]

OK, so I poked around some more and this appears to occur when using the new Developer PowerShell in Visual Studio 2022. If I open a new PowerShell and run the same commands, I get the following:

========== Build: 40 succeeded, 0 failed, 90 up-to-date, 0 skipped ==========

Maybe it's related to the following?

https://developercommunity.visualstudio.com/t/Developer-PowerShell-Uses-Directory-for/1510455

[Mike-E-angelo]

OK I see that there have been updates to that thread.

Applying the following command in Developer PowerShell does the trick: $env:DOTNET_ROOT = ""

After applying that command and running an dotnet ef command in Developer PowerShell, I now I see the following:

========== Build: 40 succeeded, 0 failed, 90 up-to-date, 0 skipped ==========

So, this isn't an EF issue after all but a known issue with Developer Powershell. Hopefully this will help someone else who might run into the same issue.

Thank you all for your assistance in figuring this out. 👍

[Mike-E-angelo]

So looking into this further @ajcvickers when using a non-VS PowerShell prompt, I run the following commands:

dotnet ef database drop -f -v --project Starbeam.Model.Schema --startup-project Starbeam.Model.Schema
dotnet build

I then see the following from the dotnet build command:

Determining projects to restore...                                                                                                   
Restored ...\Framework\DragonSpark.Composition\DragonSpark.Composition.csproj (in 835 ms).                              
Restored ...\Framework\DragonSpark\DragonSpark.csproj (in 301 ms).                                                      
Restored ...\Framework\DragonSpark.Application\DragonSpark.Application.csproj (in 902 ms).                              
Restored ...\Starbeam.Model\Starbeam.Model.csproj (in 552 ms).                                                          
Restored ...\Framework\DragonSpark.Identity.Google\DragonSpark.Identity.Google.csproj (in 120 ms).                      
Restored ...\Framework\DragonSpark.Identity.Mixcloud\DragonSpark.Identity.Mixcloud.csproj (in 433 ms).                  
Restored ...\Framework\DragonSpark.Identity.Facebook\DragonSpark.Identity.Facebook.csproj (in 120 ms).                  
Restored ...\Framework\DragonSpark.Identity.Microsoft\DragonSpark.Identity.Microsoft.csproj (in 429 ms).                
Restored ...\Framework\DragonSpark.Azure\DragonSpark.Azure.csproj (in 1.01 sec).                                        
Restored ...\Framework\DragonSpark.Identity.DeviantArt\DragonSpark.Identity.DeviantArt.csproj (in 129 ms).              
Restored ...\Framework\DragonSpark.Identity.Coinbase\DragonSpark.Identity.Coinbase.csproj (in 107 ms).                  
Restored ...\Framework\DragonSpark.Identity.Amazon\DragonSpark.Identity.Amazon.csproj (in 111 ms).                      
Restored ...\Starbeam.Integration.DeviantArt\Starbeam.Integration.DeviantArt.csproj (in 1.02 sec).                      
Restored ...\Starbeam.Integration.Facebook\Starbeam.Integration.Facebook.csproj (in 1.02 sec).                          
Restored ...\Starbeam.Integration.Google\Starbeam.Integration.Google.csproj (in 1.05 sec).                              
Restored ...\Starbeam.Integration.Coinbase\Starbeam.Integration.Coinbase.csproj (in 1.06 sec).                          
Restored ...\Starbeam.Integration.Microsoft\Starbeam.Integration.Microsoft.csproj (in 1.08 sec).                        
Restored ...\Starbeam.Integration.Twitter\Starbeam.Integration.Twitter.csproj (in 858 ms).                              
Restored ...\Starbeam.Integration.Reddit\Starbeam.Integration.Reddit.csproj (in 854 ms).                                
Restored ...\Starbeam.Model.Schema\Starbeam.Model.Schema.csproj (in 893 ms).                                            
Restored ...\Starbeam.Features.Identity.Integration\Starbeam.Features.Identity.Integration.csproj (in 2.03 sec).        
Restored ...\Starbeam\Starbeam.csproj (in 215 ms).                                                                      
Restored ...\Starbeam.Integration.Patreon\Starbeam.Integration.Patreon.csproj (in 395 ms).                              
Restored ...\Starbeam.Integration.Mixcloud\Starbeam.Integration.Mixcloud.csproj (in 480 ms).                            
Restored ...\Starbeam.Integration.PayPal\Starbeam.Integration.PayPal.csproj (in 535 ms).                                
Restored ...\Framework\DragonSpark.Presentation\DragonSpark.Presentation.csproj (in 338 ms).                            
Restored ...\Framework\DragonSpark.Identity.PayPal\DragonSpark.Identity.PayPal.csproj (in 165 ms).                      
Restored ...\Framework\DragonSpark.Identity.Patreon\DragonSpark.Identity.Patreon.csproj (in 149 ms).                    
Restored ...\Framework\DragonSpark.Identity.Reddit\DragonSpark.Identity.Reddit.csproj (in 180 ms).                      
Restored ...\Framework\DragonSpark.Server\DragonSpark.Server.csproj (in 363 ms).                                        
Restored ...\Starbeam.Integration.Amazon\Starbeam.Integration.Amazon.csproj (in 584 ms).                                
Restored ...\Framework\DragonSpark.Identity.Twitter\DragonSpark.Identity.Twitter.csproj (in 403 ms).                    
Restored ...\Starbeam.Features\Starbeam.Features.csproj (in 625 ms).                                                    
97 of 130 projects are up-to-date for restore.                                                                                       

It is my expectation and understanding that any project listed with Framework is not a dependent project and should not be built. Can you please confirm this? I am wondering why these are in the list here.

[Mike-E-angelo]

Alright I have a reproduction here for your review: https://github.com/DragonSpark/Framework/tree/issues/reported/efcore/25878

If I run these commands:

dotnet build
dotnet build

I see the following output after the last command:

Microsoft (R) Build Engine version 17.0.0-preview-21416-02+cb3144483 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  All projects are up-to-date for restore.

Whereas running these commands:

dotnet build
dotnet ef database drop -f -v --project DragonSpark.Schema --startup-project DragonSpark.Schema
dotnet build

I see the following output on the last command:

Microsoft (R) Build Engine version 17.0.0-preview-21416-02+cb3144483 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  Restored ...\Framework\DragonSpark.Composition\DragonSpark.Composition.csproj (in 427 ms).
  Restored ...\Framework\DragonSpark\DragonSpark.csproj (in 427 ms).
  Restored ...\Framework\DragonSpark.Application\DragonSpark.Application.csproj (in 320 ms).
  Restored ...\Framework\DragonSpark.Schema\DragonSpark.Schema.csproj (in 374 ms).
  23 of 27 projects are up-to-date for restore.

DragonSpark.Schema is expected. However, DragonSpark.Composition, DragonSpark.Application, and DragonSpark are not, as they are dependencies of DragonSpark.Schema.

So, it would seem there is some sort of invalidation occurring when this command is executed.

[ajcvickers]

/cc @bricelam

[bricelam]

Without debugging it, my initial thought is that we should probably remove this file before building again:

https://github.com/dotnet/efcore/blob/b634979c2f16fc2cb0489c5138067c03ef1f5216/src/dotnet-ef/Project.cs#L61-L71

[ajcvickers]

@Mike-E-angelo The link to the repro code 404's for me. Are you still seeing this issue?

[Mike-E-angelo]

Hi @ajcvickers pardon the disruption I was able to find that and retry with 8.0. The branch has been pushed and is available for your usage again:

https://github.com/DragonSpark/Framework/tree/issues/reported/efcore/25878

This issue appears to still exist and occur as described here.

Additionally, while I have you here. :D I have noticed that I will build in Visual Studio and when using dotnet ef migrations commands the build caches do not appear to be used, and do another rebuild. This takes a lot of time and CPU utilization... I have been wondering if this is due to this issue. 🤔 It would seem dotnet build would leverage the new build acceleration and only build projects that have changed. This does not appear to be occurring.

[ErikEJ]

Build acceleration is a VS only feature, not possible with dotnet build

[Mike-E-angelo]

Thank you @ErikEJ one less mystery in my world 😊

[AndriySvyryd]

Should be fixed by https://github.com/dotnet/efcore/commit/7cae3c358d5fbf329ea27cedec16e217f0385958

[Mike-E-angelo]

Should be fixed by https://github.com/dotnet/efcore/commit/7cae3c358d5fbf329ea27cedec16e217f0385958

Thank you for your efforts @AndriySvyryd and team. Were you able to verify with the steps above?

[AndriySvyryd]

Thank you for your efforts @AndriySvyryd and team. Were you able to verify with the https://github.com/dotnet/efcore/issues/25878#issuecomment-916207474?

Yes, it will still invalidate on the first time dotnet ef is called for a project, but should work as expected on further invocations.

[Mike-E-angelo]

Thank you for your continued investigation @AndriySvyryd. As I understand it, this fix requires .NET9 preview which I do not have installed. What I am asking is if someone has validated this issue against .NET9 to ensure it works as expected now, does that make sense?

Put another way: I am seeing a lot should work but not seeing does work, if that makes sense. :)

[AndriySvyryd]

You can try it with a daily build: dotnet tool update dotnet-ef --prerelease --add-source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet9/nuget/v3/index.json

[Mike-E-angelo]

Unfortunately, it does not appear this issue has been addressed using the above and the provided daily build. This is what I see on the 2nd dotnet build call:

> dotnet build
MSBuild version 17.9.4+90725d08d for .NET
  Determining projects to restore...
  Restored ...\Framework\DragonSpark\DragonSpark.csproj (in 458 ms).
  Restored ...\Framework\DragonSpark.Composition\DragonSpark.Composition.csproj (in 326 ms).
  Restored ...\Framework\DragonSpark.Diagnostics\DragonSpark.Diagnostics.csproj (in 396 ms).
  Restored ...\Framework\DragonSpark.Schema\DragonSpark.Schema.csproj (in 1.24 sec).
  Restored ...\Framework\DragonSpark.Application\DragonSpark.Application.csproj (in 1.24 sec).
  23 of 28 projects are up-to-date for restore.

[AndriySvyryd]

@Mike-E-angelo What's the output from the dotnet ef database drop -f -v --project DragonSpark.Schema --startup-project DragonSpark.Schema command?

[Mike-E-angelo]

Here you go @AndriySvyryd:

Using project '...\Framework\DragonSpark.Schema\DragonSpark.Schema.csproj'.
Using startup project '...\Framework\DragonSpark.Schema\DragonSpark.Schema.csproj'.
dotnet msbuild /target:GetEFProjectMetadata /property:EFProjectMetadataFile=...\AppData\Local\Temp\tmpt0vxdt.tmp /verbosity:quiet /nologo ...\Framework\DragonSpark.Schema\DragonSpark.Schema.csproj
dotnet msbuild /target:GetEFProjectMetadata /property:EFProjectMetadataFile=...\AppData\Local\Temp\tmpci5hj3.tmp /verbosity:quiet /nologo ...\Framework\DragonSpark.Schema\DragonSpark.Schema.csproj
Build started...
dotnet build ...\Framework\DragonSpark.Schema\DragonSpark.Schema.csproj /verbosity:quiet /nologo /p:PublishAot=false

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.27
Build succeeded.
dotnet exec --depsfile ...\Framework\DragonSpark.Schema\bin\Debug\net8.0\DragonSpark.Schema.deps.json --additionalprobingpath ...\.nuget\packages --additionalprobingpath "C:\Program Files\dotnet\sdk\NuGetFallbackFolder" --runtimeconfig ...\Framework\DragonSpark.Schema\bin\Debug\net8.0\DragonSpark.Schema.runtimeconfig.json ...\.dotnet\tools\.store\dotnet-ef\9.0.0-preview.2.24120.2\dotnet-ef\9.0.0-preview.2.24120.2\tools\net8.0\any\tools\netcoreapp2.0\any\ef.dll database drop -f --assembly ...\Framework\DragonSpark.Schema\bin\Debug\net8.0\DragonSpark.Schema.dll --project ...\Framework\DragonSpark.Schema\DragonSpark.Schema.csproj --startup-assembly ...\Framework\DragonSpark.Schema\bin\Debug\net8.0\DragonSpark.Schema.dll --startup-project ...\Framework\DragonSpark.Schema\DragonSpark.Schema.csproj --project-dir ...\Framework\DragonSpark.Schema\ --root-namespace DragonSpark.Schema --language C# --framework net8.0 --nullable --working-dir ...\Framework --verbose
Using assembly 'DragonSpark.Schema'.
Using startup assembly 'DragonSpark.Schema'.
Using application base '...\Framework\DragonSpark.Schema\bin\Debug\net8.0'.
Using working directory '...\Framework\DragonSpark.Schema'.
Using root namespace 'DragonSpark.Schema'.
Using project directory '...\Framework\DragonSpark.Schema\'.
Remaining arguments: .
Finding DbContext classes...
Finding IDesignTimeDbContextFactory implementations...
Found IDesignTimeDbContextFactory implementation 'StorageBuilder'.
Found DbContext 'ApplicationState'.
Finding application service provider in assembly 'DragonSpark.Schema'...
Finding Microsoft.Extensions.Hosting service provider...
No static method 'CreateHostBuilder(string[])' was found on class 'Program'.
No application service provider was found.
Finding DbContext classes in the project...
Using DbContext factory 'StorageBuilder'.
Using context 'ApplicationState'.
Creating DbConnection.
Created DbConnection. (26ms).
Dropping database 'dragonspark.schema' on server '(localdb)\mssqllocaldb'.
Opening connection to database 'dragonspark.schema' on server '(localdb)\mssqllocaldb'.
Opened connection to database 'dragonspark.schema' on server '(localdb)\mssqllocaldb'.
Creating DbCommand for 'ExecuteNonQuery'.
Created DbCommand for 'ExecuteNonQuery' (4ms).
Initialized DbCommand for 'ExecuteNonQuery' (8ms).
Executing DbCommand [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT 1
Executed DbCommand (17ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT 1
Closing connection to database 'dragonspark.schema' on server '(localdb)\mssqllocaldb'.
Closed connection to database 'dragonspark.schema' on server '(localdb)\mssqllocaldb' (3ms).
Creating DbConnection.
Created DbConnection. (1ms).
Opening connection to database 'master' on server '(localdb)\mssqllocaldb'.
Opened connection to database 'master' on server '(localdb)\mssqllocaldb'.
Creating DbCommand for 'ExecuteNonQuery'.
Created DbCommand for 'ExecuteNonQuery' (1ms).
Initialized DbCommand for 'ExecuteNonQuery' (3ms).
Executing DbCommand [Parameters=[], CommandType='Text', CommandTimeout='60']
IF SERVERPROPERTY('EngineEdition') <> 5
BEGIN
    ALTER DATABASE [dragonspark.schema] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
END;
Executed DbCommand (5ms) [Parameters=[], CommandType='Text', CommandTimeout='60']
IF SERVERPROPERTY('EngineEdition') <> 5
BEGIN
    ALTER DATABASE [dragonspark.schema] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
END;
Creating DbCommand for 'ExecuteNonQuery'.
Created DbCommand for 'ExecuteNonQuery' (4ms).
Initialized DbCommand for 'ExecuteNonQuery' (17ms).
Executing DbCommand [Parameters=[], CommandType='Text', CommandTimeout='60']
DROP DATABASE [dragonspark.schema];
Executed DbCommand (37ms) [Parameters=[], CommandType='Text', CommandTimeout='60']
DROP DATABASE [dragonspark.schema];
Closing connection to database 'master' on server '(localdb)\mssqllocaldb'.
Closed connection to database 'master' on server '(localdb)\mssqllocaldb' (1ms).
Disposing connection to database 'master' on server '(localdb)\mssqllocaldb'.
Disposed connection to database '' on server '' (1ms).
Successfully dropped database 'dragonspark.schema'.
'ApplicationState' disposed.
Disposing connection to database 'dragonspark.schema' on server '(localdb)\mssqllocaldb'.
Disposed connection to database '' on server '' (1ms).

[AndriySvyryd]

I see. It looks like just calling dotnet build ...\Framework\DragonSpark.Schema\DragonSpark.Schema.csproj invalidates the NuGet graph cache, so this is actually a .NET SDK / NuGet issue

[Mike-E-angelo]

Thank you for the clarification @AndriySvyryd. Where is the best place to report this?

[AndriySvyryd]

You can try https://github.com/dotnet/sdk