Analyzer memory issue

[jossgc]

EF migration problem

We have an issue with an EF migrations using Azure DevOps pipeline - always get stucked not matter how long is the timeout set. Every time the pipeline finish is cancelled by timeout.

Include your code

dotnet tool restore

dotnet ef  migrations materialSubmittalValues  finalsignature script  -i -o "$(build.sourcesDirectory)\SQLStaging\ArchiveMigration.sql" --project "$(build.sourcesDirectory)\CDOTWebsite\CDOTWebsite.csproj" --context "CDOTWebsite.Models.cdotarchiveContext" --configuration Release

dotnet ef  migrations script  submittalSignatures finalsignature script -i -o "$(build.sourcesDirectory)\SQLStaging\ProductionMigration.sql" --project "$(build.sourcesDirectory)\CDOTWebsite\CDOTWebsite.csproj" --context "CDOTWebsite.Models.cdottrackContext" --configuration Release

Include verbose output

Tool 'dotnet-ef' (version '3.1.6') was restored. Available commands: dotnet-ef

Restore was successful.
Build started...
##[debug]Re-evaluate condition on job cancellation for step: 'Command Line Script Execute migrations copy'.
Terminate batch job (Y/N)? 
##[error]The operation was canceled.
##[debug]System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at Microsoft.VisualStudio.Services.Agent.Util.ProcessInvoker.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.ProcessInvokerWrapper.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.DefaultStepHost.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.PowerShell3Handler.RunAsync()
   at Microsoft.VisualStudio.Services.Agent.Worker.TaskRunner.RunAsync()
   at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)

Include provider and version information

EF Core version: 3.1.6 Database provider: Microsoft.EntityFrameworkCore.SqlServer Target framework: Net Core 3.1.302 Operating system: Windows IDE: Visual Studio 2019

I have a repro to share.

[Saibamen]

Please update your EF Core to 3.1.9, target framework to 3.1.403 and try again.

What is your Visual Studio version?

[roji]

@jossgc your original description is a bit unclear - are you saying that what's hanging is dotnet ef migrations script? Or dotnet restore?

If it's dotnet restore, this could be caused by networking issues, or a problematic nuget feed which is being tried. For example, your NuGet.Config could be containing a server which is unreachable from your Azure DevOps servers because of a firewall or something. Running dotnet restore -v detailed could shed some light.

If it's really the script generation, that would be quite strange - can you post a project which reproduces this?

[ajcvickers]

@Saibamen I see you're asking people to update to 3.1.9 in several places. While it is generally best practice to use the latest patch version, the number of fixes in recent patches is small and unlikely to impact most reports. For example, I don't think anything has changed between EF Core 3.1.6 and 3.1.9 that would impact this issue.

I appreciate your help in trying to get good information on the report by doing this, but typically we try to ask for:

• Reproducible code, if the issue can't be reproduced
• Whether or not it works on 5.0 if it looks like something that might have been fixed, since lots of bugs have been fixed between 3.1 and 5.0
• Platform and provider. if that seems relevant

[ajcvickers]

EF Team Triage: Closing this issue as the requested additional details have not been provided and we have been unable to reproduce it.

BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.

[jossgc]

As a quick solution it is necessary to disable code analyzers. The simplest way is to add

 <Target Name="DisableAnalyzers" BeforeTargets="CoreCompile">
      <ItemGroup>
       <Analyzer Remove="@(Analyzer)"/>
     </ItemGroup>
 </Target>

[ajcvickers]

@roji Analyzer issue?

[TranSystemsGitHub]

Yes, it turns out the pipeline was crashing due to a out of memory error (our guess, what we saw when it was moved to private agent, it never errored technically, it just never returned). Once we turned off Analyzers in .csproj, the pipeline was able to build successfully.

[roji]

@TranSystemsGitHub @jossgc any chance you can post a small project? I'm very interested in understanding the cause of this and fixing it.

[TranSystemsGitHub]

Sure, I will try - put it on GitHub? I think the issue is that it's a "large" project. So a small project example is not going to work...

[roji]

@TranSystemsGitHub yes, a github repo would be ideal - just point me at it.

[roji]

FYI have received a repro via private channels, will investigate this very soon.

[TranSystemsGitHub]

Oh good, I was trying to figure out how to do this. Interestingly enough, we had another issue upgrading to Visual Studio 16.8, the build was hanging, just like it was for the pipeline. I am able to build the solution with VS 16.8 if I delete most of the migrations. Maybe it has something to do with our huge number of migrations.

[roji]

@TranSystemsGitHub sorry for the late response on this. Re the VS 16.8 issue, you're likely running into #23291 - make sure you have at least VS 16.8.2: 16.8.0 and 16.8.1 contain a compiler bug which causes this. Can you please try this out and confirm?

Re the original issue, I'm looking into it.

[roji]

Duplicate of #19648

[roji]

@TranSystemsGitHub I indeed reproduced considerable slowness when enabling the analyzers in your project (0:55 minutes without analyzers vs. 3:34 with). The analyzer that comes with EF was rewritten for perf reasons for 5.0 (see #19648), and I can confirm that once I switched the dependency to be on EF Core 5.0, the build time dropped back down to around a minute.

[TranSystemsGitHub]

Thank you for your feedback. We were never able to compile with Analyzers on, so that's interesting you could get it to compile in 3:34 minutes at all. I think that was an artifact of the pipeline not having enough memory.

[roji]

@TranSystemsGitHub that's interesting indeed... I'd give it a try with EF Core 5.0, at the very least as a test - the application doesn't need to actually run, just to see what that does to build times.

One last thought, is that maybe the 5.0.100 dotnet SDK is installed and being used in your pipeline. If that's the case, you'd definitely be running into #23291 (and the issue will have only recently appeared, the moment 5.0.100 was released and made it to your pipeline's environment). If this is the case, you should be able to work around it by pinning to a 3.1 dotnet SDK in global SDK, until 5.0.101 comes out with the fix.

[TranSystemsGitHub]

Okay, I will try EF Core 5.0.

I did not pin to sdk 5.0, here is my pipeline that I could never get to return, I was using .NET Core sdk 3.1.302

[roji]

I did not pin to sdk 5.0, here is my pipeline that I could never get to return, I was using .NET Core sdk 3.1.302

OK. I'm not very familiar with AzDo, but it's possible that the 5.0.100 SDK is already pre-installed for you or similar. I'd carefully look at the build logs; if at any point your seeing dotnet build with an MSBuild version that's 16.8.0 (as opposed to 16.7.x), then that's definitely your culprit. Probably worth looking into.

[ajcvickers]

EF Team Triage: Closing this issue as the requested additional details have not been provided and we have been unable to reproduce it.

BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.

[HenkKin]

As a quick solution it is necessary to disable code analyzers. The simplest way is to add

 <Target Name="DisableAnalyzers" BeforeTargets="CoreCompile">
      <ItemGroup>
       <Analyzer Remove="@(Analyzer)"/>
     </ItemGroup>
 </Target>

Adding <RunAnalyzersDuringBuild>False</RunAnalyzersDuringBuild> saved my day. Azure DevOps Build Pipeline runs now for +/- 12 minutes instead of timing out after one hour.

When a lot of EF Core migrations with data seeding are added to the solution then rebuilding is taking a lot of memory on my local machine (10GB+). This causes timeouts in Azure DevOps pipeline.

2 possible solutions: 1) create a new baseline (initial) EF migration (merge existing migrations) 2) disable analyzers in project files: <RunAnalyzersDuringBuild>False</RunAnalyzersDuringBuild>

[roji]

@HenkKin can you please provide some more details? Which version of EF Core are you using, and importantly, which exact version of the .NET SDK are you using? How many migrations exactly does your migrations folder have, and how much total space do they take?

[HenkKin]

@HenkKin can you please provide some more details? Which version of EF Core are you using, and importantly, which exact version of the .NET SDK are you using? How many migrations exactly does your migrations folder have, and how much total space do they take?

I've been running into this issue for a few years now. First in .NET Core 3, later also in .NET 5 and now .NET 6.

Picture of a few days ago

Picture of this morning with error: ##[error]The job running on agent Hosted Agent ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Yesterday I added a new migration and i faced Azure Devops Pipeline timeouts.

Size of the migration folder:

[roji]

@HenkKin thanks for the additional details.

Previous versions of the EF Core analyzer indeed could cause some builds to be significantly heavier, but as far as I know, these were fixed - I'm not aware of any outstanding issue.

On the other hand, compiling close to 200MB of source code is definitely not something that works efficiently, regardless of the analyzer; people run into this mainly when using data seeding with large data; this is explicitly discouraged, and we recommend finding other ways to manage your large data.

However, if you're sure that you're seeing a significant analyzer-related slowdown in compilation time - i.e. a significant gap between compilation times when the analyzer is enabled and disabled - and this is reproducible with EF Core 6.0, then please open a new issue and include a sample project where we could see this happening. If you don't want to publicly share your project, you can also email me privately (at the email on my github profile) with a link to the repro.