search

dotnet / roslyn

The Roslyn .NET compiler provides C# and Visual Basic languages with rich code analysis APIs.
https://docs.microsoft.com/dotnet/csharp/roslyn-sdk/
MIT License
19.11k stars 4.04k forks source link

Flaky test: Microsoft.CodeAnalysis.MSBuild.UnitTests.NewlyCreatedProjectsFromDotNetNew.ValidateVisualBasicTemplateProjects #74827

Open dibarbet opened 3 months ago

dibarbet commented 3 months ago

https://helix.dot.net/BuildAnalysis/SearchTests?error=&testName=Microsoft.CodeAnalysis.MSBuild.UnitTests.NewlyCreatedProjectsFromDotNetNew.ValidateVisualBasicTemplateProjects&definition=roslyn-integration-CI&branch=main&dateValue=1&dateType=Day%28s%29

e.g.

System.InvalidOperationException : `dotnet new "console" -o "T:\RoslynTests\31f69427-04d3-48b7-add0-87ac517bb283" --language "VB"` returned a non-zero exit code. Output: Error: Fatal error. 0xC0000005 Repeat 2 times: -------------------------------- at Microsoft.VisualStudio.Setup.Configuration.IEnumSetupInstances.Next(Int32, Microsoft.VisualStudio.Setup.Configuration.ISetupInstance[], Int32 ByRef) -------------------------------- at Microsoft.DotNet.Workloads.Workload.VisualStudioWorkloads.GetVisualStudioInstances() at Microsoft.DotNet.Workloads.Workload.VisualStudioWorkloads.GetInstalledWorkloads(Microsoft.NET.Sdk.WorkloadManifestReader.IWorkloadResolver, Microsoft.DotNet.Workloads.Workload.List.InstalledWorkloadsCollection, System.Nullable`1<Microsoft.NET.Sdk.WorkloadManifestReader.SdkFeatureBand>) at Microsoft.DotNet.Workloads.Workload.List.WorkloadInfoHelper.AddInstalledVsWorkloads(System.Collections.Generic.IEnumerable`1<Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadId>) at Microsoft.DotNet.Workloads.Workload.List.WorkloadInfoHelper.get_InstalledAndExtendedWorkloads() at Microsoft.DotNet.Tools.New.WorkloadsInfoProvider.GetInstalledWorkloadsAsync(System.Threading.CancellationToken) at Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory+WorkloadConstraint+<ExtractWorkloadInfoAsync>d__9.MoveNext() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory+WorkloadConstraint+<ExtractWorkloadInfoAsync>d__9, Microsoft.TemplateEngine.Edge, Version=9.0.100.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<ExtractWorkloadInfoAsync>d__9 ByRef) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.ValueTuple`2[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory+WorkloadConstraint+<ExtractWorkloadInfoAsync>d__9, Microsoft.TemplateEngine.Edge, Version=9.0.100.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<ExtractWorkloadInfoAsync>d__9 ByRef) at Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory+WorkloadConstraint.ExtractWorkloadInfoAsync(System.Collections.Generic.IEnumerable`1<Microsoft.TemplateEngine.Abstractions.Components.IWorkloadsInfoProvider>, Microsoft.Extensions.Logging.ILogger, System.Threading.CancellationToken) at Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory+WorkloadConstraint+<CreateAsync>d__6.MoveNext() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory+WorkloadConstraint+<CreateAsync>d__6, Microsoft.TemplateEngine.Edge, Version=9.0.100.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<CreateAsync>d__6 ByRef) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory+WorkloadConstraint+<CreateAsync>d__6, Microsoft.TemplateEngine.Edge, Version=9.0.100.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<CreateAsync>d__6 ByRef) at Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory+WorkloadConstraint.CreateAsync(Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings, Microsoft.TemplateEngine.Abstractions.Constraints.ITemplateConstraintFactory, System.Threading.CancellationToken) at Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory+<Microsoft-TemplateEngine-Abstractions-Constraints-ITemplateConstraintFactory-CreateTemplateConstraintAsync>d__5.MoveNext() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory+<Microsoft-TemplateEngine-Abstractions-Constraints-ITemplateConstraintFactory-CreateTemplateConstraintAsync>d__5, Microsoft.TemplateEngine.Edge, Version=9.0.100.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<Microsoft-TemplateEngine-Abstractions-Constraints-ITemplateConstraintFactory-CreateTemplateConstraintAsync>d__5 ByRef) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory+<Microsoft-TemplateEngine-Abstractions-Constraints-ITemplateConstraintFactory-CreateTemplateConstraintAsync>d__5, Microsoft.TemplateEngine.Edge, Version=9.0.100.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<Microsoft-TemplateEngine-Abstractions-Constraints-ITemplateConstraintFactory-CreateTemplateConstraintAsync>d__5 ByRef) at Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory.Microsoft.TemplateEngine.Abstractions.Constraints.ITemplateConstraintFactory.CreateTemplateConstraintAsync(Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings, System.Threading.CancellationToken) at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke() at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
JoeRobich commented 3 months ago

We definitely need to work out how to avoid this and the NuGet issue which causes us to skip the C# tests. These are the only thing that validates our project system loads all the various project types.

We could perhaps have a job which generates all the templates and restores the generated projects. We could then cache the result, hopefully avoiding AVE and NuGet issues. The existing test job could be updated to load the latest version of the cache.

The tests could be updated to check for the pre-generated projects from an expected location and generate the project if it was missing.

For more information about caching: Blog: https://devblogs.microsoft.com/devops/caching-and-faster-artifacts-in-azure-pipelines/ Docs: https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops