search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.36k stars 4.75k forks source link

[9.0.1xx-rc1] `MSB0001: Internal MSBuild Error` while building command-line-api #108143

Open ellahathaway opened 2 months ago

ellahathaway commented 2 months ago

While building in the CentOSStream9_Mono_Offline_CurrentSourceBuiltSdk_x64 leg (Mono Stage 2), the build encountered the following error:

Time Elapsed 00:00:02.47
    MSBUILD : error MSB1025: An internal failure occurred while running MSBuild.
    Microsoft.Build.Framework.InternalErrorException: MSB0001: Internal MSBuild Error: Cannot acquire required number of nodes.
    =============
    System.AggregateException: One or more errors occurred. (Method has zero rva)
     ---> Microsoft.Build.BackEnd.NodeFailedToLaunchException: Method has zero rva
     ---> System.BadImageFormatException: Method has zero rva
    File name: 'System.Diagnostics.Process'
       at System.Runtime.Serialization.SerializationGuard.ThrowIfDeserializationInProgress(String switchSuffix, Int32& cachedValue)
       at System.Diagnostics.Process.Start()
       at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
       at Microsoft.Build.BackEnd.NodeLauncher.StartInternal(String msbuildLocation, String commandLineArgs)
       --- End of inner exception stack trace ---
       at Microsoft.Build.BackEnd.NodeLauncher.StartInternal(String msbuildLocation, String commandLineArgs)
       at Microsoft.Build.BackEnd.NodeLauncher.<>c__DisplayClass3_0.<Start>b__0()
       at Microsoft.Build.BackEnd.NodeLauncher.DisableMSBuildServer(Func`1 func)
       at Microsoft.Build.BackEnd.NodeLauncher.Start(String msbuildLocation, String commandLineArgs, Int32 nodeId)
       at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.<>c__DisplayClass14_0.<GetNodes>g__StartNewNode|2(Int32 nodeId)
       at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.<>c__DisplayClass14_0.<GetNodes>b__0(Int32 nodeId)
       --- End of inner exception stack trace ---
dotnet-issue-labeler[bot] commented 2 months ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

dotnet-issue-labeler[bot] commented 2 months ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ellahathaway commented 2 months ago

This is the same error from https://github.com/dotnet/source-build/issues/4519. Based on the mitigation of that issue, I'm going to rerun the leg and will follow up based on that.

ellahathaway commented 2 months ago

Rerun succeeded - again, I think that this might just be a timing issue with mono:

The warning "cannot acquire required number of nodes" indicates the system could not allocate the necessary resources at the time of the request. This can often be a transient issue due to temporary high load or delays in resource allocation. Similarly, when the rva of a method is zero, it means that the method's il code is not correctly mapped in the executable file. The problem seems to be related to the timing of resource allocation or mapping during the initial run, which was successfully resolved on the subsequent attempt.

ccing the compiler team for verification on my assumptions @dotnet/roslyn

ellahathaway commented 2 months ago

@dotnet/roslyn - friendly ping on the question above :) TIA

333fred commented 2 months ago

Maybe I'm missing something, but this seems to be an MSBuild failure, not anything to do with Roslyn.

333fred commented 2 months ago

Maybe @rainersigwald would have a better idea.

MichaelSimons commented 2 months ago

[Triage] @uweigand - Is this anything you by chance have seen before? @omajid suggested pinging you. TIA.

rainersigwald commented 2 months ago

This is MSBuild reporting a failure to create a new worker process with a command line like dotnet path/to/MSBuild.dll. Looking at it I'd suspect a runtime problem, maybe System.Process itself?

ellahathaway commented 2 months ago

Looking at it I'd suspect a runtime problem, maybe System.Process itself?

@ericstj - do you know what might be causing this issue?

uweigand commented 2 months ago

[Triage] @uweigand - Is this anything you by chance have seen before? @omajid suggested pinging you. TIA.

Sorry, but no - I've never seen this before.

ericstj commented 2 months ago

System.BadImageFormatException: Method has zero rva

That's a new one for me. It looks like it's coming from the mono runtime. https://github.com/dotnet/runtime/blob/66e45ef152002a2e1cbea939ded6643dfa48307c/src/mono/mono/metadata/loader.c#L2065 Surprised to see the mono runtime being used at all for building the product - I thought we'd only be running on coreclr.

@steveisok @agocke

mthalman commented 2 months ago

Surprised to see the mono runtime being used at all for building the product - I thought we'd only be running on coreclr.

That's the purpose of this build leg: it first uses coreclr SDK to build an SDK with mono runtime enabled. Then it uses that built SDK to build everything again.

omajid commented 2 months ago

@swapnali911 I have seen this error a few times on ppc64le now. Are you aware/tracking/debugging this?

ellahathaway commented 2 months ago

That's a new one for me. It looks like it's coming from the mono runtime. https://github.com/dotnet/runtime/blob/66e45ef152002a2e1cbea939ded6643dfa48307c/src/mono/mono/metadata/loader.c#L2065 Surprised to see the mono runtime being used at all for building the product - I thought we'd only be running on coreclr.

@steveisok @agocke - soft ping about the comment above.

Swapnali911 commented 2 months ago

@Swapnali911 I have seen this error a few times on ppc64le now. Are you aware/tracking/debugging this?

Sorry @omajid, but no I've never seen this before.

steveisok commented 2 months ago

@ellahathaway the error indicates some kind of assembly corruption, but other than that it's hard to say why. Does this happen routinely? I no longer seeing it happen on the latest source build runs.

ellahathaway commented 2 months ago

@steveisok - we haven't seen this in our source-build builds since I filed this issue, so I don't think that I'd be able to replicate the error for you.

@omajid - you've mentioned that you've seen this a few times. Do you have access to a dump by chance?

omajid commented 2 months ago

Do you have access to a dump by chance?

I reproduced it on a ppc64le test machine using .NET 9 Preview 7 to build itself. I can test with a newer version if that will be helpful.

                  MSBUILD : error MSB1025: An internal failure occurred while running MSBuild.
                  Microsoft.Build.Framework.InternalErrorException: MSB0001: Internal MSBuild Error: Cannot acquire required number of nodes.
                  =============
                  System.AggregateException: One or more errors occurred. (Method has zero rva)
                   ---> Microsoft.Build.BackEnd.NodeFailedToLaunchException: Method has zero rva
                   ---> System.BadImageFormatException: Method has zero rva
                  File name: 'System.Diagnostics.Process'
                     at System.Runtime.Serialization.SerializationGuard.ThrowIfDeserializationInProgress(String switchSuffix, Int32& cachedValue) in /_/src/runtime/src/libraries/Common/src/System/Runtime/Serialization/SerializationGuard.cs:line 20
                     at System.Diagnostics.Process.Start() in /_/src/runtime/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1282
                     at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/runtime/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1352
                     at Microsoft.Build.BackEnd.NodeLauncher.StartInternal(String msbuildLocation, String commandLineArgs) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeLauncher.cs:line 121
                     --- End of inner exception stack trace ---
                     at Microsoft.Build.BackEnd.NodeLauncher.StartInternal(String msbuildLocation, String commandLineArgs) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeLauncher.cs:line 131
                     at Microsoft.Build.BackEnd.NodeLauncher.<>c__DisplayClass3_0.<Start>b__0() in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeLauncher.cs:line 42
                     at Microsoft.Build.BackEnd.NodeLauncher.DisableMSBuildServer(Func`1 func) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeLauncher.cs:line 203
                     at Microsoft.Build.BackEnd.NodeLauncher.Start(String msbuildLocation, String commandLineArgs, Int32 nodeId) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeLauncher.cs:line 42
                     at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.<>c__DisplayClass14_0.<GetNodes>g__StartNewNode|2(Int32 nodeId) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs:line 336
                     at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.<>c__DisplayClass14_0.<GetNodes>b__0(Int32 nodeId) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs:line 244
                     --- End of inner exception stack trace ---

                   ---> System.AggregateException: One or more errors occurred. (Method has zero rva)
                   ---> Microsoft.Build.BackEnd.NodeFailedToLaunchException: Method has zero rva
                   ---> System.BadImageFormatException: Method has zero rva
                  File name: 'System.Diagnostics.Process'
                     at System.Runtime.Serialization.SerializationGuard.ThrowIfDeserializationInProgress(String switchSuffix, Int32& cachedValue) in /_/src/runtime/src/libraries/Common/src/System/Runtime/Serialization/SerializationGuard.cs:line 20
                     at System.Diagnostics.Process.Start() in /_/src/runtime/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1282
                     at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/runtime/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1352
                     at Microsoft.Build.BackEnd.NodeLauncher.StartInternal(String msbuildLocation, String commandLineArgs) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeLauncher.cs:line 121
                     --- End of inner exception stack trace ---
                     at Microsoft.Build.BackEnd.NodeLauncher.StartInternal(String msbuildLocation, String commandLineArgs) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeLauncher.cs:line 131
                     at Microsoft.Build.BackEnd.NodeLauncher.<>c__DisplayClass3_0.<Start>b__0() in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeLauncher.cs:line 42
                     at Microsoft.Build.BackEnd.NodeLauncher.DisableMSBuildServer(Func`1 func) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeLauncher.cs:line 203
                     at Microsoft.Build.BackEnd.NodeLauncher.Start(String msbuildLocation, String commandLineArgs, Int32 nodeId) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeLauncher.cs:line 42
                     at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.<>c__DisplayClass14_0.<GetNodes>g__StartNewNode|2(Int32 nodeId) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs:line 336
                     at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.<>c__DisplayClass14_0.<GetNodes>b__0(Int32 nodeId) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs:line 244
                     --- End of inner exception stack trace ---
                     --- End of inner exception stack trace ---
                     at Microsoft.Build.Shared.ErrorUtilities.ThrowInternalError(String message, Exception innerException, Object[] args) in /_/src/msbuild/src/Shared/ErrorUtilities.cs:line 68
                     at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.GetNodes(String msbuildLocation, String commandLineArgs, Int32 nextNodeId, INodePacketFactory factory, Handshake hostHandshake, NodeContextCreatedDelegate createNode, NodeContextTerminateDelegate terminateNode, Int32 numberOfNodesToCreate) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs:line 258
                     at Microsoft.Build.BackEnd.NodeProviderOutOfProc.CreateNodes(Int32 nextNodeId, INodePacketFactory factory, Func`2 configurationFactory, Int32 numberOfNodesToCreate) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeProviderOutOfProc.cs:line 102
                     at Microsoft.Build.BackEnd.NodeManager.AttemptCreateNode(INodeProvider nodeProvider, NodeConfiguration nodeConfiguration, Int32 numberOfNodesToCreate) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeManager.cs:line 333
                     at Microsoft.Build.BackEnd.NodeManager.CreateNodes(NodeConfiguration configuration, NodeAffinity nodeAffinity, Int32 numberOfNodesToCreate) in /_/src/msbuild/src/Build/BackEnd/Components/Communications/NodeManager.cs:line 105
                     at Microsoft.Build.Execution.BuildManager.PerformSchedulingActions(IEnumerable`1 responses) in /_/src/msbuild/src/Build/BackEnd/BuildManager/BuildManager.cs:line 2615
                     at Microsoft.Build.Execution.BuildManager.HandleNewRequest(Int32 node, BuildRequestBlocker blocker) in /_/src/msbuild/src/Build/BackEnd/BuildManager/BuildManager.cs:line 2338
                     at Microsoft.Build.Execution.BuildManager.ProcessPacket(Int32 node, INodePacket packet) in /_/src/msbuild/src/Build/BackEnd/BuildManager/BuildManager.cs:line 1591
                     at Microsoft.Build.Execution.BuildManager.<>c__DisplayClass81_0.<Microsoft.Build.BackEnd.INodePacketHandler.PacketReceived>b__0() in /_/src/msbuild/src/Build/BackEnd/BuildManager/BuildManager.cs:line 1187
                     at Microsoft.Build.Execution.BuildManager.ProcessWorkQueue(Action action) in /_/src/msbuild/src/Build/BackEnd/BuildManager/BuildManager.cs:line 1544
                  --- End of stack trace from previous location ---
                     at Microsoft.Build.Execution.BuildManager.EndBuild() in /_/src/msbuild/src/Build/BackEnd/BuildManager/BuildManager.cs:line 1093
                     at Microsoft.Build.CommandLine.MSBuildApp.BuildProject(String projectFile, String[] targets, String toolsVersion, Dictionary`2 globalProperties, Dictionary`2 restoreProperties, ILogger[] loggers, LoggerVerbosity verbosity, DistributedLoggerRecord[] distributedLoggerRecords, Int32 cpuCount, Boolean enableNodeReuse, TextWriter preprocessWriter, TextWriter targetsWriter, Boolean detailedSummary, ISet`1 warningsAsErrors, ISet`1 warningsNotAsErrors, ISet`1 warningsAsMessages, Boolean enableRestore, ProfilerLogger profilerLogger, Boolean enableProfiler, Boolean interactive, ProjectIsolationMode isolateProjects, GraphBuildOptions graphBuildOptions, Boolean lowPriority, Boolean question, Boolean isBuildCheckEnabled, String[] inputResultsCaches, String outputResultsCache, Boolean saveProjectResult, BuildResult& result, String[] commandLine) in /_/src/msbuild/src/MSBuild/XMake.cs:line 1628

...

    24>/home/omajid/dotnet9.0/dotnet-9.0.0-preview.7.24405.7/repo-projects/Directory.Build.targets(428,5): error MSB3073: The command "/home/omajid/dotnet9.0/dotnet-9.0.0-preview.7.24405.7/src/cecil/eng/common/build.sh --restore --build --pack --publish --ci --configuration Release -bl /p:DotNetBuildRepo=true /p:DotNetBuildOrchestrator=true /p:RestoreConfigFile=/home/omajid/dotnet9.0/dotnet-9.0.0-preview.7.24405.7/artifacts/obj/cecil/NuGet.config /p:SourceBuildUseMonoRuntime=true /p:SourceBuiltAssetsDir=/home/omajid/dotnet9.0/dotnet-9.0.0-preview.7.24405.7/artifacts/assets/Release/ /p:SourceBuiltAssetManifestsDir=/home/omajid/dotnet9.0/dotnet-9.0.0-preview.7.24405.7/artifacts/obj/AssetManifests/cecil/ /p:SourceBuiltSymbolsDir=/home/omajid/dotnet9.0/dotnet-9.0.0-preview.7.24405.7/artifacts/obj/Symbols/cecil/ /p:ArcadeBuildFromSource=true /p:DotNetBuildSourceOnly=true /p:PreviouslySourceBuiltNupkgCacheDir="/home/omajid/dotnet9.0/dotnet-9.0.0-preview.7.24405.7/prereqs/packages/previously-source-built/" /p:ReferencePackageNupkgCacheDir="/home/omajid/dotnet9.0/dotnet-9.0.0-preview.7.24405.7/prereqs/packages/reference/" /p:SourceBuiltShippingPackagesDir=/home/omajid/dotnet9.0/dotnet-9.0.0-preview.7.24405.7/artifacts/packages/Release/Shipping/cecil/ /p:SourceBuiltNonShippingPackagesDir=/home/omajid/dotnet9.0/dotnet-9.0.0-preview.7.24405.7/artifacts/packages/Release/NonShipping/cecil/ /v:n" exited with code 1. [/home/omajid/dotnet9.0/dotnet-9.0.0-preview.7.24405.7/repo-projects/cecil.proj]

Here's the binlogs from the cecil project: binlogs.tar.gz

MichaelSimons commented 1 month ago

@steveisok - The SB team is a bit of a middle man here. @omajid has a repo and is willing to help diagnose. Can I assign to someone on the runtime or transfer to the runtime?

steveisok commented 1 month ago

@steveisok - The SB team is a bit of a middle man here. @omajid has a repo and is willing to help diagnose. Can I assign to someone on the runtime or transfer to the runtime?

Feel free to transfer it to runtime.