search

dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.1k stars 9.91k forks source link

'The process cannot access the file' during CI build sometimes #32219

Open BrennanConroy opened 3 years ago

BrennanConroy commented 3 years ago

This has been happening a few times recently. It's odd because BlazorTemplates.Tests doesn't depend on Microsoft.DotNet.Web.Spa.ProjectTemplates so it shouldn't be copying it anywhere.

There is some custom msbuild logic being invoked right before the error: https://github.com/dotnet/aspnetcore/blob/main/src/ProjectTemplates/TestInfrastructure/PrepareForTest.targets So maybe that has something to do with it. We might just need to add /bl to this part of the build temporarily.

Example build: https://dev.azure.com/dnceng/public/_build/results?buildId=1106179&view=logs&j=48c285a2-daf4-5ecb-cb99-42232769ca29&t=73646f8d-7520-5361-e1ad-d9d207b2cd21

./src/ProjectTemplates/build.sh --ci --nobl --pack --no-restore --no-build-deps
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /datadisks/disk1/workspace/_work/_temp/7404866d-8c11-4324-8bfe-b62b36813c80.sh
  RepoTasks -> /datadisks/disk1/workspace/_work/1/s/artifacts/bin/RepoTasks/Release/net6.0/RepoTasks.dll
  Successfully created package '/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.Client.ItemTemplates.6.0.0-ci.nupkg'.
  Successfully created package '/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.ItemTemplates.6.0.0-ci.nupkg'.
  Preparing environment for tests
  Removed directory /datadisks/disk1/workspace/_work/1/s/src/ProjectTemplates/BlazorTemplates.Tests/bin/Release/net6.0/TestTemplates/
  Removed directory /datadisks/disk1/workspace/_work/1/s/obj/template-restore/
  Created directory /datadisks/disk1/workspace/_work/1/s/src/ProjectTemplates/BlazorTemplates.Tests/bin/Release/net6.0/TestTemplates/
  Created directory /datadisks/disk1/workspace/_work/1/s/obj/template-restore/
  Successfully created package '/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.ProjectTemplates.6.0.6.0.0-ci.nupkg'.
/datadisks/disk1/workspace/_work/1/s/.dotnet/sdk/6.0.100-preview.4.21222.6/Microsoft.Common.CurrentVersion.targets(4965,5): error MSB3026: Could not copy "/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.Spa.ProjectTemplates.6.0.6.0.0-ci.nupkg" to "bin/Release/net6.0/Microsoft.DotNet.Web.Spa.ProjectTemplates.6.0.6.0.0-ci.nupkg". Beginning retry 1 in 1000ms. The process cannot access the file '/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.Spa.ProjectTemplates.6.0.6.0.0-ci.nupkg' because it is being used by another process.  [/datadisks/disk1/workspace/_work/1/s/src/ProjectTemplates/BlazorTemplates.Tests/BlazorTemplates.Tests.csproj]
##[error].dotnet/sdk/6.0.100-preview.4.21222.6/Microsoft.Common.CurrentVersion.targets(4965,5): error MSB3026: (NETCORE_ENGINEERING_TELEMETRY=Build) Could not copy "/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.Spa.ProjectTemplates.6.0.6.0.0-ci.nupkg" to "bin/Release/net6.0/Microsoft.DotNet.Web.Spa.ProjectTemplates.6.0.6.0.0-ci.nupkg". Beginning retry 1 in 1000ms. The process cannot access the file '/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.Spa.ProjectTemplates.6.0.6.0.0-ci.nupkg' because it is being used by another process. 
  Successfully created package '/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.Spa.ProjectTemplates.6.0.6.0.0-ci.nupkg'.
  Preparing environment for tests
  Removed directory /datadisks/disk1/workspace/_work/1/s/src/ProjectTemplates/test/bin/Release/net6.0/TestTemplates/
  Removed directory /datadisks/disk1/workspace/_work/1/s/obj/template-restore/
  Created directory /datadisks/disk1/workspace/_work/1/s/src/ProjectTemplates/test/bin/Release/net6.0/TestTemplates/
  Created directory /datadisks/disk1/workspace/_work/1/s/obj/template-restore/
  BlazorTemplates.Tests -> /datadisks/disk1/workspace/_work/1/s/src/ProjectTemplates/BlazorTemplates.Tests/bin/Release/net6.0/BlazorTemplates.Tests.dll
  ProjectTemplates.Tests -> /datadisks/disk1/workspace/_work/1/s/src/ProjectTemplates/test/bin/Release/net6.0/ProjectTemplates.Tests.dll
dougbu commented 3 years ago

@JunTaoLuo this may be the start of the general issue around file contention in our builds that we discussed yesterday.

JunTaoLuo commented 3 years ago

We're hitting this somewhat frequently. Today there were two occurences: https://dev.azure.com/dnceng/public/_build/results?buildId=1172134&view=results, https://dev.azure.com/dnceng/public/_build/results?buildId=1172278&view=results

TanayParikh commented 3 years ago

Bin Logs:

CI: Pipelines - https://dev.azure.com/dnceng/public/_build/results?buildId=1176795&view=artifacts&pathAsName=false&type=publishedArtifacts

Commit Used: 0d75fcb05908ec54f1530b64b0158215f0ec3173

Original Failure: https://dev.azure.com/dnceng/public/_build/results?buildId=1172134&view=results

dougbu commented 3 years ago

Thought someone considered putting up a PR to implement @rainersigwald's idea (thanks @rainersigwald). Was that dotnet/aspnetcore#33565 @pranavkm❔

Given that's done, do we have additional monitoring to do to ensure we addressed the main problem or to find additional build contention problems❔ Basically wondering if we can close this or need an over-arching "our build contends w/ itself (ick ❕) issue❔

If we don't need to track other build ordering / contention issues (@Pilchie @dotnet/aspdoi your thoughts ❔), suggest closing this near the end of your ops rotation @BrennanConroy.

pranavkm commented 3 years ago

Was that #33565 @pranavkm❔

Yeah, that was based on @rainersigwald's analysis. He had suggested making the outputs different, after playing around with it enough I realized we didn't need two sets of them.

BrennanConroy commented 3 years ago

Today:

##[error].dotnet/sdk/6.0.100-preview.6.21313.2/Microsoft.Common.CurrentVersion.targets(2202,5):
error MSB3248: (NETCORE_ENGINEERING_TELEMETRY=Build) Parameter "AssemblyFiles" has invalid value 
"/Users/runner/work/1/s/artifacts/bin/Microsoft.AspNetCore.Hosting/Release/net6.0/Microsoft.AspNetCore.Hosting.dll".
The process cannot access the file '/Users/runner/work/1/s/artifacts/bin/Microsoft.AspNetCore.Hosting/Release/net6.0/Microsoft.AspNetCore.Hosting.dll'
because it is being used by another process.

https://dev.azure.com/dnceng/public/_build/results?buildId=1198450&view=logs&jobId=f43b187f-6c15-531c-a59b-e4fba4e9ed20&j=f43b187f-6c15-531c-a59b-e4fba4e9ed20&t=98a09275-a4dd-593e-c21a-1f8bea3be5b1

Eilon commented 3 years ago

Failing on multiple runs in my PR: https://dev.azure.com/dnceng/public/_build/results?buildId=1199902&view=logs&jobId=f43b187f-6c15-531c-a59b-e4fba4e9ed20&j=f43b187f-6c15-531c-a59b-e4fba4e9ed20&t=98a09275-a4dd-593e-c21a-1f8bea3be5b1

wtgodbe commented 3 years ago

Not necessarily fixed yet, we've seen the issue on other projects other than just BasicTestApp

TanayParikh commented 3 years ago

Do you know which other projects? Maybe just a matter of adding this to the appropriate references?

https://github.com/dotnet/aspnetcore/pull/34137#discussion_r664927578

TanayParikh commented 3 years ago 
eng/AfterSolutionBuild.targets(15,5): error MSB4018: (NETCORE_ENGINEERING_TELEMETRY=Build) The "RepoTasks.RemoveSharedFrameworkDependencies" task failed unexpectedly.
System.IO.IOException: The process cannot access the file '/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.AspNetCore.App.Runtime.linux-x64.6.0.0-ci.nupkg' because it is being used by another process.
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Init(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize) in System.Private.CoreLib.dll:token 0x60000ce+0x0
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize) in System.Private.CoreLib.dll:token 0x60000cc+0x31
   at System.IO.Strategies.Net5CompatFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize) in System.Private.CoreLib.dll:token 0x600601b+0x6f
   at System.IO.File.Open(String path, FileMode mode) in System.Private.CoreLib.dll:token 0x6005c1d+0x0
   at RepoTasks.RemoveSharedFrameworkDependencies.FilterDependencies(String targetPath, ISet`1 dependencyToRemove) in /_/eng/tools/RepoTasks/RemoveSharedFrameworkDependencies.cs:line 48
   at RepoTasks.RemoveSharedFrameworkDependencies.Execute() in /_/eng/tools/RepoTasks/RemoveSharedFrameworkDependencies.cs:line 38
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() in Microsoft.Build.dll:token 0x60015fc+0x3e
   at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) in Microsoft.Build.dll:token 0x600147b+0x804

Seen in https://dev.azure.com/dnceng/public/_build/results?buildId=1224315&view=logs&jobId=b72e85ab-3386-5aa9-6405-3837662d9688

TanayParikh commented 3 years ago

Still failing on BasicTestApp

D:\workspace\_work\1\s\.dotnet\sdk\6.0.100-preview.7.21356.3\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.Current.targets(637,5): error : The process cannot access the file 'D:\workspace\_work\1\s\artifacts\obj\BasicTestApp\Release\net6.0\compress\J3Pxu+Zq.gz' because it is being used by another process. [D:\workspace\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
##[error].dotnet\sdk\6.0.100-preview.7.21356.3\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.Current.targets(637,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) The process cannot access the file 'D:\workspace\_work\1\s\artifacts\obj\BasicTestApp\Release\net6.0\compress\J3Pxu+Zq.gz' because it is being used by another process.

https://dev.azure.com/dnceng/public/_build/results?buildId=1225341&view=logs&j=3f6d4e0f-1b71-56b5-361e-d95b6e6da15a&t=9605d305-57d3-55b8-ba79-e50442d32137&l=2220

https://github.com/dotnet/aspnetcore/pull/34033/checks?check_run_id=3013535904

Created PR to generate bin logs: https://github.com/dotnet/aspnetcore/pull/34189

Eilon commented 3 years ago

And still failing on my PR: https://github.com/dotnet/aspnetcore/pull/33482/checks?check_run_id=3080384712

dougbu commented 3 years ago

Failing in rolling builds too e.g. https://dev.azure.com/dnceng/public/_build/results?buildId=1242042&view=results

/datadisks/disk1/workspace/_work/1/s/.dotnet/sdk/6.0.100-preview.7.21364.4/Sdks/Microsoft.NET.Sdk.BlazorWebAssembly/targets/Microsoft.NET.Sdk.BlazorWebAssembly.Current.targets(637,5): error : The process cannot access the file '/datadisks/disk1/workspace/_work/1/s/artifacts/obj/BasicTestApp/Release/net6.0/compress/5P5wgw90.gz' because it is being used by another process. [/datadisks/disk1/workspace/_work/1/s/src/Components/test/testassets/BasicTestApp/BasicTestApp.csproj]

@HaoK thoughts on taking BasicTestApp out of the equation again❔ We had things in a better state at one point but the problem got worse after we re-enabled use of this app and tried to fix the underlying issue.

BrennanConroy commented 2 years ago

There are plenty more than this, this is just a few of the "cannot access the file" issues from the past couple days.

Tests: Helix x64, Build shared fx

D:\workspace\_work\1\s\.dotnet\sdk\6.0.100-rtm.21466.6\Microsoft.Common.CurrentVersion.targets(5417,5): error MSB3491: Could not write lines to file "D:\workspace\_work\1\s\artifacts\obj\Microsoft.JSInterop.WebAssembly\Release\net6.0\Microsoft.JSInterop.WebAssembly.csproj.FileListAbsolute.txt". The process cannot access the file 'D:\workspace\_work\1\s\artifacts\obj\Microsoft.JSInterop.WebAssembly\Release\net6.0\Microsoft.JSInterop.WebAssembly.csproj.FileListAbsolute.txt' because it is being used by another process. [D:\workspace\_work\1\s\src\Components\WebAssembly\JSInterop\src\Microsoft.JSInterop.WebAssembly.csproj]
##[error].dotnet\sdk\6.0.100-rtm.21466.6\Microsoft.Common.CurrentVersion.targets(5417,5): error MSB3491: (NETCORE_ENGINEERING_TELEMETRY=Build) Could not write lines to file "D:\workspace\_work\1\s\artifacts\obj\Microsoft.JSInterop.WebAssembly\Release\net6.0\Microsoft.JSInterop.WebAssembly.csproj.FileListAbsolute.txt". The process cannot access the file 'D:\workspace\_work\1\s\artifacts\obj\Microsoft.JSInterop.WebAssembly\Release\net6.0\Microsoft.JSInterop.WebAssembly.csproj.FileListAbsolute.txt' because it is being used by another process.

Tests: Helix x64, Build shared fx

D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018: The "GetFileHash" task failed unexpectedly. [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018: System.AggregateException: One or more errors occurred. (The process cannot access the file 'D:\a\_work\1\s\artifacts\obj\BasicTestApp\Release\net7.0\linked\System.Private.CoreLib.dll' because it is being used by another process.) [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:  ---> System.IO.IOException: The process cannot access the file 'D:\a\_work\1\s\artifacts\obj\BasicTestApp\Release\net7.0\linked\System.Private.CoreLib.dll' because it is being used by another process. [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options) [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize) [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize) [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at Microsoft.Build.Tasks.GetFileHash.ComputeHash(Func`1 algorithmFactory, String filePath) [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at Microsoft.Build.Tasks.GetFileHash.<>c__DisplayClass28_0.<Execute>b__0(Int32 index) [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018: --- End of stack trace from previous location --- [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at System.Threading.Tasks.TaskReplicator.Replica.Execute() [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    --- End of inner exception stack trace --- [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 action, ParallelOptions options, Boolean stopOnFirstFailure) [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally) [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018: --- End of stack trace from previous location --- [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException) [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally) [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at System.Threading.Tasks.Parallel.For(Int32 fromInclusive, Int32 toExclusive, Action`1 body) [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at Microsoft.Build.Tasks.GetFileHash.Execute() [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
##[error].dotnet\sdk\7.0.100-alpha.1.21463.2\Sdks\Microsoft.NET.Sdk.BlazorWebAssembly\targets\Microsoft.NET.Sdk.BlazorWebAssembly.6_0.targets(555,5): error MSB4018: (NETCORE_ENGINEERING_TELEMETRY=Build) The "GetFileHash" task failed unexpectedly.

Test: Windows 2016 x64, Run build.cmd

D:\workspace\_work\1\s\.dotnet\sdk\6.0.100-rtm.21466.6\Microsoft.Common.CurrentVersion.targets(2315,5): error MSB3248: Parameter "AssemblyFiles" has invalid value "D:\workspace\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Hosting\Release\net6.0\Microsoft.AspNetCore.Hosting.dll". The process cannot access the file 'D:\workspace\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Hosting\Release\net6.0\Microsoft.AspNetCore.Hosting.dll' because it is being used by another process. [D:\workspace\_work\1\s\src\SignalR\samples\SocialWeather\SocialWeather.csproj]
##[error].dotnet\sdk\6.0.100-rtm.21466.6\Microsoft.Common.CurrentVersion.targets(2315,5): error MSB3248: (NETCORE_ENGINEERING_TELEMETRY=Build) Parameter "AssemblyFiles" has invalid value "D:\workspace\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Hosting\Release\net6.0\Microsoft.AspNetCore.Hosting.dll". The process cannot access the file 'D:\workspace\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Hosting\Release\net6.0\Microsoft.AspNetCore.Hosting.dll' because it is being used by another process.

Test: macOS 10.15, Run build.sh

CSC : error CS0009: Metadata file '/Users/runner/work/1/s/artifacts/bin/Microsoft.AspNetCore.Server.Kestrel.Transport.Quic/Release/net6.0/ref/Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.dll' could not be opened -- The process cannot access the file '/Users/runner/work/1/s/artifacts/bin/Microsoft.AspNetCore.Server.Kestrel.Transport.Quic/Release/net6.0/ref/Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.dll' because it is being used by another process. [/Users/runner/work/1/s/src/Middleware/Localization/testassets/LocalizationWebsite/LocalizationWebsite.csproj]
##[error]CSC(0,0): error CS0009: (NETCORE_ENGINEERING_TELEMETRY=Build) Metadata file '/Users/runner/work/1/s/artifacts/bin/Microsoft.AspNetCore.Server.Kestrel.Transport.Quic/Release/net6.0/ref/Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.dll' could not be opened -- The process cannot access the file '/Users/runner/work/1/s/artifacts/bin/Microsoft.AspNetCore.Server.Kestrel.Transport.Quic/Release/net6.0/ref/Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.dll' because it is being used by another process.

Tests: Helix x64, Build shared fx

D:\workspace\_work\1\s\.dotnet\sdk\6.0.100-rtm.21466.6\Microsoft.Common.CurrentVersion.targets(5417,5): error MSB3491: Could not write lines to file "D:\workspace\_work\1\s\artifacts\obj\Microsoft.JSInterop.WebAssembly\Release\net6.0\Microsoft.JSInterop.WebAssembly.csproj.FileListAbsolute.txt". The process cannot access the file 'D:\workspace\_work\1\s\artifacts\obj\Microsoft.JSInterop.WebAssembly\Release\net6.0\Microsoft.JSInterop.WebAssembly.csproj.FileListAbsolute.txt' because it is being used by another process. [D:\workspace\_work\1\s\src\Components\WebAssembly\JSInterop\src\Microsoft.JSInterop.WebAssembly.csproj]
##[error].dotnet\sdk\6.0.100-rtm.21466.6\Microsoft.Common.CurrentVersion.targets(5417,5): error MSB3491: (NETCORE_ENGINEERING_TELEMETRY=Build) Could not write lines to file "D:\workspace\_work\1\s\artifacts\obj\Microsoft.JSInterop.WebAssembly\Release\net6.0\Microsoft.JSInterop.WebAssembly.csproj.FileListAbsolute.txt". The process cannot access the file 'D:\workspace\_work\1\s\artifacts\obj\Microsoft.JSInterop.WebAssembly\Release\net6.0\Microsoft.JSInterop.WebAssembly.csproj.FileListAbsolute.txt' because it is being used by another process.

Test: Windows Server 2016 x64, Run build.cmd

D:\workspace\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21463.2\Microsoft.Common.CurrentVersion.targets(4646,5): error MSB3026: Could not copy "D:\workspace\_work\1\s\artifacts\obj\Microsoft.AspNetCore.Connections.Abstractions\Release\net6.0\Microsoft.AspNetCore.Connections.Abstractions.dll" to "D:\workspace\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Connections.Abstractions\Release\net6.0\Microsoft.AspNetCore.Connections.Abstractions.dll". Beginning retry 1 in 1000ms. The requested operation cannot be performed on a file with a user-mapped section open. : 'D:\workspace\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Connections.Abstractions\Release\net6.0\Microsoft.AspNetCore.Connections.Abstractions.dll'  [D:\workspace\_work\1\s\src\Servers\Connections.Abstractions\src\Microsoft.AspNetCore.Connections.Abstractions.csproj]
##[error].dotnet\sdk\7.0.100-alpha.1.21463.2\Microsoft.Common.CurrentVersion.targets(4646,5): error MSB3026: (NETCORE_ENGINEERING_TELEMETRY=Build) Could not copy "D:\workspace\_work\1\s\artifacts\obj\Microsoft.AspNetCore.Connections.Abstractions\Release\net6.0\Microsoft.AspNetCore.Connections.Abstractions.dll" to "D:\workspace\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Connections.Abstractions\Release\net6.0\Microsoft.AspNetCore.Connections.Abstractions.dll". Beginning retry 1 in 1000ms. The requested operation cannot be performed on a file with a user-mapped section open. : 'D:\workspace\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Connections.Abstractions\Release\net6.0\Microsoft.AspNetCore.Connections.Abstractions.dll'
dougbu commented 2 years ago

Have seen another variant of this in the last few days, most recently in validation of #37412. We're back to what #34137 should have fixed ☹️ though ILLink may also have regressed (@eerhardt @ViktorHofer❔)

ILLink : error IL1012: IL Trimmer has encountered an unexpected error. Please report the issue at https://github.com/dotnet/linker/issues [D:\a\_work\2\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
##[error]ILLink(0,0): error IL1012: (NETCORE_ENGINEERING_TELEMETRY=Build) IL Trimmer has encountered an unexpected error. Please report the issue at https://github.com/dotnet/linker/issues
  Fatal error in IL Linker
  Unhandled exception. Mono.Linker.InternalErrorException: Step 'OutputStep' failed when processing assembly 'System.Private.Xml, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
   ---> Mono.Linker.LinkerFatalErrorException: ILLink: error IL1011: Failed to write 'D:\a\_work\2\s\artifacts\obj\BasicTestApp\Release\net7.0\linked\System.Private.Xml.dll'.
   ---> System.IO.IOException: The process cannot access the file 'D:\a\_work\2\s\artifacts\obj\BasicTestApp\Release\net7.0\linked\System.Private.Xml.dll' because it is being used by another process.
     at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
     at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
     at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
     at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
     at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
     at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
     at Mono.Cecil.ModuleDefinition.GetFileStream(String fileName, FileMode mode, FileAccess access, FileShare share)
     at Mono.Cecil.ModuleDefinition.Write(String fileName, WriterParameters parameters)
     at Mono.Cecil.AssemblyDefinition.Write(String fileName, WriterParameters parameters)
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory, WriterParameters writerParameters)
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory, WriterParameters writerParameters)
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory)
     at Mono.Linker.Steps.OutputStep.OutputAssembly(AssemblyDefinition assembly)
     at Mono.Linker.Steps.OutputStep.ProcessAssembly(AssemblyDefinition assembly)
     at Mono.Linker.Steps.BaseStep.Process(LinkContext context)
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.BaseStep.Process(LinkContext context)
     at Mono.Linker.Pipeline.ProcessStep(LinkContext context, IStep step)
     at Mono.Linker.Pipeline.Process(LinkContext context)
eerhardt commented 2 years ago

FYI @agocke @vitek-karas @sbomer

@dougbu - do you happen to have a binlog or repro steps to recreate the error?

agocke commented 2 years ago

There's only one place we use memory mapping in the linker and it looks like that file hasn't changed since Aug 31, so it seems unlikely this is a linker change.

dougbu commented 2 years ago

@dougbu - do you happen to have a binlog or repro steps to recreate the error?

That pipeline doesn't capture binary logs but there's a possibly-related dotnet core dump at https://dev.azure.com/dnceng/_apis/resources/Containers/8427220/Helix_quarantined_x64_Dumps?itemPath=Helix_quarantined_x64_Dumps%2Fdotnet-5452.1633736208.core

dougbu commented 2 years ago

The problem doesn't reproduce reliably. I've seen a couple of instances like this but problems go away on retry.

halter73 commented 2 years ago

Here's another instance of this issue:

##[error]ILLink(0,0): error IL1012: (NETCORE_ENGINEERING_TELEMETRY=Build) IL Trimmer has encountered an unexpected error. Please report the issue at https://github.com/dotnet/linker/issues
  Fatal error in IL Linker
  Unhandled exception. Mono.Linker.InternalErrorException: Step 'OutputStep' failed when processing assembly 'Microsoft.Extensions.Configuration, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
   ---> Mono.Linker.LinkerFatalErrorException: ILLink: error IL1011: Failed to write 'D:\a\_work\1\s\artifacts\obj\BasicTestApp\Release\net7.0\linked\Microsoft.Extensions.Configuration.dll'.
   ---> System.IO.IOException: The process cannot access the file 'D:\a\_work\1\s\artifacts\obj\BasicTestApp\Release\net7.0\linked\Microsoft.Extensions.Configuration.dll' because it is being used by another process.
     at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
     at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
     at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
     at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
     at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
     at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
     at Mono.Cecil.ModuleDefinition.GetFileStream(String fileName, FileMode mode, FileAccess access, FileShare share)
     at Mono.Cecil.ModuleDefinition.Write(String fileName, WriterParameters parameters)
     at Mono.Cecil.AssemblyDefinition.Write(String fileName, WriterParameters parameters)
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory, WriterParameters writerParameters)
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory, WriterParameters writerParameters)
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory)
     at Mono.Linker.Steps.OutputStep.OutputAssembly(AssemblyDefinition assembly)
     at Mono.Linker.Steps.OutputStep.ProcessAssembly(AssemblyDefinition assembly)
     at Mono.Linker.Steps.BaseStep.Process(LinkContext context)
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.BaseStep.Process(LinkContext context)
     at Mono.Linker.Pipeline.ProcessStep(LinkContext context, IStep step)
     at Mono.Linker.Pipeline.Process(LinkContext context)
     at Mono.Linker.Driver.Run(ILogger customLogger)
     at Mono.Linker.Driver.Main(String[] args)
  Optimizing assemblies for size, which may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
  Wasm.Prerendered.Server -> D:\a\_work\1\s\artifacts\bin\Wasm.Prerendered.Server\Release\net7.0\Wasm.Prerendered.Server.dll
  Optimizing assemblies for size, which may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
D:\a\_work\1\s\.dotnet\sdk\7.0.100-alpha.1.21518.14\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ILLink.targets(108,5): error NETSDK1144: Optimizing assemblies for size failed. Optimization can be disabled by setting the PublishTrimmed property to false. [D:\a\_work\1\s\src\Components\test\testassets\BasicTestApp\BasicTestApp.csproj]
##[error].dotnet\sdk\7.0.100-alpha.1.21518.14\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ILLink.targets(108,5): error NETSDK1144: (NETCORE_ENGINEERING_TELEMETRY=Build) Optimizing assemblies for size failed. Optimization can be disabled by setting the PublishTrimmed property to false.

https://dev.azure.com/dnceng/public/_build/results?buildId=1428866&view=logs&j=6c33d704-163a-5170-38fa-caa7bc99f17c&t=4a2ac175-72f0-54eb-c8b6-eb1e5d86c7d5

And another with the SelfHostDeployer instead of the linker:

| [0.016s] TestLifetime Information: Starting test CanDelegateOutOfProcess at 2021-10-18T23:41:59
| [0.001s] TestLifetime Information: Starting test CanDelegateOutOfProcess at 2021-10-18T23:41:59
| [0.010s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: Deploying [Variation] :: ServerType=HttpSys, Runtime=CoreClr, Arch=x64, BaseUrlHint=, Publish=True
| [0.015s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: Executing command dotnet publish  --output "C:\Users\cloudtest\AppData\Local\Temp\9998662e73ef4e0497e1ddd1cf0a15d0" --framework net6.0 --configuration Release --no-dependencies /p:TargetArchitecture=x64 --no-restore -p:UseAppHost=false 
| [0.274s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout: Microsoft (R) Build Engine version 17.0.0-preview-21477-03+e123a0c1f for .NET
| [0.274s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout: Copyright (C) Microsoft Corporation. All rights reserved.
| [5.953s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout:   Microsoft.Net.Http.Headers -> D:\a\_work\1\s\artifacts\bin\Microsoft.Net.Http.Headers\Release\net6.0\Microsoft.Net.Http.Headers.dll
| [6.507s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout:   Microsoft.Extensions.Features -> D:\a\_work\1\s\artifacts\bin\Microsoft.Extensions.Features\Release\net6.0\Microsoft.Extensions.Features.dll
| [7.056s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout:   Microsoft.AspNetCore.Connections.Abstractions -> D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Connections.Abstractions\Release\net6.0\Microsoft.AspNetCore.Connections.Abstractions.dll
| [7.218s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout:   Microsoft.Extensions.ObjectPool -> D:\a\_work\1\s\artifacts\bin\Microsoft.Extensions.ObjectPool\Release\net6.0\Microsoft.Extensions.ObjectPool.dll
| [7.956s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout:   Microsoft.AspNetCore.WebUtilities -> D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.WebUtilities\Release\net6.0\Microsoft.AspNetCore.WebUtilities.dll
| [8.127s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout:   Microsoft.AspNetCore.Http.Features -> D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Http.Features\Release\net6.0\Microsoft.AspNetCore.Http.Features.dll
| [9.464s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout:   Microsoft.AspNetCore.Http.Abstractions -> D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Http.Abstractions\Release\net6.0\Microsoft.AspNetCore.Http.Abstractions.dll
| [9.483s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout:   Microsoft.AspNetCore.Hosting.Server.Abstractions -> D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Hosting.Server.Abstractions\Release\net6.0\Microsoft.AspNetCore.Hosting.Server.Abstractions.dll
| [10.448s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout:   Microsoft.AspNetCore.Authentication.Abstractions -> D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Authentication.Abstractions\Release\net6.0\Microsoft.AspNetCore.Authentication.Abstractions.dll
| [11.091s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout:   Microsoft.AspNetCore.Hosting.Abstractions -> D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Hosting.Abstractions\Release\net6.0\Microsoft.AspNetCore.Hosting.Abstractions.dll
| [11.505s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout:   Microsoft.AspNetCore.Http -> D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Http\Release\net6.0\Microsoft.AspNetCore.Http.dll
| [11.623s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout:   Microsoft.AspNetCore.Http.Extensions -> D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Http.Extensions\Release\net6.0\Microsoft.AspNetCore.Http.Extensions.dll
| [13.257s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout: CSC : error CS2012: Cannot open 'D:\a\_work\1\s\artifacts\obj\Microsoft.AspNetCore.Hosting\Release\net6.0\Microsoft.AspNetCore.Hosting.dll' for writing -- 'The process cannot access the file 'D:\a\_work\1\s\artifacts\obj\Microsoft.AspNetCore.Hosting\Release\net6.0\Microsoft.AspNetCore.Hosting.dll' because it is being used by another process.' [D:\a\_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
| [13.938s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: dotnet-publish stdout:   Microsoft.AspNetCore.Authentication.Core -> D:\a\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Authentication.Core\Release\net6.0\Microsoft.AspNetCore.Authentication.Core.dll
| [15.697s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Error: dotnet publish exited with exit code : 1
| [15.701s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Warning: Host process already exited or never started successfully.
| [15.702s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: [Time]: Total time taken for this test variation '15.6917664' seconds
| [15.703s] TestLifetime Information: Finished test CanDelegateOutOfProcess in 15.7012342s
| [15.735s] Microsoft.AspNetCore.Server.HttpSys.NonHelixTests.DelegateOutOfProcTests Error: Test threw an exception.
| System.Exception: dotnet publish exited with exit code : 1
|    at Microsoft.AspNetCore.Server.IntegrationTesting.ApplicationPublisher.Publish(DeploymentParameters deploymentParameters, ILogger logger) in /_/src/Hosting/Server.IntegrationTesting/src/ApplicationPublisher.cs:line 90
|    at Microsoft.AspNetCore.Server.IntegrationTesting.ApplicationDeployer.DotnetPublish(String publishRoot) in /_/src/Hosting/Server.IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 87
|    at Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer.DeployAsync() in /_/src/Hosting/Server.IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 59
|    at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) in /_/src/Shared/TaskExtensions.cs:line 85
|    at Microsoft.AspNetCore.Server.HttpSys.NonHelixTests.DelegateOutOfProcTests.CanDelegateOutOfProcess() in /_/src/Servers/HttpSys/test/NonHelixTests/DelegateOutOfProcTests.cs:line 46
|    at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<<InvokeTestMethodAsync>b__1>d.MoveNext() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 264
| --- End of stack trace from previous location ---
|    at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 48
|    at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 90
| [15.775s] TestLifetime Information: Finished test CanDelegateOutOfProcess in 15.7686839s

https://dev.azure.com/dnceng/public/_build/results?buildId=1428585&view=ms.vss-test-web.build-test-results-tab&runId=41304146&resultId=105545&paneView=attachments

I considered quarantining CanDelegateOutOfProcess, but it looks to be a general issue with SelfHostDeployer.

dougbu commented 2 years ago

@wtgodbe and / or @halter73 please have a look at the dumps for these recent failures (https://dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_apis/build/builds/1428866/artifacts?artifactName=Windows_Test_Dumps&api-version=6.0&%24format=zip and https://dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_apis/build/builds/1428585/artifacts?artifactName=Windows_Test_Dumps&api-version=6.0&%24format=zip) as well as the binary logs (https://dev.azure.com/dnceng/_apis/resources/Containers/8514346/Windows_Test_Logs?itemPath=Windows_Test_Logs%2FRelease%2FBuild.binlog; second build targeted release/6.0 and we don't gather test binary logs there).

wtgodbe commented 2 years ago

Where did you get the dumps? The binlog at least is from attempt #2, which was a success. I don't see anything interesting (yet) in the dump I'm looking at

dougbu commented 2 years ago

Ah, rerun overwrote the _Logs artifact 😦 I got the dumps from the Windows_Test_Dumps artifact. Went looking there because we capture dotnet dumps now and the process didn't exit normally according to the console logs for the first attempt.

Might be interesting to include the attempt number in some of the artifact names❔

wtgodbe commented 2 years ago

Might be interesting to include the attempt number in some of the artifact names❔

Looks like source build does that already. I think it's probably ok to do that - we would be increasing the amount of stuff we upload, but wouldn't be going beyond what one would typically expect (that is, we wouldn't be uploading more sets of logs than jobs ran. Whereas today we may upload fewer)

TanayParikh commented 2 years ago

Is:

/private/tmp/helix/working/B54D0A27/p/dotnet-cli/sdk/7.0.100-alpha.1.21518.14/NuGet.targets(130,5): error : The file '/private/tmp/helix/working/B54D0A27/w/AC3B0975/e/Templates/BaseFolder/AspNet.gkbjaf22lj3/Shared/obj/project.nuget.cache' already exists. [/tmp/helix/working/B54D0A27/w/AC3B0975/e/Templates/BaseFolder/AspNet.gkbjaf22lj3/AspNet.gkbjaf22lj3.sln]

the same issue, or should I create a new one?

https://dev.azure.com/dnceng/public/_build/results?buildId=1432636&view=ms.vss-test-web.build-test-results-tab&runId=41386032&resultId=120049&paneView=debug

wtgodbe commented 2 years ago

That's a different issue, see https://github.com/dotnet/aspnetcore/pull/37715 and https://teams.microsoft.com/l/message/19:8d76f8f67d8e4237b9f0b87235ec870e@thread.skype/1634747033503?tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47&groupId=fdff90ed-0b3b-4caa-a30a-efb4dd47665f&parentMessageId=1634747033503&teamName=DotNet&channelName=ASP.NET%20Build&createdTime=1634747033503 for context

dougbu commented 2 years ago

This remains a problem in some of our branches, especially release/5.0. We probably have fixes or partial fixes in 'main' that haven't been backported to either or both release/3.1 or release/5.0. Might even have a couple that aren't in release/6.0. For example, are the following everywhere they could be (and did they help 😃)❔

dougbu commented 2 years ago

I think this problem is behind us in 'main' only question remaining is whether we need to backport some of the relevant fixes to servicing branches. Thoughts❔

wtgodbe commented 2 years ago

I don't think I've seen this issue in servicing lately, but I'm not 100% sure. It definitely seems to be less prevalent, at least.

ayakael commented 1 year ago

I'm encountering this on linux-musl-s390x on commit c6865355c01e1fb170b65427846d939559ab3789 with sdk version 7.0.100-rtm.22519.39 when building in a source-build workflow. Is there something I can do to mitigate this issue? It isn't a problem on linux-musl-s390x with sdk version 6.0.110.

edit Note that the runtime is Mono flavored, and so far the issue hasn't occured on arm, arm64 or x64.

dougbu commented 1 year ago

@ayakael the problem is rather inconsistent and doesn't happen frequently. Building a second time is usually successful.

ayakael commented 1 year ago

@ayakael the problem is rather inconsistent and doesn't happen frequently. Building a second time is usually successful.

The issue is that is guaranteed to happen at least once during the building process of dotnet7 for Alpine on s390x. As our builders restart the building process from scratch when there's a failure, it'll often take 5-6 full rebuilds before getting through. I'm more perplexed by it not being an issue on dotnet6, but suddently it is on dotnet7, suggesting a regression.

ayakael commented 1 year ago

I've reproduced this with ppc64le, thus seems to be specific to mono flavored runtime with dotnet7. This particular instance is with runtime: link

Another failure with roslyn is shown here

Usually, error appears as so:

/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error : The process cannot access the file '/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/artifacts/obj/System.IO.Ports' because it is being used by another process. [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at System.IO.FileSystem.CreateDirectory(String fullPath, UnixFileMode unixCreateMode) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at System.IO.FileSystem.CreateDirectory(String fullPath) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at System.IO.Directory.CreateDirectory(String path) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.BuildAssetsUtils.WriteFiles(IEnumerable`1 files, ILogger log) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreResult.CommitAssetsFileAsync(LockFileFormat lockFileFormat, IRestoreResult result, ILogger log, Boolean toolCommit, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreResult.CommitAsync(ILogger log, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.CommitAsync(RestoreResultPair restoreResult, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.ExecuteAndCommitAsync(RestoreSummaryRequest summaryRequest, IRestoreProgressReporter progressReporter, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.CompleteTaskAsync(List`1 restoreTasks) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.RunAsync(IEnumerable`1 restoreRequests, RestoreArgs restoreArgs, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.RunAsync(RestoreArgs restoreContext, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Build.Tasks.BuildTasksUtility.RestoreAsync(DependencyGraphSpec dependencyGraphSpec, Boolean interactive, Boolean recursive, Boolean noCache, Boolean ignoreFailedSources, Boolean disableParallel, Boolean force, Boolean forceEvaluate, Boolean hideWarningsAndErrors, Boolean restorePC, Boolean cleanupAssetsForUnsupportedProjects, ILogger log, CancellationToken cancellationToken) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Build.Tasks.Console.MSBuildStaticGraphRestore.RestoreAsync(String entryProjectFilePath, IDictionary`2 globalProperties, IReadOnlyDictionary`2 options) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]

Thus occurs during restore. Further research has shown that there are many different ways that this bug can occur, so it likely doesn't have to do with aspnetcore. Some pointers would be appreciated, though.

dougbu commented 1 year ago

I suggest examining a binary log and trying to determine what else is accessing the file. We don't use mono-based runtimes ourselves and unfortunately can't help directly.

thebeard-lifebell commented 1 year ago

I think this is related to dotnet publish or dotnet build invoking msbuild with -maxcpucount and not providing an integer for the # cpus to use. This makes it so that msbuild auto-selects how many cpus to use, which on multi-core/multi-threaded machines is more than 1.

We were having this issue in our Github Actions build, so we added the -maxcpucount:1 argument to the dotnet publish command, which overrode the -maxcpucount switch passed by dotnet to msbuild, which made the process single-threaded, which relieved the file contention issue.

I'm not sure if this is the solution, but it might help. Good luck!

P.S: A Stackoverflow about this

wtgodbe commented 1 year ago

Closing as we haven't seen this in a while - will re-open if it comes back.

wtgodbe commented 4 months ago

We're still seeing this occasionally - @amcasey suggested trying to NoWarn MSB3026, which could be worth a try if we can convince ourselves it won't get us into trouble w/ further SDK logic. https://dev.azure.com/dnceng-public/public/_build/results?buildId=645948&view=results

amcasey commented 4 months ago

We're still seeing this occasionally - @amcasey suggested trying to NoWarn MSB3026, which could be worth a try if we can convince ourselves it won't get us into trouble w/ further SDK logic. https://dev.azure.com/dnceng-public/public/_build/results?buildId=645948&view=results

This suggestion assumes, perhaps naively, that there's a different error code when the last retry fails.