PR from arcade to arcade-validation on the master branch is failing during dry run of symbol publishing

[riarenas]

• [X] This issue is blocking Arcade flow to other repositories is blocked
• [ ] This issue is causing unreasonable pain

PR https://github.com/dotnet/arcade-validation/pull/2140 is stuck with a failing build. The build is currently failing with:

Publishing symbol packages to MSDL ...
  Performing symbol publish (Dry Run)...
  SymbolServerPath: https://microsoftpublicsymbols.artifacts.visualstudio.com/DefaultCollection
  ExpirationInDays specified: 3650
  Symbol publish finished
  Publishing symbol packages to SymWeb ...
  Performing symbol publish (Dry Run)...
  SymbolServerPath: https://microsoft.artifacts.visualstudio.com/DefaultCollection
  ExpirationInDays specified: 3650
D:\workspace\_work\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21123.3\tools\Publish.proj(253,5): error : The process cannot access the file 'D:\workspace\_work\1\s\artifacts\SymStore\Debug\Validation\net5.0\Validation.pdb' because it is being used by another process.
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21123.3\tools\Publish.proj(253,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) The process cannot access the file 'D:\workspace\_work\1\s\artifacts\SymStore\Debug\Validation\net5.0\Validation.pdb' because it is being used by another process.
D:\workspace\_work\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21123.3\tools\Publish.proj(253,5): error : The process cannot access the file 'D:\workspace\_work\1\s\artifacts\SymStore\Debug\Validation\net5.0\Validation.pdb' because it is being used by another process.
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21123.3\tools\Publish.proj(253,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) The process cannot access the file 'D:\workspace\_work\1\s\artifacts\SymStore\Debug\Validation\net5.0\Validation.pdb' because it is being used by another process.
  Symbol publish finished

Example build: https://dev.azure.com/dnceng/public/_build/results?buildId=1007838&view=results

This has been failing as of build 20210219.1 of arcade, which corresponds to this commit that updates the symbol uploader: https://github.com/dotnet/arcade/commit/39e79a66b276f5633a74feed2efbb9b65367228d

[riarenas]

CC @epananth This error looks familiar to you? I get the impression we had this happen not too long ago but haven't been able to find the issue yet.

CC @dotnet/dnceng adding this to FR as critical, as this is blocking arcade flow.

[MattGal]

CC @epananth This error looks familiar to you? I get the impression we had this happen not too long ago but haven't been able to find the issue yet.

CC @dotnet/dnceng adding this to FR as critical, as this is blocking arcade flow.

Taking a look

[MattGal]

Taking myself for now, can discuss different owner @ standup. It's clear that dotnet.exe is holding on a file handle in the first task invocation, and keeping it regardless of how long of a delay is introduced between the legs. I've pinged @mikem8361 and @epananth for any suggestions as well.

[MattGal]

Created a revert PR, but am also taking a stab since I was able to generate a local repro at figuring out what's going on.

[MattGal]

After hacking up a fix in symuploader, it turns out there's already a branch with a fix so we can probably just use that :)

[MattGal]

The update got backed out, I tested the changes in the PR in my local repro, and now we're just waiting for https://dnceng.visualstudio.com/internal/_git/dotnet-symuploader/pullrequest/13154 to merge.

[MattGal]

This should now be unblocked; the arcade version has been rolled back, and the next update will contain the fix in the above PR. Will leave the issue in rollout a few days in case any fallout occurs.

[MattGal]

No problems observed, closing.