Native tools bootstrapping failing with access denied

[jonfortescue]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1671390&view=logs&jobId=987e5851-59ff-5ad3-0c32-2d9d35c2807b&j=987e5851-59ff-5ad3-0c32-2d9d35c2807b&t=05844f13-4ab9-56d6-865f-18cd26ce6550

https://dev.azure.com/dnceng/public/_build/results?buildId=1671851&view=logs&j=119f12ab-97c9-53f0-7ea6-00e6f97fda11&t=d2e57700-e144-5940-d919-9fe6f85bd854&s=6884a131-87da-5381-61f3-d7acc3b91d76

Build leg reported

Plain_Build_Windows, EndToEndBuildTests

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• [x] Open the failing build above and investigate
• [x] Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"

2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

   {
   "errorMessage" : "CommonLibrary.psm1:382 char:5"
   }

Additional information about the issue reported

Error is occurring in the 1ES hosted pools. Possibly antivirus related?

Processing D:\a\_work\1\s\global.json
Move-Item : Access to the path 'C:\Users\cloudtest\.netcoreeng\native\bin\perl\5.32.1.1.tmp' is denied.
At D:\a\_work\1\s\eng\common\native\CommonLibrary.psm1:382 char:5
+     Move-Item -Path $TempOutputDirectory -Destination $OutputDirector ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (C:\Users\cloudt...rl\5.32.1.1.tmp:DirectoryInfo) [Move-Item], IOException
    + FullyQualifiedErrorId : MoveDirectoryItemIOError,Microsoft.PowerShell.Commands.MoveItemCommand

at <ScriptBlock>, D:\a\_work\1\s\eng\common\native\install-tool.ps1: line 103
at <ScriptBlock>, D:\a\_work\1\s\eng\common\init-tools-native.ps1: line 103
at <ScriptBlock>, D:\a\_work\1\s\eng\common\init-tools-native.ps1: line 80

cc/ @KevinRansom

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[michellemcdaniel]

This is weird. It looks like... it doesn't always happen? Passes on retry

[michellemcdaniel]

@ulisesh @AlitzelMendez What's the best way to fill out the known issue error message in a case like this? I don't know if it's just this perl install, or native tools bootstrapping in general, and I don't know if we can regex the error message so things can bucket. But I do feel like this is a known issues thing, just in case?

[ulisesh]

Unfortunately, we don't support regrex in the error message right now. We have discussed it but we have some perf concerns. I'll make sure there is an issue for it.

For this issue we could use something in the call stack to identify future breaks. Something like "ErrorMessage" : "CommonLibrary.psm1:382 char:5" could identify all the times where the script crashed in the same place

[michellemcdaniel]

Sounds good

[jonfortescue]

Seems "native tools bootstrapping" might be too narrow a scope for where this is happening: something very similar occurred here.

[MattGal]

Kicking out of FR (please see me if I am missing something)... please handle in your existing artifact-management epic.

[jonfortescue]

This is not part of my epic as it doesn't have to do with artifacts. I'm pretty convinced this is FR as it's a build failure that's actively affecting customers. I don't believe the cause is native tools bootstrapping as it's an access denied error.

Kicking out of my epic for now and will let triage decide.

[Chrisboh]

@MattGal / @ilyas1974 shouldn't this be an FR issue as it is happening to a customer build?

[MattGal]

@MattGal / @ilyas1974 shouldn't this be an FR issue as it is happening to a customer build?

I would clarify that it's not happening, rather, it happened; the idea was that since the thing that happened was something Jon had to address as part of his epic (downloading perl as part of the build is going to be forbidden) this would just be tracked there. It's not FR because it's not consistently happening.

I'd be fine just closing the issue, but ideally after a change that puts Perl onto the machines used for build proactively is rolled out to production first, since that's when FSharp can get better.

[Chrisboh]

Gotcha. I am fine either way you want to track it but it looks like we have a plan on how this should go away in the future.

[MattGal]

I confirmed this happened only once int he last 10 failures in main (20220318.5) as Jon originally opened the issue. It's most definitely AzSecPack keeping file handles open to things and most definitely would be fixed by perl already being on the disk before the build started.

[jonfortescue]

Perl will be on a set of new images starting with next week's rollout, so fsharp can use those images with perl pre-installed to obviate the need for native tool bootstrapping. Closing.

[ilyas1974]

It appears that this is still occurring. @MattGal can you please follow up?

https://dev.azure.com/dnceng/public/_build/results?buildId=1778417&view=logs&j=119f12ab-97c9-53f0-7ea6-00e6f97fda11&t=d2e57700-e144-5940-d919-9fe6f85bd854

[MattGal]

The images in question just don't have the c:\arcade-tools folder yet. This is still under the work @jonfortescue is doing in https://github.com/dotnet/arcade/issues/8813 so I moved it over there. The disconnect seems to be we think this image has these artifacts, which it does not.

[jonfortescue]

With fsharp taking the latest arcade update, this should be resolved.