yonzhan
commented
1 year ago Thank you for opening this issue, we will look into it.
Open jessehouwing opened 1 year ago
yonzhan
commented
1 year ago Thank you for opening this issue, we will look into it.
jessehouwing
commented
1 year ago Updated with logs. From what I gather looking at the debug logs, the hosted runner doesn't have a Command index in its %USERPROFILE%\.azure folder, this is correct, since all calls of az are done using the provisioning user and a fresh profile is created for the hosted runner.
The Visual Studio warmup moves a bunch of files from the provisioning user profile to the default user profile.
My proposed change in the warmup scripts does exactly that after running az at least once.
It also looks like extensions aren't fully compiled until they're called at least once. That'd why I'm adding az pipelines --help | out-null, looking at the procmon logs, upon calling the extension some python compilation takes place. The hosted agent already redirects the extension directory to the program files directory, so those files should survive the user profile switch.
Is there anything else we can do to speed up first-call performance?
jessehouwing
commented
1 year ago Confirming a massive speedup after carrying over the following folders:
C:\Program Files\Common Files\AzureCliExtensionDirectory
C:\Users\VssAdministrator\.azurehttps://github.com/actions/runner-images/pull/8294#issuecomment-1721702363
Still seeing a perf hit on first call of az pipelines as it caches instance information.
ilia-shipitsin
commented
1 year ago Hello, runner images team here.
the issue with "C:\Users\VssAdministrator.azure" is that it resides in ephemeral user profile. user is deleted after image generation.
is it possible to store that folder somewhere ? not in user profile
jessehouwing
commented
1 year ago Hello, runner images team here.
the issue with "C:\Users\VssAdministrator.azure" is that it resides in ephemeral user profile. user is deleted after image generation.
is it possible to store that folder somewhere ? not in user profile
In my PR the folder is copied to the default profile and will thus be copied to the new user that actually runs the cli. That also works. Getting all of the data out of the profile would be even better.
ilia-shipitsin
commented
1 year ago Hello, runner images team here. the issue with "C:\Users\VssAdministrator.azure" is that it resides in ephemeral user profile. user is deleted after image generation. is it possible to store that folder somewhere ? not in user profile
In my PR the folder is copied to the default profile and will thus be copied to the new user that actually runs the cli. That also works. Getting all of the data out of the profile would be even better.
that would be poor man approach indeed, copying files that (we know) are to be deleted.
if it is supported I would prefer to store files in some permanent folder
another question is "what are folders to be kept". they were discovered, may they change in future ?
jessehouwing
commented
1 year ago Hello, runner images team here. the issue with "C:\Users\VssAdministrator.azure" is that it resides in ephemeral user profile. user is deleted after image generation. is it possible to store that folder somewhere ? not in user profile
In my PR the folder is copied to the default profile and will thus be copied to the new user that actually runs the cli. That also works. Getting all of the data out of the profile would be even better.
that would be poor man approach indeed, copying files that (we know) are to be deleted.
if it is supported I would prefer to store files in some permanent folder
another question is "what are folders to be kept". they were discovered, may they change in future ?
Looks like the .azure is governed by another environment variable: $AZURE_CONFIG_DIR
I changed my PR to use that instead. Where would you put it, commonfiles\azurecliconfigdirectory?
https://learn.microsoft.com/en-us/cli/azure/azure-cli-configuration#cli-configuration-file
ilia-shipitsin
commented
1 year ago that looks good. give me some time to test it
jessehouwing
commented
1 year ago Found 2 more environment variables while searching the sources for the azure-devops extension. Added those and cleaned up the pull request.
microsoft-github-policy-service[bot]
commented
1 year ago Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @v-anvashist, @V-hmusukula.
jessehouwing
commented
1 year ago With the world moving to more and more ephemeral runners (if only for the added security benefits), first run performance of tools will become more and more important. The following things would be things to consider:
Have the ability to install az in ephemeral mode or for az to detect it's running in a CICD environment which would:
az though az extension addaz login is still attempted "just in case".
az runs in a CI/CD environment.
go, rust and kotlin.Or offer a single command we can run to ensure all of the above after installation.
Describe the bug
When calling
azand especiallyaz pipelineson the hosted runner the first call takes a performance hit varying between 60 and 90 seconds on windows and 10-15 seconds on linux.That time is spent rebuilding the command index and interpreting/compiling python code for AZ and te extensions.
I've been looking into improving performance by 'warming up' the agent during image generation.
We've measured improvements in the Windows Hosted Runner by:
~/.azureto a system shared location~/.azure-devopsto a system shared location~/.azure-devops/python/cacheto a system shared locationaz --helpduring image generation to force a command index rebuildaz --helpandaz devops --helpto force python interpretation/compilationaz devops loginto force installation of keyringCombined this results in a 1 minute improvement of the first-run on Windows. It results in a ~12s improvement on Ubuntu.
Not every tool vendor should have to figure out how to do this, plus the behavior might change from
az's end at any time. Theactions/actions-runnerteam isn't keen on taking on having to configureazas part of image generation.With the world moving to more and more ephemeral runners (if only for the added security benefits), first run performance of tools will become more and more important. The following things would be things to consider.
Have the ability to install
azin ephemeral mode or forazto detect it's running in a CICD environment which would:azthoughaz extension addaz loginis still attempted "just in case".azruns in a CI/CD environment.go,rustandkotlin.Or offer a single command we can run to ensure all of the above after installation.
Related command
az pipelines runs show --idErrors
First execution takes way too long.
Issue script & Debug output
Script being called in azure-pipelines windows hosted runner:
Debug logs
Expected behavior
Faster first run time in hosted agent by calling az in a way that preloads most extensions.
Environment Summary
azure-cli 2.52.0
core 2.52.0 telemetry 1.1.0
Extensions: azure-devops 0.26.0
Dependencies: msal 1.24.0b1 azure-mgmt-resource 23.1.0b2
Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe' Extensions directory 'C:\Program Files\Common Files\AzureCliExtensionDirectory'
Python (Windows) 3.10.10 (tags/v3.10.10:aad5f6a, Feb 7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)]
Additional context
No response