Extremely Long Task Times

Axemasta commented 3 years ago

I've noticed recently the build times on my pipelines becoming extraordinarily long, like 40-50 minutes.

It looks like the majority of the time is spent downloading sdk's, after a while the tasks appear to hang and never report back until i cancel them. The following tasks did this:

installer: The software was successfully installed......
installer: The upgrade was successful.
Deleting /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/3an1ew5g.kba.pkg
* Automatic URL resolving is a new feature. File issues at: https://github.com/jonathanpeppers/boots/issues
Querying https://software.xamarin.com/Service/Updates?v=2&pv964ebddd-1ffe-47e7-8128-5ce17ffffb05=0&pv4569c276-1397-4adb-9485-82a7696df22e=0&pvd1ec039f-f3db-468b-a508-896d7c382999=0&pv0ab364ff-c0e9-43a8-8747-3afb02dc7731=0&level=Stable
Downloading https://download.visualstudio.microsoft.com/download/pr/573004d1-39a9-4cf7-87b0-e0eea351cd00/4ab0eae8f20e3d59c08393aa77c8c123/xamarin.ios-14.8.0.3.pkg
Writing to /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/3hrjprt1.cn2.pkg
##[error]The operation was canceled.

Before i cancelled the operation it has been on the downloading stage for about 30 minutes. Is there any extra logging i can enable on my pipeline to help gather more information about the cause of these long wait times?

My YAML tasks looks like this:

- script: |
    dotnet tool install --global boots --ignore-failed-sources
    boots --stable Mono
    boots --stable XamarinAndroid
    boots --stable XamariniOS
  displayName: Ensure Latest Xamarin & Mono SDKs

Axemasta commented 3 years ago

After cancelling the pipeline and rerunning the task, it completed in 4 minutes which is about how long im used to it taking

jonathanpeppers commented 3 years ago

We recently were talking with the VS Engineering team (they work on CI for anything Visual Studio), and one of the suggestions is to make any network requests have good timeout/retry logic. You can hit random failures like this on Azure DevOps otherwise...

If you look at the downloading code right now, it is basic and simple:

https://github.com/jonathanpeppers/boots/blob/ee1ec0158192a2f814d2b6da85aa957d98349c2e/Boots.Core/Downloader.cs#L26-L38

This needs to timeout/retry in a reasonable fashion, and have command-line options to change the settings.

It is odd, I haven't hit something like this myself -- I have quite a few projects that use boots...

jonathanpeppers / boots

Extremely Long Task Times #61