search

dotnet / sdk

Core functionality needed to create .NET Core projects, that is shared between Visual Studio and CLI
https://dot.net/core
MIT License
2.69k stars 1.06k forks source link

Silent workload updates break builds #40222

Open svaldetero opened 6 months ago

svaldetero commented 6 months ago

Description

Another update Tuesday and another broken Azure Pipelines CI build. One build works and then an hour late the build breaks.

I have a step to use a dotnet version 8.0.x. It downloads 8.0.203. Specifically: https://download.visualstudio.microsoft.com/download/pr/87f31249-1415-4edb-87d5-7f16e63414e9/a1ad58c05a131c618ad458a1564784e4/dotnet-sdk-8.0.203-osx-x64.tar.gz.

Next step is to: dotnet workload install maui-ios maui-android --source https://api.nuget.org/v3/index.json.

The first time it ran, it installed successfully:

Installing workload manifest microsoft.net.sdk.android version 34.0.85... Installing workload manifest microsoft.net.sdk.ios version 17.2.8043... Installing workload manifest microsoft.net.sdk.maccatalyst version 17.2.8043... Installing workload manifest microsoft.net.sdk.macos version 14.2.8043... Installing workload manifest microsoft.net.sdk.maui version 8.0.7... Installing workload manifest microsoft.net.sdk.tvos version 17.2.8043... Installing workload manifest microsoft.net.sdk.aspire version 8.0.0-preview.5.24201.12... Installing pack Microsoft.AspNetCore.Components.WebView.Maui version 8.0.7... Writing workload pack installation record for Microsoft.AspNetCore.Components.WebView.Maui version 8.0.7... Installing pack Microsoft.Maui.Sdk version 8.0.7...

The second time it ran (an hour later), it failed:

Installing workload manifest microsoft.net.sdk.android version 34.0.95... Installing workload manifest microsoft.net.sdk.ios version 17.2.8043... Installing workload manifest microsoft.net.sdk.maccatalyst version 17.2.8043... Installing workload manifest microsoft.net.sdk.macos version 14.2.8043... Installing workload manifest microsoft.net.sdk.maui version 8.0.7... Installing workload manifest microsoft.net.sdk.tvos version 17.2.8043... Installing workload manifest microsoft.net.workload.mono.toolchain.current version 8.0.4... Installing workload manifest microsoft.net.sdk.aspire version 8.0.0-preview.5.24201.12... Workload installation failed. Rolling back installed packs... Installation rollback failed: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/Users/runner/hostedtoolcache/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json] Workload installation failed: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/Users/runner/hostedtoolcache/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]

Looking at the nugets: https://www.nuget.org/packages/Microsoft.NET.Workload.Mono.ToolChain.Current.Manifest-8.0.100/8.0.4 was published between the builds.

HOWEVER, the nuget package that it claims is required: https://www.nuget.org/packages/Microsoft.NET.Workload.Emscripten.Current.Manifest-8.0.100/8.0.4 DOES NOT EXIST.

I understand these may be separate teams, however these nugets should have been deployed at the same time.

Steps to Reproduce

No response

Link to public reproduction project repository

No response

Version with bug

8.0.7 SR2

Is this a regression from previous behavior?

No, this is something new

Last version that worked well

8.0.7 SR2

Affected platforms

iOS, Android

Affected platform versions

No response

Did you find any workaround?

I changed my workload restore to include : --from-rollback-file https://maui.blob.core.windows.net/metadata/rollbacks/8.0.7.json

Relevant log output

No response

svaldetero commented 6 months ago

Update: After submitting this, the dependent nuget package showed up (~3-4 hours later).

The issue still remains, we need a way to pin a version of the workload (without using the undocumented and hard to find rollback file). Because pinning a dotnet version doesn't stop the problem.

svaldetero commented 6 months ago

Same type of issue, but different package: https://github.com/dotnet/maui/issues/19934

Redth commented 6 months ago

@svaldetero please try again. It seems there was a delay in publishing of some packages for this update, which should now be available.

Redth commented 6 months ago

Related: https://github.com/dotnet/sdk/issues/23820

svaldetero commented 6 months ago

@Redth Yes it works now after I removed the rollback option.

As a developer, how do I prevent this in the future? How can I pin a workload version without having to find a hidden rollback file? In that issue you linked, it doesn't happen every time, but it happens frequently enough. Today it just happened to be ~3 hours before all the nugets were available. That suddenly blocks and backlogs ALL of my builds for close to half the day. Realistically, I can't just wait for the delay to catch up because I don't know how long it will be.

kgouraw commented 6 months ago

I experienced the same with "dotnet workload install maui" in my GitHub workflow today. I changed it to "dotnet workload install maui-mobile" and it works for me for now.

Ghostbird commented 6 months ago

For us, the way we managed to "pin" the MAUI version for Android CI builds is by building in docker, which we need to do anyway for CI. For an example see our repository. Note: We do not use the docker for builds on the developer's own machine, but updating MAUI there takes only a minute when it breaks. Alas, our iOS builds keep automatically breaking, but that happens from Apple's side too. Their updates often break things too, so in that sense MAUI is not doing any worse.

MatthewRamsey commented 6 months ago

Problem: We were experiencing similar issues this week which were the result of an issue with the Android sdk version bump. Specifically, Microsoft.Android.Sdk.Windows.Msi.x64 (34.0.95) which caused malformed content in in our Android http responses.

Solution: As others have mentioned, we found a workaround by pinning a rollback file for the specific version of the workloads we need. The rollback file below reverted the Android dependencies to 34.0.79 which solved our problem.

dotnet workload install maui --source https://api.nuget.org/v3/index.json --from-rollback-file https://maui.blob.core.windows.net/metadata/rollbacks/8.0.7.json

dotnet workload install android ios --source https://api.nuget.org/v3/index.json --from-rollback-file https://maui.blob.core.windows.net/metadata/rollbacks/8.0.7.json

Suggestion: It would be great to have a page with a list of available rollback files. We're content to pin a version while new releases come out so we can control stability of our applications.

marcpopMSFT commented 5 months ago

Without functionality in nuget to batch these, the best we have is pushing all of the packs first and then the manifest. Looking forward, we'll have workload versions coming soon that will make this easier as we can ensure all of the packs and manifests are published and then publish a single workload set version package.