Cannot update workload manifests from authenticated AzDO feeds on macOS

[DamianEdwards]

I'm not able to get any dotnet workload related commands to work when one of the workload manifests has updates on an authenticated AzDO artifacts feed on macOS. I've installed the nuget credential provider and dotnet restore from that feed works just fine.

I've tried relying on on-demand token acquisition (via device flow and passing --interactive), configuring a PAT in the NuGet.config file, passing the source directly, etc. Every time, sudo dotnet workload update fails with the following error message:

Failed to update the advertising manifest microsoft.net.sdk.android: Unable to load the service index for source https://pkgs.dev.azure.com/<feed-name>/nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.aspire: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.ios: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.maccatalyst: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.macos: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.maui: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.tvos: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.current: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.emscripten.current: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.emscripten.net6: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.emscripten.net7: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.net6: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.net7: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json.

Output of dotnet --info:

.NET SDK:
 Version:   8.0.100-rc.2.23472.8
 Commit:    e13fd5c19e

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  13.4
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /usr/local/share/dotnet/sdk/8.0.100-rc.2.23472.8/

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.0-rc.2.23466.4
  Architecture: arm64
  Commit:       287c10d253

.NET SDKs installed:
  6.0.402 [/usr/local/share/dotnet/sdk]
  8.0.100-rc.2.23472.8 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.10 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.7 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.0-rtm.23468.24 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.10 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.0-rc.2.23466.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  x64   [/usr/local/share/dotnet/x64]
    registered at [/etc/dotnet/install_location_x64]

Environment variables:
  Not set

global.json file:
  /Users/damian/src/MyProject/global.json

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

[marcpopMSFT]

Chet believes the code in question might be here but it is correctly passing the interactive flag through: https://github.com/dotnet/sdk/blob/6483a69e9a26df4f0fd598a523f6f09fab9e1c10/src/Cli/dotnet/commands/dotnet-workload/WorkloadCommandNuGetRestoreActionConfigOptions.cs#L55-L58

[DamianEdwards]

I didn't have this issue on Ubuntu FWIW, and others have hit this on macOS too, but I appreciate it might be hard to repro without a workload manifest update ready to go on a private feed.

[rmarinho]

@DamianEdwards did you tried --configfile pointing to the global Nuget.config with the PAT ?

[dsplaisted]

Thanks a bunch to @rmarinho and @mattleibow for helping investigate this.

The problem is caused by running under su/sudo. When running a workload command under sudo, we override the HOME directory to point to a temporary directory. This is so that we don't end up writing files owned by root (for example first run flags or other things stored in the <HOME>\.dotnet folder) which would then not be accessible to the .NET CLI when running under normal permissions.

This means that when run under sudo, the credential provider isn't found by NuGet (since the path to search for it is in the HOME directory). This is also probably why putting the PATs in the NuGet.config file didn't work. (We do try to copy the NuGet.config to the new temporary home directory, but maybe this is failing).

We will discuss to see what we can do to improve this. For now, putting PATs in a NuGet.config file and specifying that file via the --configfile option to workload commands should work as a workaround.

[DamianEdwards]

That will explain why it always runs the first-run experience too I guess.

I'm still unable to get it to work with the --configfile option using the nuget.config in the repo, modified to contain a PAT in plaintext. It does report it's trying to access the correct feed though, so it must be reading at least that from the nuget.config, but not the PAT for some reason.

[DamianEdwards]

That said, I'd expect an authorization failure to be logged exactly as that, but what I'm seeing is "NotFound" results. I'll share the repo details and logs internally to see if that can aid in diagnosing what's going on.

[dsplaisted]

The output is pretty verbose, and it's checking for packages on all the different feeds that are set up. So you get a lot of NotFound log messages before you get the message related to the authorization failure, which looks something like:

Failed to update the advertising manifest microsoft.net.sdk.android: Unable to load the service index for source https://pkgs.dev.azure.com/<feed-name>/nuget/v3/index.json..

We might be able to detect the authorization error specifically and generate a more specific error message.

It might also be a good idea to generate a warning if --interactive is used under sudo, since we now know that doesn't work.

[DamianEdwards]

Any idea why I'm unable to make it work with a PAT and the --configfile option?

[dsplaisted]

Any idea why I'm unable to make it work with a PAT and the --configfile option?

Not really. Are you sure the syntax in the config file for the PAT is correct?

[DamianEdwards]

Any idea why I'm unable to make it work with a PAT and the --configfile option?

Not really. Are you sure the syntax in the config file for the PAT is correct?

Not sure. I'm just working from documentation. The fact it's trying the right feed suggests it is picking up the nuget.config, and there are no parsing errors. I'm not sure what else to try.

[Eilon]

BTW I think I'm hitting the same issue on Windows (@DamianEdwards had it on macOS). I have Azure Cred Provider, but it doesn't seem to get used when I restore workloads. If I pass in --interactive then I get the devicelogin workflow and everything works.

[marcpopMSFT]

BTW I think I'm hitting the same issue on Windows (@DamianEdwards had it on macOS). I have Azure Cred Provider, but it doesn't seem to get used when I restore workloads. If I pass in --interactive then I get the devicelogin workflow and everything works.

We are discussing making --interactive default in some situations but the behavior you report is expected. Having the cred provider installed today is not sufficient to trigger the device login details as interactive is needed as well.

[baronfel]

@DamianEdwards per this doc the nuget plugin discovery is extensible via environment variables. Can you try:

• setting the NUGET_PLUGIN_PATHS variable in your terminal profiles to somewhere that both sudo and non-sudo users have access to, and
• copying the contents of ~/.nuget/plugins to that path, and
• trying the workload install/update again?

[baronfel]

Another thing to try - the azure feed provider puts session tokens in a directory that's configurable via env var, and NuGet will also look in this location. Consider setting the location per these docs to somewhere outside of $HOME and trying that as well.

[baronfel]

cc @kartheekp-ms for visibility per Jon's suggestion

[kartheekp-ms]

As pointed out in https://github.com/dotnet/sdk/issues/35912#issuecomment-1775855165, the following environment variable might be helpful to work around this issue:

1. NUGET_NETCORE_PLUGIN_PATHS - Defines the plugins that will be used by the .NET Core-based tooling (dotnet.exe).

I guess we don't have to set NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED to false because the credential provider (plugin) anyway writes to the $HOME/.local/share/MicrosoftCredentialProvider/ location on Mac and Linux. From what I understood in this thread, the $HOME environment variable will be pointing to a temporary location that is created while running the sudo dotnet workload install command.

In case it helps, this document contains environment variables that can be set to manage NuGet global packages and cache folders.

[gtbuchanan]

~Would this issue prevent the latest version of workloads from being installed?~

I started encountering a MAUI iOS build issue that only seems to occur on Azure Pipelines. After investigating, I found that we were encountering this authentication error at the dotnet workload install maui step and the workload being installed on the agent was only v8.0.3 (whereas the latest on my Mac is v8.0.61).

We don't actually load workloads from authenticated sources, so I'll try passing a NuGet.Config without our internal source to --configfile and see what happens.

EDIT:

Yep, that was the problem and the solution worked. Here's the PowerShell step I used in my pipeline:

$NuGetConfig = Join-Path $env:AGENT_TEMPDIRECTORY "NuGet.Config"
Write-Host "Writing temporary NuGet.Config to $NuGetConfig..."
New-Item $NuGetConfig
Set-Content $NuGetConfig '<configuration><packageSources><add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" /></packageSources></configuration>'
dotnet workload install maui --configfile $NuGetConfig
dotnet workload list

After applying the above workaround, the newest version of the workload was successfully installed.

[guyulmaz]

Hi, I have same problem, cannot install workload to my macbook for sudo needed ones. Getting error=

Failed to update the advertising manifest microsoft.net.sdk.android: Unable to load the service index for source https://api.nuget.org/v3/index.json..

I tried adding a nuget.config myself and use --configfile, but it did not work how can I install android , wasm workloads to my mac ? I'm trying to setup avalonia cross platform vscode template at my macbook, that will cover all desktop, web and mobile Can please help

% dotnet --info .NET SDK: Version: 8.0.400 Commit: 36fe6dda56 Workload version: 8.0.400-manifests.56cd0383 MSBuild version: 17.11.3+0c8610977

Runtime Environment: OS Name: Mac OS X OS Version: 14.6 OS Platform: Darwin RID: osx-arm64 Base Path: /usr/local/share/dotnet/sdk/8.0.400/

.NET workloads installed: Configured to use loose manifests when installing new manifests. [ios] Installation Source: SDK 8.0.400 Manifest Version: 17.5.8020/8.0.100 Manifest Path: /usr/local/share/dotnet/sdk-manifests/8.0.100/microsoft.net.sdk.ios/17.5.8020/WorkloadManifest.json Install Type: FileBased

Host: Version: 8.0.8 Architecture: arm64 Commit: 08338fcaa5

.NET SDKs installed: 8.0.300 [/usr/local/share/dotnet/sdk] 8.0.303 [/usr/local/share/dotnet/sdk] 8.0.400 [/usr/local/share/dotnet/sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 8.0.5 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 8.0.7 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 8.0.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 8.0.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found: None

Environment variables: DOTNET_ROOT [/usr/local/share/dotnet]

global.json file: Not found

Learn more: https://aka.ms/dotnet/info

Download .NET: https://aka.ms/dotnet/download

[petrkoutnycz]

Tried the hack with setting NUGET_NETCORE_PLUGIN_PATHS env variable but with no avail :-/