Mac agent unable to restore nuget packages.

[uragonda-rajesh]

Description

We recently see the image version is update to '20220614.2' and from then we see the nuget package failures.

" Restoring packages for /Users/runner/work/1/s/VI.iOS/VI.iOS.csproj...
/Library/Frameworks/Mono.framework/Versions/6.12.0/lib/mono/msbuild/Current/bin/NuGet.targets(131,5): error : Unable to resolve 'Xam.Plugin.LatestVersion (>= 2.1.0)' for 'Xamarin.iOS,Version=v1.0'. [/Users/runner/work/1/s/VI.iOS/VI.iOS.csproj]
/Library/Frameworks/Mono.framework/Versions/6.12.0/lib/mono/msbuild/Current/bin/NuGet.targets(131,5): error : Unable to resolve 'Xamanimation (>= 1.3.0)' for 'Xamarin.iOS,Version=v1.0'. [/Users/runner/work/1/s/VI.iOS/VI.iOS.csproj]
...
...
...
  Committing restore...
  Generating MSBuild file /Users/runner/work/1/s/VI.iOS/obj/VI.iOS.csproj.nuget.g.props.
  Generating MSBuild file /Users/runner/work/1/s/VI.iOS/obj/VI.iOS.csproj.nuget.g.targets.
  Writing assets file to disk. Path: /Users/runner/work/1/s/VI.iOS/obj/project.assets.json
  Failed to restore /Users/runner/work/1/s/VI.iOS/VI.iOS.csproj (in 432 ms).
 NuGet Config files used:
      /Users/runner/.config/NuGet/NuGet.Config
Done Building Project "/Users/runner/work/1/s/VI.iOS/VI.iOS.csproj" (Clean;Restore target(s)) -- FAILED.

Build FAILED.
...
...
##[error]Error: The process '/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild' failed with exit code 1
Finishing: MSBuild

Platforms affected

• [X] Azure DevOps
• [ ] GitHub Actions

Virtual environments affected

• [ ] Ubuntu 18.04
• [ ] Ubuntu 20.04
• [ ] Ubuntu 22.04
• [ ] macOS 10.15
• [X] macOS 11
• [ ] macOS 12
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Image version and build link

Environment: macos-11 Version: 20220614.2

Starting: Initialize job
Agent name: 'Azure Pipelines 2'
Agent machine name: 'Mac-1655484182240'
Current agent version: '2.204.0'
Operating System
Virtual Environment
Virtual Environment Provisioner
Current image version: '20220614.2'
Agent running as: 'runner'
Prepare build directory.
Set build variables.
Download all required tasks.
Downloading task: NuGetToolInstaller (0.202.0)
Downloading task: replacetokens (3.10.0)
Downloading task: MSBuild (1.199.0)
Downloading task: InstallAppleProvisioningProfile (1.200.0)
Downloading task: InstallAppleCertificate (2.200.0)
Downloading task: ios-bundle-version (1.0.0)
Downloading task: XamariniOS (2.200.1)
Downloading task: CopyFiles (2.200.0)
Downloading task: PublishBuildArtifacts (1.200.0)
Checking job knob settings.
   Knob: AgentToolsDirectory = /Users/runner/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY} 
Finished checking job knob settings.
Start tracking orphan processes.

Is it regression?

Yes. Build was successful in image version 20220605.1

Expected behavior

Build should be successful with any issues.

Actual behavior

Failed to restore nuget packages.

Repro steps

I just ran the build, no specific steps followed.

[DarryStonem]

Same is happening to my team. We cannot generate any build with the latest from macos-latest, or any other version.

The last image version (20220605.1) was working successfully

[keiji]

Same issue is happening to our project.

And my workaround below that add a step...

    - name: Setup NuGet.Config
      run: echo '<configuration><packageSources><add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" /></packageSources></configuration>' > ~/.config/NuGet/NuGet.Config

[uragonda-rajesh]

@keiji could you please let us know where we need to add this.

   - name: Setup NuGet.Config
      run: echo '<configuration><packageSources><add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" /></packageSources></configuration>' > ~/.config/NuGet/NuGet.Config

[keiji]

@uragonda-rajesh In my case, macos-latest image(Version: 20220614.2) has empty NuGet.Config like that.

Run cat ~/.config/NuGet/NuGet.Config
<?xml version="1.0"?>
<configuration>
  <packageSources>
  </packageSources>
</configuration>

We can add Nuget repository url to NuGet.Config. https://docs.microsoft.com/en-us/nuget/reference/nuget-config-file

https://api.nuget.org/v3/index.json is NuGet repository(nuget.org) url that our project need.

[uragonda-rajesh]

@keiji In the above task we are trying to restore the nuget packages, could you please let us know if we need to add the Nuget.Config file in the project or add from pipeline yml file?

If it is from the pipeline yml could you please elaborate, also can you provide the modified Nuget.Config file?

[keiji]

@uragonda-rajesh I don't know what NuGet repositories your project depends on. I recommend you check which NuGet repositories are using for your project.

NuGet.Config structure is not complicated. You can create NuGet.Config easily 😃

And my workaround is just temporary. I will remove this workaround step if macos-latest image is updated.

[mikhailkoliada]

@uragonda-rajesh Hi!

Thanks! We will take a look.

[rubverma]

Same isssue is happening to my team. We cannot generate any build using macos-latest, or any other version.

We are building using Github Actions. dotnet-version: 3.1.401 ProductName: macOS ProductVersion:11.6.6 BuildVersion: 20G624

[vtamanam]

The same is happening with our projects too. When can we expect a fix for this? We are stuck because of this issue.

[miketimofeev]

@rubverma @vtamanam and all the others who are experiencing the issue. Could you try running the following snippet before restoring packages and report back if it helps?

rm -rf ~/.config/NuGet/NuGet.Config
dotnet nuget list source

[keiji]

@miketimofeev

Welcome to .NET 5.0!
---------------------
SDK Version: 5.0.408
Telemetry
---------
The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.
Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry
----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
Registered Sources:
  1.  MyNuget [Enabled]
      /Users/runner/work/cocoa/cocoa/TempNugetFeed
  2.  nuget.org [Enabled]
     [ https://api.nuget.org/v3/inde](https://api.nuget.org/v3/index.json)

And I have removed ~/.config/NuGet/NuGet.Config. The workflow works well.

[rubverma]

@miketimofeev thank you for the solution. This is working.

[IliyanPopov]

This workaround works: Quote: "I ran into this on a project I'm on, and while I don't know exactly what's causing it, I do know a solution (which might help support figure out what's going on). It's acting like it doesn't have any nuget sources, so placing a nuget.config file in the same directory as your solution, with this as the contents:"

<?xml version="1.0" encoding="utf-8"?>

Original post: https://github.com/microsoft/azure-pipelines-tasks/issues/16465#issuecomment-1159289085

[anish1984]

@miketimofeev Thank for solution. Now we We are able to generate build using macos-latest

[vtamanam]

Thanks, it worked.

[miketimofeev]

We will deploy new images with the fix in the next 2-3 business days.

[erdembayar]

@miketimofeev If understood correctly you're using pwsh which is Powershell Core right? PowerShell Core should be using updated NuGet.Configuration.dll, and should not be creating an empty nuget.config. Fyi @zivkan.

[miketimofeev]

@erdembayar correct, but the empty config file is being created after any powershell module installation, for example, Az.

[erdembayar]

Az

Is it the one https://docs.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-8.0.0?

What happens if you try previous version like 7.0? Does it non-empty nuget.config file?

[al-cheb]

Az

Is it the one https://docs.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-8.0.0?

What happens if you try previous version like 7.0? Does it non-empty nuget.config file?

@erdembayar, @zivkan: The nuget.config file is created by PackageManagement module.

PS /home/user> cat .config/NuGet/nuget.config
/usr/bin/cat: .config/NuGet/nuget.config: No such file or directory
PS /home/user> PackageManagement\Find-Package ImportExcel

Name                           Version          Source           Summary
----                           -------          ------           -------
ImportExcel                    7.6.0            PSGallery        PowerShell module to import/export Excel spreadsheets, without Excel.…

PS /home/user> cat .config/NuGet/nuget.config
<?xml version="1.0"?>
<configuration>
  <packageSources>
  </packageSources>
</configuration>

Info:

DEBUG: 00:00:00.0478035 Create a file '/home/user/.config/NuGet/nuget.config'.
DEBUG: 00:00:00.0486301 Calling 'NuGetRequest'::'RegisteredPackageSources', '/home/user/.config/NuGet/nuget.config'.
DEBUG: 00:00:00.0486422 INVOKING PowerShell Fn Find-Package with args System.String[], , ,  that has length 4
DEBUG: 00:00:00.0491085 Loading a configuration from file '/home/user/.config/NuGet/nuget.config'.
DEBUG: 00:00:00.0504660 Loaded a configuration from file '/home/user/.config/NuGet/nuget.config'.
DEBUG: 00:00:00.0514773 There are '0' registered sources in 'NuGet' provider.
VERBOSE: Total package yield:'0' for the specified package 'ImportExcel'.

DefaultConfig - https://github.com/OneGet/NuGetProvider/blob/16519ed5cbbae967a1aeaa383614bcb4505f46ab/NugetLightRequest.cs#L85:

PS /home/user> [Microsoft.PackageManagement.NuGetProvider.NugetRequest]::DefaultConfig
<?xml version="1.0"?>
<configuration>
  <packageSources>
  </packageSources>
</configuration>

https://github.com/OneGet/NuGetProvider/blob/16519ed5cbbae967a1aeaa383614bcb4505f46ab/NugetLightRequest.cs#L1495

                        //create place holder config file
                        if (!File.Exists(_configurationFileLocation))
                        {
                            Debug(Resources.Messages.CreateFile, _configurationFileLocation);
                            File.WriteAllText(_configurationFileLocation, DefaultConfig);
                        }

[kewalker]

@miketimofeev The bash commands just helped unblock a restore and build for me. +1

[zivkan]

@al-cheb When I test (on Windows) with PowerShell Core 7.2.4 (and Get-Module PackageManagement says version 1.4.7), I don't get this behaviour. It doesn't create the nuget.config at all, meaning the first time the dotnet cli runs, NuGet creates its own default.

I don't know what OneGet is, but someone should report to them that they're creating a different default to the official NuGet client, and this is causing customer problems.

For what it's worth, I recommend to EVERYONE to use dotnet new nugetconfig in your own repo root, and commit it. It creates a nuget.config that uses <clear/> to remove any inherited package sources, and adds nuget.org. This way your repo is not affected by machine state, which is much more important on CI agents where the VM is not wiped clean between every run. If you check Microsoft repos on github, I expect every single one of them will have their own nuget.config which clears inherited package sources.

[zivkan]

I see that OneGet is a microsoft repo, and that in April someone created a pull request to improve the default: https://github.com/OneGet/NuGetProvider/pull/66

I'm going to start an internal email thread trying to get more traction on this. If there's anyone in GitHub or Azure DevOps who wants to me included, ping me on Teams and I'll add you to the email.

[al-cheb]

@al-cheb When I test (on Windows) with PowerShell Core 7.2.4 (and Get-Module PackageManagement says version 1.4.7), I don't get this behaviour. It doesn't create the nuget.config at all, meaning the first time the dotnet cli runs, NuGet creates its own default.

I don't know what OneGet is, but someone should report to them that they're creating a different default to the official NuGet client, and this is causing customer problems.

For what it's worth, I recommend to EVERYONE to use dotnet new nugetconfig in your own repo root, and commit it. It creates a nuget.config that uses <clear/> to remove any inherited package sources, and adds nuget.org. This way your repo is not affected by machine state, which is much more important on CI agents where the VM is not wiped clean between every run. If you check Microsoft repos on github, I expect every single one of them will have their own nuget.config which clears inherited package sources.

@zivkan , I can easily reproduce as well on Windows:

PS > $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.2.5
PSEdition                      Core
GitCommitId                    7.2.5
OS                             Microsoft Windows 10.0.22000
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

PS > gc $env:USERPROFILE\AppData\Roaming\NuGet\nuget.config -ea 0
PS > Find-Package ImportExcel

Name                           Version          Source           Summary
----                           -------          ------           -------
ImportExcel                    7.6.0            PSGallery        PowerShell module to import/export Excel spreadsheets…

PS > gc $env:USERPROFILE\AppData\Roaming\NuGet\nuget.config -ea 0
<?xml version="1.0"?>
<configuration>
  <packageSources>
  </packageSources>
</configuration>

info:

DEBUG: 00:00:00.0432323 Create a file 'C:\Users\user\AppData\Roaming\NuGet\nuget.config'.
DEBUG: 00:00:00.0436914 Calling 'NuGetRequest'::'RegisteredPackageSources', 'C:\Users\user\AppData\Roaming\NuGet\nuget.config'.
DEBUG: 00:00:00.0440550 Loading a configuration from file 'C:\Users\user\AppData\Roaming\NuGet\nuget.config'.
DEBUG: 00:00:00.0451948 Loaded a configuration from file 'C:\Users\user\AppData\Roaming\NuGet\nuget.config'.

[mikhailkoliada]

Has been deployed.

[gep13]

@zivkan I noticed that the PR that you mentioned in the OneGet repository has been merged. Do you know if there is going to be a new release?

[zivkan]

Eventually, yes, I'm sure there will be. But it's complicated.

The PR I linked to is for a module called PackageManagement, and the version with the fix is here: https://www.powershellgallery.com/packages/PackageManagement/1.4.8. Notice how it says the package is unlisted. More on that below.

However, when I looked at PowerShell Core's repo for the dependabot upgrade of this module, I found this: https://github.com/PowerShell/PowerShell/pull/17606 It was closed because of this issue: https://github.com/OneGet/oneget/issues/533. Hence PackageManagement version 1.4.8 was unlisted because it was causing issues unrelated to the nuget.config file change.

So now the process is:

1. https://github.com/OneGet/oneget/issues/533 needs to be fixed
2. New version of PackageManagement published to PowerShell Gallery
3. dependabot to create PR on https://github.com/PowerShell/PowerShell to update to whatever this new version is
4. After dependabot's PR is merged, pwsh needs to release a new version
5. actions/virtual-environments needs to use that new version of pwsh

So, at this time, you're best off following the PowerShell repo. For what it's worth, I'm not going to be following up on this myself, and since pwsh is open on github, you can follow it as easily as I can.

Note that the above is only for PowerShell Core (pwsh). For Windows PowerShell (powershell.exe), it turns out a completely different team in Microsoft manage it (some team in Windows servicing). I've been super busy this week and haven't had time to follow up, but now that I learned that PackageManagement has a bug, I'll just wait for that to be fixed first.

It's highly likely that PowerShell Core is going to update before Windows PowerShell (after all, WinPS needs a Windows Update), so my best suggestion is to try to migrate off Windows PowerShell onto PowerShell Core, where possible. But this issue is about Mac agents, so WinPS is out of scope anyway.

I still HIGHLY recommend everyone run dotnet new nugetconfig in their repo root, add any package sources your repo needs (remove nuget.org if desired), and commit it. If your repo already has a nuget.config, then make sure <clear /> is in the package sources section before any other source. It makes your repo less prone to machine state, which is good for build reliability and reproducability.