Nuget Restore Task fails with new image 20230820.1.0

lbass2 commented 1 year ago

Description

Hello,

As part of our build in Azure Pipelines, we use nuget restore. Currently the pipeline is set to use "windows-latest" as agent specification.
Since today, multiple builds failed, looking randomly at first. But comparing failed builds to ones that succeed, the image version differs. No other changes are known to the sourcecode/the pipeline itself.

Builds working run on version 20230804.1.0, while builds that run on version 20230820.1.0 always seem to fail.

Platforms affected

[X] Azure DevOps
[ ] GitHub Actions - Standard Runners
[ ] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 20.04
[ ] Ubuntu 22.04
[ ] macOS 11
[ ] macOS 12
[ ] macOS 13
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link

Starting: Initialize job Agent name: 'Azure Pipelines 6' Agent machine name: 'fv-az767-114' Current agent version: '3.224.1' Operating System Runner Image Runner Image Provisioner Current image version: '20230820.1.0' Agent running as: 'VssAdministrator' Prepare build directory. Set build variables. Download all required tasks. Downloading task: GitVersion (5.0.1) Downloading task: Assembly-Info-NetFramework (2.3.105) Downloading task: NuGetToolInstaller (0.221.0) Downloading task: NuGetCommand (2.222.0) Downloading task: VSBuild (1.214.0) Downloading task: VSTest (2.224.0) Downloading task: CmdLine (1.1.3) Downloading task: PublishBuildArtifacts (1.225.0) Downloading task: CmdLine (2.212.0) Downloading task: variabledehydration (0.2.11) Downloading task: PowerShell (2.226.1) Downloading task: PublishSymbols (2.225.0) Downloading task: PostBuildCleanup (3.1.1) Checking job knob settings. Knob: DockerActionRetries = true Source: $(VSTSAGENT_DOCKER_ACTION_RETRIES) Knob: AgentToolsDirectory = C:\hostedtoolcache\windows Source: ${AGENT_TOOLSDIRECTORY} Knob: AgentPerflog = c:\vsts\perflog Source: ${VSTS_AGENT_PERFLOG} Knob: AgentEnablePipelineArtifactLargeChunkSize = true Source: $(AGENT_ENABLE_PIPELINEARTIFACT_LARGE_CHUNK_SIZE) Knob: ContinueAfterCancelProcessTreeKillAttempt = true Source: $(VSTSAGENT_CONTINUE_AFTER_CANCEL_PROCESSTREEKILL_ATTEMPT) Finished checking job knob settings. Start tracking orphan processes. Finishing: Initialize job

Is it regression?

20230804.1.0

Expected behavior

Expect to run nuget restore without error

Actual behavior

Nuget restore fails with the following exception:

Starting: NuGet restore
==============================================================================
Task         : NuGet
Description  : Restore, pack, or push NuGet packages, or run a NuGet command. Supports NuGet.org and authenticated feeds like Azure Artifacts and MyGet. Uses NuGet.exe and works with .NET Framework apps. For .NET Core and .NET Standard apps, use the .NET Core task.
Version      : 2.222.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/package/nuget
==============================================================================
C:\Windows\system32\chcp.com 65001
Active code page: 65001
Detected NuGet version 4.4.1.4656 / 4.4.1
C:\hostedtoolcache\windows\NuGet\4.4.1\x64\nuget.exe restore D:\a\1\s\Solution.sln -PackagesDirectory D:\a\1\s/packages -Verbosity Detailed -NonInteractive -ConfigFile D:\a\1\Nuget\tempNuGet_73248.config
Error parsing solution file at D:\a\1\s\Solution.sln: Exception has been thrown by the target of an invocation.  Could not load file or assembly 'Microsoft.Build.Framework, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
NuGet Version: 4.4.1.4656
MSBuild auto-detection: using msbuild version '17.7.2.37605' from 'C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\bin'. Use option -MSBuildVersion to force nuget to use a specific version of MSBuild.
NuGet.CommandLine.CommandLineException: Error parsing solution file at D:\a\1\s\Solution.sln: Exception has been thrown by the target of an invocation.  Could not load file or assembly 'Microsoft.Build.Framework, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
   at NuGet.CommandLine.MsBuildUtility.GetAllProjectFileNamesWithMsBuild(String solutionFile, String msbuildPath)
   at NuGet.CommandLine.RestoreCommand.ProcessSolutionFile(String solutionFileFullPath, PackageRestoreInputs restoreInputs)
   at NuGet.CommandLine.RestoreCommand.<DetermineRestoreInputsAsync>d__42.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NuGet.CommandLine.RestoreCommand.<ExecuteCommandAsync>d__34.MoveNext()
##[error]The nuget command failed with exit code(1) and error(Error parsing solution file at D:\a\1\s\Solution.sln: Exception has been thrown by the target of an invocation.  Could not load file or assembly 'Microsoft.Build.Framework, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.)
##[error]Packages failed to restore
Finishing: NuGet restore

Repro steps

Start a build. If running on the new image, it will likely fail.

Alexey-Ayupov commented 1 year ago

Hi @lbass2, could you please provide minimal repro steps? Getting a code snippet would be ideal.

vpolikarpov-akvelon commented 1 year ago

Hey @lbass2. It's possible that the recent upgrade of Visual Studio could be causing compatibility issues with your older NuGet version (4.4.1), which was released more than four years ago. I recommend updating NuGet to the newer version. Newer versions of NuGet are typically more compatible with latest Visual Studio releases.

vjout commented 1 year ago

We are also facing same issue with new image and MSBuild version 17.7.2.37605. Our nuget version is 6.4.

All our builds worked fine yesterday. MSBuild version of previous images was 17.6.3.22601.

The error is Could not load file or assembly 'Microsoft.Build.Framework, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

Nuget restore fails with the following exception: Active code page: 65001 Detected NuGet version 6.4.0.123 / 6.4.0+da5b9fa1595e212914854a9ff4a18a6759ed4729.da5b9fa1595e212914854a9ff4a18a6759ed4729 C:\hostedtoolcache\windows\NuGet\6.4.0\x64\nuget.exe restore -Verbosity Detailed -NonInteractive NuGet Version: 6.4.0.123 MSBuild auto-detection: using msbuild version '17.7.2.37605' from 'C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\bin'. Use option -MSBuildVersion to force nuget to use a specific version of MSBuild. MSBuild P2P timeout [ms]: 312000

Exception has been thrown by the target of an invocation. Could not load file or assembly 'Microsoft.Build.Framework, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

mikemaat commented 1 year ago

Unpinning the Nuget version number (clear the number out of the Nuget Tool Installer task) worked for me. Or re-add the Nuget Tool Installer task and leave the defaults.

lbass2 commented 1 year ago

I set the used Nuget version to the latest available (at least for the Nuget installer task), which is 6.6.1. The restore task now runs fine again.

mikhailkoliada commented 1 year ago

Looks like duplicate of https://github.com/actions/runner-images/issues/8137, lets move there, we are working on it right now.

stajs commented 1 year ago

Just saying, I don't see how this (NuGetCommand/ NuGetToolInstaller) is a dupe of #8137 (UseDotNet). So I'll just add my info.

YAML snippet:

- task: NuGetToolInstaller@0
  displayName: 'Use NuGet 4.4.1'
  inputs:
    versionSpec: 4.4.1

- task: NuGetCommand@2
  displayName: 'NuGet restore'
  inputs:
    restoreSolution: '$(solution)'

Yes, it's old and using @0 instead of @1, but testing both versions didn't help.

Log snippets:

Starting: Use NuGet 4.4.1
...
Caching tool: NuGet 4.4.1 x64
Using version: 4.4.1
Found tool in cache: NuGet 4.4.1 x64
Using tool path: C:\hostedtoolcache\windows\NuGet\4.4.1\x64
Prepending PATH environment variable with directory: C:\hostedtoolcache\windows\NuGet\4.4.1\x64

Starting: NuGet restore
...
Error parsing solution file at D:\a\1\s\server\[REDACTED].sln: Exception has been thrown by the target of an invocation.  Could not load file or assembly 'Microsoft.Build.Framework, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
NuGet Version: 4.4.1.4656
MSBuild auto-detection: using msbuild version '17.7.2.37605' from 'C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\bin'. Use option -MSBuildVersion to force nuget to use a specific version of MSBuild.

I did try to specify the -MSBuildVersion but couldn't change it (none of the many versions I tried worked).

I ended up removing the NuGetToolInstaller task to unpin the version and letting ADO use a recent one (in this case 6.4.0) which worked.

Enigma644 commented 1 year ago

@stajs Thank You for posting this!! I've been having this exact same issue for the last few days, Googling solutions with no useful results, then I found this!! Updating nuget fixed my build pipeline! :-D

stijnherreman commented 1 year ago

Same fix works for broken Azure DevOps pipelines on Microsoft-hosted agents.

actions / runner-images