search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.47k stars 2.6k forks source link

[REGRESSION]: The nuget command failed with exit code(null) and error() #20351

Open NealOConnell opened 2 weeks ago

NealOConnell commented 2 weeks ago

New issue checklist

Task name

NuGetCommand

Breaking task version

2.244.1

Last working task version

2.238.1

Regression Description

Builds just started failing with The nuget command failed with exit code(null) and error().

When comparing failed and successful builds, noticed that the within 'initialize job', all of the failed jobs were using this newer task version 2.244.1. After forcing the task back to 2.238.1, builds became successful again

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

windows-latest

Relevant log output

[only a tiny snippet of the end of the log]

Added file 'ManuScripts/DCTTemplates/CarrierProperty/MI/Carrier_Property_MI_00_35.xml'.
Added file 'ManuScripts/DCTTemplates/CarrierProperty/MI/Carrier_Property_MI_00_36.xml'.
Added file 'ManuScripts/DCTTemplates/CarrierProperty/MI/Carrier_Property_MI_00_37_01.xml'.
Added file 'ManuScripts/DCTTemplates/CarrierProperty/MI/Carrier_Property_MI_00_38.xml'.
Added file 'ManuScripts/DCTTemplates/CarrierProperty/MI/Carrier_Property_MI_00_39.xml'.
Added file 'ManuScripts/DCTTemplates/CarrierProperty/MI/Carrier_Property_MI_00_40.xml'.
Added file 'ManuScripts/DCTTemplates/CarrierProperty/MI/Carrier_Property_MI_00_41.xml'.
Added file 'ManuScripts/DCTTemplates/CarrierProperty/MI/Carrier_Property_MI_00_42.xml'.
Added file 'ManuScripts/DCTTemplates/CarrierProperty/MI/Carrier_Property_MI_00_43.xml'.
##[error]The nuget command failed with exit code(null) and error()
##[error]An error occurred while trying to pack the files.
Finishing: NuGetCommand

Full task logs with system.debug enabled

UNSUCCESSFUL RUN 
[FailedBuild.log](https://github.com/user-attachments/files/16750944/FailedBuild.log)
SUCCESSFUL RUN 
[SuccessfulBuild.log](https://github.com/user-attachments/files/16750948/SuccessfulBuild.log)

Repro steps

# unclear if it matters, but encountered this with a parameterized build
# ADO pipeline is hooked up to a YML file that extends a parameterized YML file with components like the below
# this is not a full YML file - just snips of our particular YML file that highlights the problem
parameters:
- name: buildtype
  displayName: 'Build Type?'
  type: string
  default: ci
  values:
  - ci
  - feature
  - develop
  - hotfix
  - main
  - upgrade

variables:
  partialbuildnumber:  $[ format('{0:yyyy}.{0:MM}.{0:dd}.{1}', pipeline.startTime,counter(format('{0:yyyyMMdd}', pipeline.startTime), 1))]
  fullbuildnumber: "$(partialbuildnumber)-${{parameters.buildtype}}"

  # instance of the problem  happens to check out two separate repos
  # one of the repos is checked out to a folder named "Policy"
  - task: NuGetCommand@2.244.1
    inputs:
      command: 'pack'
      packagesToPack: '$(Pipeline.Workspace)/**/Policy.nuspec'
      versioningScheme: byEnvVar
      versionEnvVar: fullbuildnumber
      buildProperties: 'PolicyFolder=$(Pipeline.Workspace)\Policy'
gandalf1990PL commented 2 weeks ago

We're facing the same but with Self-Hosted pools based on ubuntu image. After forcing version 2.238.1 for nuget command step all has been working smoothly.

NealOConnell commented 2 weeks ago

If I'm reading things correctly, release info implies that the latest release is v243: https://github.com/microsoft/azure-pipelines-tasks/releases

Did unstable builds that are still in development get published early?

EdmentumDevOps commented 2 weeks ago

Confirming same issue on MS Hosted systems - One is from a Scale set and the the other on any MS hosted. Downgrading this step to 2.238.1 is allowing builds to pass.

jano-kucera commented 2 weeks ago

Confirming same issue on MS Hosted systems - One is from a Scale set and the the other on any MS hosted. Downgrading this step to 2.238.1 is allowing builds to pass.

I'm having trouble finding how to specify this version, could you give a hint please?

Edit: simple as this: - task: NuGetCommand@2.238.1

james1301 commented 2 weeks ago

We are getting the same issue with the same version.

LeeMeyrickBR commented 2 weeks ago

We have the same on a self-hosted VMSS

nourkilany commented 2 weeks ago

We have the same issue reverting to 2.238.1 solves the issue for now

Glacial commented 2 weeks ago

Same here. The workaround to fix the task version to 2.238.1 is getting stuff running for now.

niallmccabe commented 2 weeks ago

We are experiencing the same issue. Unfortunately it is on a classic pipeline where the task version can't be amended, beyond a dropdown containing "2., 1. 0.*". Currently we don't have the capacity to rebuild this pipeline using YAML. Is there a fix for this underway?

awaldow commented 2 weeks ago

Same here on the classic pipeline woes, lack of ability to change versions means we had to run our nuget restore stuff in a script command instead, which I think will be an easy workaround for simple cases but probably not good for all.

clb900528 commented 2 weeks ago

I am having the same issue. When I change the version to 2.238.1 then it works

leaningazure commented 2 weeks ago

Same here on the classic pipeline woes, lack of ability to change versions means we had to run our nuget restore stuff in a script command instead, which I think will be an easy workaround for simple cases but probably not good for all.

How did you create script task? I have classic setup and looking to override the version with minimal work.

josh-cooley commented 2 weeks ago

For me, this happens restoring nugets in a solution that has .NET Framework 4.8 projects. I'm not seeing the error in pipelines that have no framework projects. Pinning the version to 2.238.1 works around the issue for those pipelines affected.

m-soltani commented 2 weeks ago

It's happening in our builds as well, not all pipelines containing NuGet restore are affected by this regression. Not sure why some pipelines are passing and some not

awaldow commented 2 weeks ago

Same here on the classic pipeline woes, lack of ability to change versions means we had to run our nuget restore stuff in a script command instead, which I think will be an easy workaround for simple cases but probably not good for all.

How did you create script task? I have classic setup and looking to override the version with minimal work.

I just created a Powershell task and had it run essentially the same command that the nuget restore task would run. You can look at a previous run and look at the top of the output to see what all the task options turn into as a CLI command and then just do some variable substitution.

chongtian commented 2 weeks ago

It's happening in our builds as well, not all pipelines containing NuGet restore are affected by this regression. Not sure why some pipelines are passing and some not

We see the same behavior. Even within the same build run, most of our solutions are successfully restored by NuGet, while only one solution fails. The failed solution has thousands of packages to restore. Maybe this new NuGet cannot handle a large volume of packages.

cormacpayne commented 2 weeks ago

Hey folks, apologies for the regression in behavior for the NuGetCommand task -- we have finished a deployment that rolls back the latest version of the task to point to the previously working version of 2.238.1 (including classic YAML pipelines). Please feel free to let us know if you are still seeing issues when targeting the latest version of the NuGetCommand task.

Biboulin commented 2 weeks ago

I can confirm that the issue doesn't appear anymore. When targeting the latest version of the NuGetCommand it takes the 2.238.1 version by default.

jano-kucera commented 4 days ago

Just now: our pipeline used version NuGet@2.245.3 and is failing once again.

stuartag commented 4 days ago

Same here with 2.245.3 now being used and displaying the same issue.

maykQics commented 4 days ago

Same on our side. Pipeline just stopped working.

NealOConnell commented 4 days ago

Same

First observed failure for me was when after fetching 2.245.3 at 2024-09-09T09:07:43.9120785Z

Last observed success for me was after fetching 2.238.1 at 2024-09-09T08:46:47.0764253Z

Noticing that the project page currently shows v243 as the latest release.

https://github.com/microsoft/azure-pipelines-tasks image

This raises some questions:

  1. If 243 is the latest release, then why (until very recently) haven't any of my pipelines runs been pulling anything except 238?
  2. If 243 is the latest release, then why have the buggy 244 and 245 releases been getting pulled into my pipelines?

@cormacpayne - thanks for handling this relatively quickly 2 weeks ago. Can you offer any insights on why this task is getting so unstable?

jano-kucera commented 4 days ago

Today it's getting the 2.238.1 version again and works.

the-coding-panda commented 3 days ago

Sorry its getting version 2.245.3 for us today and failing again with the same issue.

htzhang2 commented 3 days ago

Our pipeline breaks because of new version 2.245.3. Works fine with 2.238.1. Our pipeline script:

Error: Starting: Download NuGet packages

Task : NuGet Description : Restore, pack, or push NuGet packages, or run a NuGet command. Supports NuGet.org and authenticated feeds like Azure Artifacts and MyGet. Uses NuGet.exe and works with .NET Framework apps. For .NET Core and .NET Standard apps, use the .NET Core task. Version : 2.245.3 Author : Microsoft Corporation Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/package/nuget

...

[error]The nuget command failed with exit code(null) and error()

[debug]Processed: ##vso[task.issue type=error;]The nuget command failed with exit code(null) and error()

[debug]task result: Failed

[error]Packages failed to restore

Can you add regression test to catch this failure?

phuoctung28 commented 2 days ago

Do we have any workaround solutions for this kind of issue? Any script can help

jano-kucera commented 2 days ago

Do we have any workaround solutions for this kind of issue? Any script can help

You can use a specific version of the NuGet command - see my comment from previous weeks way above.

JesperNoerregaard commented 2 days ago

Do we have any workaround solutions for this kind of issue? Any script can help

You can use a specific version of the NuGet command - see my comment from previous weeks way above.

What about users of the classic pipeline?

phuoctung28 commented 2 days ago

Do we have any workaround solutions for this kind of issue? Any script can help

You can use a specific version of the NuGet command - see my comment from previous weeks way above.

What about users of the classic pipeline?

You can create a ps script test as it works for me: 

# Define paths for the solution and NuGet configuration
$solutionPath = ......
$nugetConfigPath = ....

# Check if NuGet is installed
if (-not (Get-Command nuget -ErrorAction SilentlyContinue)) {
    Write-Error "NuGet is not installed or available in the PATH."
    exit 1
}

# Restore NuGet packages for the solution
Write-Host "Restoring NuGet packages for solution $solutionPath using config $nugetConfigPath"
nuget restore $solutionPath -ConfigFile $nugetConfigPath

# Check for success or failure
if ($LASTEXITCODE -ne 0) {
    Write-Error "NuGet restore failed."
    exit $LASTEXITCODE
}
rumukh commented 2 days ago

@cormacpayne our pipeline started to fail recently with this error. Has the issue been reintroduced?

jimmycartrette commented 2 days ago

Same exact situation here, classic pipelines so no quick workaround for us

LukaszNowakowski commented 2 days ago

For us it's YAML pipelines (but converted from classic, not sure if it matters) and started today. Using NuGetCommand@2.

leomoty commented 2 days ago

For us it's YAML pipelines (but converted from classic, not sure if it matters) and started today. Using NuGetCommand@2.

As previously said, just use the specific version that doesn't have issues: NuGetCommand@2.238.1 instead of NuGetCommand@2

LukaszNowakowski commented 2 days ago

@leomoty, thanks, sorry, must have missed the comment.

naresh-confirmtkt commented 2 days ago

We are also facing this issue with nuget restore, is there a way to to force the version to 2.238.1 when using classic pipeline ? We are also using private feeds, any script to run the nuget restore with private feeds using powershell ? Or any solution for this ?

nbouteiller commented 2 days ago

@naresh-confirmtkt Use latest "Nuget Tool installer" step to version 6.11.0 It resolved the issue for classic pipelines.

jimmycartrette commented 2 days ago

@naresh-confirmtkt Use latest "Nuget Tool installer" step to version 6.11.0 It resolved the issue for classic pipelines.

I tried this but it didn't change anything

nbouteiller commented 2 days ago

@naresh-confirmtkt Use latest "Nuget Tool installer" step to version 6.11.0 It resolved the issue for classic pipelines.

I tried this but it didn't change anything

You can check the version used by finding "NuGet Version: 6.11.0.119" in NuGet restore step. If the version is this one and still not working, I have no other solution to propose.

slapshot36227 commented 2 days ago

This is happening for us too, hundreds of pipelines (combination of classic and yml) are now failing with no easy workaround.

naresh-confirmtkt commented 2 days ago

@naresh-confirmtkt Use latest "Nuget Tool installer" step to version 6.11.0 It resolved the issue for classic pipelines.

@nbouteiller Thank you, this had fixed the issue :)

ericrrichards commented 2 days ago

Do we have any workaround solutions for this kind of issue? Any script can help

You can use a specific version of the NuGet command - see my comment from previous weeks way above.

What about users of the classic pipeline?

You can create a ps script test as it works for me:

# Define paths for the solution and NuGet configuration
$solutionPath = ......
$nugetConfigPath = ....

# Check if NuGet is installed
if (-not (Get-Command nuget -ErrorAction SilentlyContinue)) {
    Write-Error "NuGet is not installed or available in the PATH."
    exit 1
}

# Restore NuGet packages for the solution
Write-Host "Restoring NuGet packages for solution $solutionPath using config $nugetConfigPath"
nuget restore $solutionPath -ConfigFile $nugetConfigPath

# Check for success or failure
if ($LASTEXITCODE -ne 0) {
    Write-Error "NuGet restore failed."
    exit $LASTEXITCODE
}

Thanks, I couldn't get anything else suggested here to work, but just replacing the broken step with a basic powershell step like this worked out

ruibragahitachi commented 2 days ago

Hi guys,

image

For the classic pipeline users, create an Agent.Version variable in the pipeline with value 2.238.1 to force to use this version instead of the 2.245.3

ruibragahitachi commented 2 days ago

Hi guys,

image

For the classic pipeline users, create an Agent.Version variable in the pipeline with value 2.238.1 to force to use this version instead of the 2.245.3

Seems it was not this fix that solved it, now I am always getting 2.238.1, even without the variable. Not sure.

slapshot36227 commented 2 days ago

This is happening for us too, hundreds of pipelines (combination of classic and yml) are now failing with no easy workaround.

it's working for me now, it looks like they reverted back to 2.238.1 again (both classic and yml)

NealOConnell commented 1 day ago

This is happening for us too, hundreds of pipelines (combination of classic and yml) are now failing with no easy workaround.

it's working for me now, it looks like they reverted back to 2.238.1 again (both classic and yml)

I'm waiting for someone from Microsoft to chime on and state that they've rolled back 2.245.3.

Until Microsoft actually addresses this specific open issue, the current data set suggests to me that every new version of this task is going to include this same problem.

After a Microsoft dev reported that 2.244.1 was rolled back, I rolled back my own pipeline changes.

After 2.245.3 was published and revealed to include this same problem, I am sticking explicitly with 2.238.1 until this issue has been proven to be fixed. My teams simply cannot afford the disruption caused by this bug.