Stop using the setAuthForSourcesInTempNuGetConfig in all package downloading scenarios, such as nuget install, restore, because it completely breaks PackageSourceMapping

nkolev92 commented 2 years ago

Note

Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo

For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks

If you have an issue or request for the Azure Pipelines service, use developer community instead:

https://developercommunity.visualstudio.com/spaces/21/index.html )

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: Tasks/DotNetCoreCLIV2/restorecommand.ts, but really all commands in this list https://github.com/microsoft/azure-pipelines-tasks/search?q=setAuthForSourcesInTempNuGetConfig

list here (V# not needed):
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks

Environment

Server - Azure Pipelines or TFS on-premises?
- If using TFS on-premises, provide the version: N/A
- If using Azure Pipelines, provide the account name, team project name, build definition name/build number: N/A
Agent - Hosted or Private:
- If using Hosted agent, provide agent queue name: N/A
- If using private agent, provide the OS of the machine running the agent and the agent version: N/A

Issue Description

[Include task name(s), screenshots and any other relevant details]

NuGet recently release a supply chain feature called PackageSourceMapping. That feature uses the package source keys from a NuGet.Config.

See details here: https://devblogs.microsoft.com/nuget/introducing-package-source-mapping/ and https://docs.microsoft.com/en-us/nuget/consume-packages/package-source-mapping.

https://docs.microsoft.com/en-us/nuget/consume-packages/package-source-mapping#enabling-package-source-mapping

Azure DevOps has many tasks that rewrite the config and add a feed- prefix, but unfortunately this breaks all source mapping configurations.

See https://github.com/NuGet/Home/issues/11406

Please stop relying on rewriting the nuget.config for these operations.

Task logs

https://github.com/microsoft/azure-pipelines-tasks/search?q=setAuthForSourcesInTempNuGetConfig

[Enable debug logging and please provide the zip file containing all the logs for a speedy resolution]

Troubleshooting

Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting

Error logs

[Insert error from the logs here for a quick overview]

nkolev92 commented 2 years ago

Collecting all the potentil tasks with issues based on the query:

https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops
- push - it doesn't really use package source mapping, but it'd be nice to go away from this completely.
- restore - This is the biggest problem, as this is the core target.
DownloadGitHubNugetPackageV1 - it has no docs afaik
- Restore command
https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/package/nuget?view=azure-devops
- restore
- Unsure what nugetpublisher, but that also uses that code.

If I understand it correctly, pretty much any dotnet restore or nuget.exe restore command would have issues.

nkolev92 commented 2 years ago

Any workarounds for this issue?

Would using NuGetAuthenticate negate the need for the addition of the feed- and a PAT? Is that a workaround for customers facing this issue?

magleaso commented 2 years ago

Yes, the NuGet Authenticate is what we recommend using for all scenarios going forward. We're in the process of deprecating the heavyweight NuGet command task and its equivalent in other protocols including dotnet

nkolev92 commented 2 years ago

Based on customer feedback, NuGetAuthenticate doesn't negate the issues here: https://github.com/NuGet/Home/issues/11406#issuecomment-978944155

Is this expected?

magleaso commented 2 years ago

Responded there. The user has a misunderstanding of how the authenticate task is supposed to be used. It only handles authentication and should then be followed by a command line invocation of the standard CLI client. Any bad behavior at that point is bad behavior in the standard client, not any code that was written in a pipeline task.

nkolev92 commented 2 years ago

Do you think there's something that can be done about how changing the tasks behavior to not change up the config?

My concern is that the errors customers would hit are difficult to diagnose.

magleaso commented 2 years ago

Our team is still discussing internally how we'll handle this. We are trying to discontinue any work on the heavyweight tasks and encourage adoption of the authenticate tasks

nkolev92 commented 2 years ago

Thank you!

Please let us know if there's anything we can do to help from NuGet side.

fyi @aortiz-msft @jondouglas

nkolev92 commented 2 years ago

Hey @magleaso,

Any progress on an in-situ fix for the DotnetCLI task?

Thanks!

magleaso commented 2 years ago

No, but I can tell you the outcome where we patch the DotnetCLI task, if it does happen, will still take further time. For the time being, the solution is to use the authenticate task.

ransagy commented 2 years ago

Just to be clear - Are we saying people should, in general, not ever use DotNetCoreCLIV2 task anymore and replace it by cmd/bash/pwsh calls? (and maybe NuGetV0/NuGetCommandV2? others?)

Peter-B- commented 2 years ago

Hello,

I just want to sum up the suggestion by @nkolev92 for reference:

Replace the normal dotnet restore task

- task: DotNetCoreCLI@2
  displayName: dotnet restore
  inputs:
    command: 'restore'
    restoreArguments: '--nologo'
    feedsToUse: 'config'
    nugetConfigPath: 'Nuget.config'
    verbosityRestore: 'Normal'

with a separate Nuget authenticate and a Powershell dotnet restore task:

- task: NuGetAuthenticate@1
  displayName: 'Authenticate with NuGet'

- task: PowerShell@2
  displayName: 'dotnet restore'
  inputs:
    targetType: 'inline'
    script: 'dotnet restore --nologo'

zivkan commented 2 years ago

FWIW, I think it was actually @magleaso's suggestion: https://github.com/microsoft/azure-pipelines-tasks/issues/15542#issuecomment-982018356

Tiberriver256 commented 1 year ago

I agree with the goal to deprecate that task... it's always been a bit of a strange one. I would definitely recommend getting some communication out on this though in a larger way. The migration effort for a lot of companies is going to be fairly significant. Early comms on this can help stop the bleeding and reduce the number of migrations. I just luckily happened to stumble on this Github issue and will stop using that task, I can't imagine too many others are going to luck out like I did.

icecoldfire commented 1 year ago

We still have the same bug using package source mapping

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

jbrezina commented 1 year ago

Would be nice if anyone fixes this issue as it is annoying and time consuming to fix it. I spent three days searching for a solution until I found this issue and noticed a little note at the end of the documentation for Source Mapping that it is not recommended to use the task which is supposed to do the job (DotNetCoreCLI@2 with restore command) but it's not able to do it properly..

philippedurocher commented 9 months ago

The suggestion from Peter-B- is how I was able to make it work, but the pipeline crashes 1 to 2 times a day with this error :

Starting: Restore NuGet packages w/PowerShell --== Task : PowerShell Description : Run a PowerShell script on Linux, macOS, or Windows Version : 2.228.0 Author : Microsoft Corporation Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell --== Generating script. --======================== Starting Command Output =========================== "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a_temp\da76d530-4668-45c1-8f08-a739a4ed726c.ps1'" Determining projects to restore... C:\hostedtoolcache\windows\dotnet\sdk\8.0.100-rc.1.23463.5\NuGet.targets(156,5): error : Problem starting the plugin 'C:\Users\VssAdministrator.nuget\plugins\netcore\CredentialProvider.Microsoft\CredentialProvider.Microsoft.dll'. Plugin 'CredentialProvider.Microsoft' failed within 30.074 seconds with exit code -1. [D:\a\1\s\OXXXX.sln] C:\hostedtoolcache\windows\dotnet\sdk\8.0.100-rc.1.23463.5\NuGet.targets(156,5): error : Unable to load the service index for source https://pkgs.dev.azure.com/cXXXXXX/2XXXXXX/_packaging/WXXXXX-Feed/nuget/v3/index.json. [D:\a\1\s\OXXXX.sln] C:\hostedtoolcache\windows\dotnet\sdk\8.0.100-rc.1.23463.5\NuGet.targets(156,5): error : Response status code does not indicate success: 401 (Unauthorized). [D:\a\1\s\OXXXX.sln]

[error]PowerShell exited with code '1'.

Finishing: Restore NuGet packages w/PowerShell

When it happens, I restart the failed task and it just works. It's annoying and time consuming.

github-actions[bot] commented 3 months ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

zivkan commented 3 months ago

I haven't tested this recently, but nobody has replied here to say that it's fixed. I think it would still be useful.

While Azure Artifacts supports upstream sources, allowing some customers to use nuget.config with a single source, and avoid needing to use Package Source Mapping, it still has a limitation where it only allows other Azure Artifacts feeds from the same org, and nuget.org as upstream sources. The "custom registry" upstream source configuration only allows nodejs feeds, not NuGet. Therefore, if a project needs to use packages across multiple Azure DevOps organization accounts, or a different custom NuGet feed (such as something running on-prem), Azure Artfiacts doesn't provide an alternative to multiple feeds in nuget.config, at which point people might want to use Package Source Mapping.

smkanadl commented 3 months ago

I think the DotNetCoreCli task is pretty dead. When I implemented package source mapping, I moved to use the new NugetAuthenticate task and simply call dotnet restore from a script task. That works quite well.

ChristoWolf commented 3 months ago

Yep, I have also been doing that for years now, the task is too unreliable to use.

MFunction96 commented 1 month ago

I met this issue recently. I had replace dotnet restore in DotNetCoreCLI@2 with dotnet restore in PowerShell@2 and it was worked quite well. So, is there any updates here?

stevenmolencsat commented 1 week ago

The problem with the assumption that using the nugetauthenticate task is always going to be the tool for something like this. What happens if we're doing a build in the docker context where it doesn't have access to nugetauthenticate?

microsoft / azure-pipelines-tasks