Paket does not seem to work with Azure Artifacts Credential Provider

daniel-chambers commented 4 years ago

Description

If I install the Azure Artifacts Credential Provider, Paket does not seem to work with it, and fails to understand what the credential provider provides. (I've investigated why, read on!)

Repro steps

Install the Azure Artifacts Credentials Provider using the installation steps here. It installs the credential provider into %UserProfile%/.nuget/plugins/netcore and %UserProfile%/.nuget/plugins/netfx which is apparently the standard location for NuGet plugins.
Paket expects the Credentials Provider in a different location (maybe an old location?), so copy %UserProfile%\.nuget\plugins\netcore\CredentialProvider.Microsoft to %LOCALAPPDATA%\NuGet\CredentialProviders%.

Create a paket.dependencies with a source pointing to your Azure Artifacts feed and use a package that is only in your private feed.


source https://api.nuget.org/v3/index.json
source https://pkgs.dev.azure.com/COMPANY/_packaging/FEEDNAME/nuget/v3/index.json
framework auto-detect

nuget FSharp.Core nuget MyInternalPackage 0.0.72

* `> paket install -v` on the command line.

### Expected behavior

A successful `paket install` that grabs the necessary credentials from the provider.

### Actual behavior

It fails. 😭 The relevant part of the logs are:

-> Credential provider returned an invalid result: [Information] [CredentialProvider]Username: optional [Information] [CredentialProvider]Password: CENSOREDFORBUGREPORT Error: -> JsonSerializationException: Cannot deserialize the current JSON array (e.g. [1,2,3]) into type 'Paket.CredentialProviderResultMessage' because the type requires a JSON object (e.g. {"name":"value"}) to deserialize correctly. To fix this error either change the JSON to a JSON object (e.g. {"name":"value"}) or change the deserialized type to an array or a type that implements a collection interface (e.g. ICollection, IList) like List that can be deserialized from a JSON array. JsonArrayAttribute can also be added to the type to force it to deserialize from a JSON array. Path '', line 1, position 1.


The problem appears to be twofold.
1. Paket is looking for credential providers in the wrong (old?) location
2. The Azure Artifacts Credential Provider has changed the command line arguments it expects to receive. (TLDR: It seems like you need to pass `-F json` for it to print JSON on the command line, otherwise it defaults to "Human Readable")

If I run the credential provider by hand:

& "C:\Program Files\dotnet\dotnet.exe" "C:\Users\dchambers.NuGet\plugins\netcore\CredentialProvider.Microsoft\CredentialProvider.Microsoft.dll" -uri https://pkgs.dev.azure.com/COMPANY/_packaging/FEEDNAME/nuget/v3/index.json [Information] [CredentialProvider]Username: optional [Information] [CredentialProvider]Password: CENSOREDFORBUGREPORT

According to the help text for the credential provider:

OutputFormat (-F)     In standalone mode, format the results for human readability or as JSON. If JSON is selected, then logging (which may include Device Code
                      instructions) will be logged to standard error instead of standard output.
                      HumanReadable
                      Json

If I add -F json to the command line:

> & "C:\Program Files\dotnet\dotnet.exe" "C:\Users\dchambers\.NuGet\plugins\netcore\CredentialProvider.Microsoft\CredentialProvider.Microsoft.dll" -uri https://pkgs.dev.azure.com/COMPANY/_packaging/FEEDNAME/nuget/v3/index.json -F json
{"Username":"optional","Password":"CENSOREDFORBUGREPORT"}

Given that Paket's code seems to copy what is in NuGet, I was surprised that such a breaking change would have been made by Microsoft! So I investigated what dotnet does when it uses the credential provider by using Process Monitor to watch the command line args it uses.

So apparently NuGet must use a completely different mode of interaction with these plugins! This seems to be this "plugin" mode, whereas paket is using "standalone" mode, and they've broken back-compat by having this -F flag. 😭

Known workarounds

None for us. We build on internal TeamCity build agents, and these have been configured with the Azure Artifacts Credential Provider. The old CredentialProvider.VSS.exe (which is no longer available via Azure Artifacts as in the Paket doco) seems to pop an interactive dialog and therefore is not suitable on the build agents (it does not seem to respect the VSS_NUGET_EXTERNAL_FEED_ENDPOINTS environment variable the new provider uses to go non-interactive).

daniel-chambers commented 4 years ago

Further reading leads me to believe that what Paket supports is "nuget.exe credential providers", whereas Azure Artifacts provides a "NuGet cross platform authentication plugin". These appear to be a new way of providing credentials to NuGet. 😢

Edit: There's some history about these older and newer providers and doco on how NuGet.exe interacts with them here.

forki commented 4 years ago

We're open to pull requests. @isaacabraham do you know something about this?

Daniel Chambers notifications@github.com schrieb am Fr., 20. März 2020, 07:39:

Further reading leads me to believe that what Paket supports is "nuget.exe credential providers https://docs.microsoft.com/en-au/nuget/reference/extensibility/nuget-exe-credential-providers", whereas Azure Artifacts provides a "NuGet cross platform authentication plugin https://docs.microsoft.com/en-au/nuget/reference/extensibility/nuget-cross-platform-authentication-plugin". These appear to be a new way of providing credentials to NuGet. 😢

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/fsprojects/Paket/issues/3813#issuecomment-601556629, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAOANCMBOJ5DJFR2SUYK5LRIMFRPANCNFSM4LQA3FBA .

daniel-chambers commented 4 years ago

I've done a little sniffing around NuGet and the Azure Artifacts Credential Provider code. It seems like this "Plugin" model is implemented in the NuGet.Protocol package. Would we be against using that in Paket so we don't have to re-implement everything from scratch?

Some code links for my reference and if anyone else is interested:

forki commented 4 years ago

The problem is usually lots and lots of other deps that come indirectly

Daniel Chambers notifications@github.com schrieb am So., 22. März 2020, 07:59:

I've done a little sniffing around NuGet and the Azure Artifacts Credential Provider code. It seems like this "Plugin" model is implemented in the NuGet.Protocol package. Would we be against using that in Paket so we don't have to re-implement everything from scratch?

Some code links for my reference and if anyone else is interested:

Connecting to a Plugin - PluginFactory.CreatePluginAsync() https://github.com/NuGet/NuGet.Client/blob/6f7be0986028256f05c894eac2f7816c85611984/src/NuGet.Core/NuGet.Protocol/Plugins/PluginFactory.cs#L146

Credentials Request Contract - GetAuthenticationCredentialsRequest https://github.com/NuGet/NuGet.Client/blob/6f7be0986028256f05c894eac2f7816c85611984/src/NuGet.Core/NuGet.Protocol/Plugins/Messages/GetAuthenticationCredentialsRequest.cs

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/fsprojects/Paket/issues/3813#issuecomment-602157731, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAOANFZ5NEK3SJMDZOG4OTRIWZORANCNFSM4LQA3FBA .

matthid commented 4 years ago

Note that IIRC there is already an open issue/discussion somewhere. There is even an open source external adapter implementation from someone. Before starting to implement anything we should try to link the existing work/discussion.

daniel-chambers commented 4 years ago

Looks like that issue is https://github.com/fsprojects/Paket/issues/3531. @slang25 went down the same path with NuGet.Protocol as I was thinking.

Evangelink commented 3 years ago

Any news on the support of Azure credentials? Would it be possible to get help from people in the nuget team?

FedericoBinda commented 3 years ago

I found a solution to the issue! Here is what I did (Mac OS):

Install the Azure Artifacts Credential Provider following the manual installation (here). Notice that you need to copy only the netcore folder. DO NOT COPY the netfx folder
Manually run the provider to obtain the credentials: dotnet CredentialProvider.Microsoft.dll -U <feed_url> and follow the instructions (not sure if this step is necessary)
clone the Gen2Support repo (here)
change the target framework of the Gen2Support project to net5.0
build Gen2Support
Copy the build output to $HOME/.local/share/NuGet/CredentialProviders

After doing this, Paket correctly installed a dependency from my private feed 🙂

FedericoBinda commented 3 years ago

Adding to my previous comment: I did the same on Windows and it worked. The only difference was the path in step 6: ~\AppData\Local\NuGet\CredentialProviders

mfschumann commented 2 years ago

@FedericoBinda: Would you mind sharing the command and dotnet sdk version you used for building Gen2Support? I am asking because running dotnet build with dotnet sdk 5.0.401 always yields an error on my Windows machine: Paket.CredentialProvider.Gen2Support\Program.fs(22,9): error FS0039: The value, namespace, type or module 'PluginManager' is not defined. Maybe you want one of the following: PluginResource PluginCredentialRequest [C:\Users\y1mschum\Documents\MATLAB\Paket.CredentialProvider.Gen2Support\Paket.CredentialProvider.Gen2Support\Paket.CredentialProvider.Gen2Support.fsproj]

FedericoBinda commented 2 years ago

@mfschumann I just checked: They have made an update on the repo to target net5.0. Along with that update, they also updated the package references:

<ItemGroup>
      <PackageReference Include="NuGet.Credentials" Version="5.11.0" />
      <PackageReference Include="NuGet.Protocol" Version="5.11.0" />
</ItemGroup>

In the version I used some months ago, the references were:

<ItemGroup>
      <PackageReference Include="NuGet.Credentials" Version="5.0.0" />
      <PackageReference Include="NuGet.Protocol" Version="5.0.0" />
</ItemGroup>

If I change back to the old versions it works for me. I cannot explain why.

mfschumann commented 2 years ago

@FedericoBinda: Thank you so much for the quick reply! Reverting to version 5.0.0 of the references fixed the build issue for me too. I can run the generated CredentialProvider.Gen2Support.dll using dotnet directly, but somehow paket does not seem to pick up the credential provider for authenticating with my private Azure feed. When trying to paket install I keep getting Could not load resources from ...: Unauthorized (401). Is there anything I have to do to make paket aware of the credential provider? I did copy all the contents of Paket.CredentialProvider.Gen2Support\bin\Debug\net5.0 to %LOCALAPPDATA%\NuGet\CredentialProviders\Paket.CredentialProvider.Gen2Support. Is that the correct location?

FedericoBinda commented 2 years ago

Yes, that's the location where I have it. I have to mention that I saw the same behaviour some time ago, but then it started working again, and I did not understand what had changed (!!). Maybe the Paket version? I am using 6.2.1. This is all very confusing to me as well, I am no expert in how Paket works, I'm just a user.

mfschumann commented 2 years ago

I am using Paket 6.2.1 too. Can you tell what commit you used to build Gen2Support? Maybe the current version that I tried does not work properly for some reason?

FedericoBinda commented 2 years ago

The commit I have is 72ad0da. But looking at the history of the commits after that one, it doesn't look like they have done any significant changes, only documentation and upgrade to .NET 5.

FedericoBinda commented 2 years ago

Actually, 3 days ago they checked in with commit message "fix build".

slang25 commented 2 years ago

Yeah, sorry all, I saw the comments in here and had realised that I broken it with my previous changes

slang25 commented 2 years ago

The latest version of that repo should build. However I have to admit that I no longer use this tool or Paket, so probably shouldn't be maintaining it. If I get some time, I'd like to set up a personal Azure DevOps and properly test it (maybe even get it working on Windows and macOS in CI)

mfschumann commented 2 years ago

@slang25 @FedericoBinda: Thanks again for helping me out. In the meantime I decided to go with dotnet restore instead of paket.exe because it has all the features I need and authentification with Azure Artifacts works flawlessly.

geysernrd commented 2 years ago

I can confirm that the process outlined above by @FedericoBinda is working as written (at least in my environment: Windows 10, VS Code). I first got paket working against my Azure Artifacts repository using the same process this summer, and then it just worked again yesterday after my hard drive died. Thanks so much for the step-by-step explanation! I never would have figured this out on my own.

mfschumann commented 2 years ago

I gave paket another shot, but I still was not able to get authentication to work with my private Azure feed: I was able to compile the latest version of Gen2Support and installed it to %LOCALAPPDATA%\NuGet\CredentialProviders\Paket.CredentialProvider.Gen2Support.

When I run

CredentialProvider.Gen2Support.exe -uri https://pkgs.dev.azure.com/[my-organization]/[my-project]/_packaging/[my-feed]/nuget/v3/index.json

the provider returns valid credentials:

{ "Username" : "VssSessionToken",
"Password" : "[redacted]",
"Message" : "" }

However if I try e.g. paket install I keep getting

Could not load resources from 'https://pkgs.dev.azure.com/[my-organization]/[my-project]/_packaging/[my-feed]/nuget/v3/index.json': Unauthorized (401)

So it seems that paket does not use the Gen2Support credential provider.

@FedericoBinda @slang25 : Do you see any way for debugging this?

dg-eparizzi commented 2 years ago

Until paket migrate from nuget command works with Azure DevOps feeds it's going to be very hard for people to adopt this tool. Any updates on this issue?

Flohack74 commented 2 years ago

We are also using the older CredentialProvider.VSS.exe (probably obtained with this compilation method mentioned above) and this worked well for years, however now it seems that it starts failing randomly in CI machines, and also since today with user PCs.

I would ask to focus here on this topic since AZure DevOps got more and more important in today´s .NET business. If MS really dropped the old plugin way of things, this needs to be followed up also in Paket.

Flohack74 commented 2 years ago

Additionally the solution with proxying V1 requests to the official V2 provider is very cumbersome, we have to use device token protocol which means manual refresh every time those tokens expire

gregorriegler commented 2 months ago

I would also need this. Any update?

pb-413 commented 1 month ago

For future reference, I tried the steps outlined by @FedericoBinda above, but they did not seem to work for me at first.

I got an error that the JSON format was expected:

JsonSerializationException: Cannot deserialize the current JSON array (e.g. [1,2,3]) into type
'Paket.CredentialProviderResultMessage' because the type requires a JSON object (e.g. {"name":"value"}) to deserialize correctly.
To fix this error either change the JSON to a JSON object (e.g. {"name":"value"}) or change the deserialized type to an array
or a type that implements a collection interface (e.g. ICollection, IList) like List<T> that can be deserialized from a JSON array.
JsonArrayAttribute can also be added to the type to force it to deserialize from a JSON array.

For some reason, I had CredentialProvider.Microsoft as a sibling of Paket.CredentialProvider.Gen2Support in AppData\Local\NuGet\CredentialProviders. After removing CredentialProvider.Microsoft, the work around steps appeared to work for me!

(Note that CredentialProvider.Microsoft was present in .nuget\plugins\netcore per the steps)

fsprojects / Paket