System.Security.Cryptography.CryptographicException on remote connections

[junalmeida]

Steps to reproduce:

1. Spin up a Windows Server Core
2. Install openssh server
3. Connect via ssh
4. Run powershell.exe
5. Set organization, project and feed variables of a private azure devops project
6. Run ~\.nuget\plugins\netfx\CredentialProvider.Microsoft\CredentialProvider.Microsoft.exe -I -V Verbose -U "https://pkgs.dev.azure.com/$organization/$project/_packaging/$feed/nuget/v3/index.json"

Actual:

Unhandled Exception: System.Security.Cryptography.CryptographicException: Access is denied.

   at System.Security.Cryptography.ProtectedData.Protect(Byte[] userData, Byte[] optionalEntropy, DataProtectionScope scope)
   at NuGetCredentialProvider.Util.EncryptedFile.WriteFileBytes(String filePath, Byte[] bytes, Boolean writeUnencrypted) in D:\a\_work\1\s\CredentialProvider.Microsoft\Util\EncryptedFile.cs:line 36
   at NuGetCredentialProvider.Util.SessionTokenCache.Remove(Uri key) in D:\a\_work\1\s\CredentialProvider.Microsoft\Util\SessionTokenCache.cs:line 194
   at NuGetCredentialProvider.RequestHandlers.GetAuthenticationCredentialsRequestHandler.TryCache(GetAuthenticationCredentialsRequest request, String& cachedToken) in D:\a\_work\1\s\CredentialProvider.Microsoft\RequestHandlers\GetAuthenticationCredentialsRequestHandler.cs:line 147
   at NuGetCredentialProvider.RequestHandlers.GetAuthenticationCredentialsRequestHandler.<HandleRequestAsync>d__5.MoveNext() in D:\a\_work\1\s\CredentialProvider.Microsoft\RequestHandlers\GetAuthenticationCredentialsRequestHandler.cs:line 70
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NuGetCredentialProvider.Program.<Main>d__3.MoveNext() in D:\a\_work\1\s\CredentialProvider.Microsoft\Program.cs:line 173
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NuGetCredentialProvider.Program.<Main>(String[] args)

Expected: Aquire a session token and connect successfuly

[embetten]

Looks like it doesn't have write access to the cache location? You can test this theory by turning off the cache to see if that works. you can turn off the cache by setting the NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED environment variable to false.

Might be related to #448 and the workarounds there might help.

[junalmeida]

Using the workaround suggested at #448 didn't fixed the issue, however it did shown a different error message:

Unhandled Exception: System.Security.Cryptography.CryptographicException: Key not valid for use in specified state.

   at System.Security.Cryptography.ProtectedData.Unprotect(Byte[] encryptedData, Byte[] optionalEntropy, DataProtectionScope scope)
   at NuGetCredentialProvider.Util.EncryptedFile.ReadFileBytes(String filePath, Boolean readUnencrypted) in D:\a\_work\1\s\CredentialProvider.Microsoft\Util\EncryptedFile.cs:line 17

Setting the env variable suggested to false is also throwing different error messages:

[Verbose] [CredentialProvider]Invalidating SessionToken cache for https://pkgs.dev.azure.com/......./nuget/v3/index.json
[Verbose] [CredentialProvider]GET https://pkgs.dev.azure.com/...../nuget/v3/index.json
[Verbose] [CredentialProvider]Found AAD Authority from 401 headers: https://login.windows.net/cb543e80-00d2-4aa9-b2dd-c39de3e90501
[Verbose] [CredentialProvider]VstsCredentialProvider - Using AAD authority: https://login.windows.net/cb543e80-00d2-4aa9-b2dd-c39de3e90501
[Warning] [CredentialProvider]Warning: Cannot persist Microsoft authentication token cache securely!
[Verbose] [CredentialProvider]VstsCredentialProvider - Attempting to acquire bearer token using provider 'MSAL Silent'
[Warning] [CredentialProvider]Unknown Status: Unexpected
Error: 0xffffffff80040154
Context: Winrt exception was thrown during GetTokenSilently '(pii)'.
Tag: 0x2339e502 (error code -2147221164) (internal error code 590996738)
[Verbose] [CredentialProvider]VstsCredentialProvider - Bearer token provider 'MSAL Silent' failed with exception:\nSystem.NullReferenceException: Object reference not set to an instance of an object.
   at NuGetCredentialProvider.CredentialProviders.Vsts.VstsCredentialProvider.<HandleRequestAsync>d__8.MoveNext() in D:\a\_work\1\s\CredentialProvider.Microsoft\CredentialProviders\Vsts\VstsCredentialProvider.cs:line 138[Verbose] [CredentialProvider]VstsCredentialProvider - Attempting to acquire bearer token using provider 'MSAL Windows Integrated Authentication'
[Verbose] [CredentialProvider]VstsCredentialProvider - Bearer token provider 'MSAL Windows Integrated Authentication' failed with exception:\nSystem.NullReferenceException: Object reference not set to an instance of an object.
   at NuGetCredentialProvider.CredentialProviders.Vsts.VstsCredentialProvider.<HandleRequestAsync>d__8.MoveNext() in D:\a\_work\1\s\CredentialProvider.Microsoft\CredentialProviders\Vsts\VstsCredentialProvider.cs:line 138[Verbose] [CredentialProvider]VstsCredentialProvider - Attempting to acquire bearer token using provider 'MSAL Interactive'
[Verbose] [CredentialProvider]VstsCredentialProvider - Bearer token provider 'MSAL Interactive' failed with exception:\nSystem.NullReferenceException: Object reference not set to an instance of an object.
   at NuGetCredentialProvider.CredentialProviders.Vsts.VstsCredentialProvider.<HandleRequestAsync>d__8.MoveNext() in D:\a\_work\1\s\CredentialProvider.Microsoft\CredentialProviders\Vsts\VstsCredentialProvider.cs:line 138[Verbose] [CredentialProvider]VstsCredentialProvider - Attempting to acquire bearer token using provider 'MSAL Device Code'
[Minimal] [CredentialProvider]DeviceFlow: https://pkgs.dev.azure.com/....../nuget/v3/index.json
[Minimal] [CredentialProvider]ATTENTION: User interaction required.

    **********************************************************************

    To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code AQG76XBXF to authenticate.

    **********************************************************************

[Information] [CredentialProvider]VstsCredentialProvider - Acquired bearer token using 'MSAL Device Code'
[Information] [CredentialProvider]VstsCredentialProvider - Attempting to exchange the bearer token for an Azure DevOps session token.
[Verbose] [CredentialProvider]Requesting a Compact token valid for duration 90.00:00:00, valid until 3/17/2024 7:55:42 PM UTC. Note that the generated token may have different validity than requested.
[Verbose] [CredentialProvider]Response: OK
[Verbose] [CredentialProvider] ActivityId: 9b4fa8aa-9b27-41c0-879e-4a0e9768eb55
[Verbose] [CredentialProvider]VstsCredentialProvider - Created SessionToken for https://pkgs.dev.azure.com/...../nuget/v3/index.json
[Verbose] [CredentialProvider]Caching SessionToken for https://pkgs.dev.azure.com/......./nuget/v3/index.json
[Information] [CredentialProvider]Username: VssSessionToken
[Information] [CredentialProvider]Password:......

(urls redacted)

[embetten]

@junalmeida I just created a new 1.1.0-beta pre-release that will let the cred provider return acquired tokens, even if we are not able to cache it by improving our error handling on writes to the cache. If there is a MSAL token cached in the MSAL cache, this fix should unblock the silent auth flow in this scenario. This will not fix the underlying cryptographic exception - as I believe that is an access issue to the windows cryptographic key.

Let me know if you are able to download the 1.1.0-beta release and confirm this fixes your issue. Thanks!

[junalmeida]

@embetten Hi! Thanks for looking at this. I just tried the net 6 version you provided. I first removed all plugin under .nuget\plugins, then I add there the netcore folder from the zip. This is the result trying a nuget restore:

WARNING: NU1608: Detected package version outside of dependency constraint: Microsoft.NET.Sdk.Functions 3.0.13 requires Microsoft.Azure.WebJobs.Extensions.Http (>= 3.0.2 && < 3.1.0) but version Microsoft.Azure.WebJobs.Extensions.Http 3.2.0 was resolved.
WARNING:     [CredentialProvider]Warning: Cannot persist Microsoft authentication token cache securely!
WARNING:     [CredentialProvider]Unknown Status: Unexpected
Error: 0xffffffff80040154
Context: Winrt exception was thrown during GetTokenSilently '(pii)'.
Tag: 0x2339e502 (error code -2147221164) (internal error code 590996738)
    [CredentialProvider]DeviceFlow: https://pkgs.dev.azure.com/REDACTED/nuget/v3/index.json
    [CredentialProvider]ATTENTION: User interaction required. 

    **********************************************************************

    To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code E3REDACTED to authenticate.

    **********************************************************************

    [CredentialProvider]VstsCredentialProvider - Acquired bearer token using 'MSAL Device Code'
    [CredentialProvider]VstsCredentialProvider - Attempting to exchange the bearer token for an Azure DevOps session token.
Please provide credentials for: https://pkgs.dev.azure.com/REDACTED/nuget/v3/index.json
UserName:     [CredentialProvider]Failed to acquire session token: System.Security.Cryptography.CryptographicException: Access is denied.

   at System.Security.Cryptography.ProtectedData.Protect(Byte[] userData, Byte[] optionalEntropy, DataProtectionScope scope)
   at NuGetCredentialProvider.Util.EncryptedFile.WriteFileBytes(String filePath, Byte[] bytes, Boolean writeUnencrypted)
   at NuGetCredentialProvider.Util.SessionTokenCache.set_Item(Uri key, String value)
   at NuGetCredentialProvider.RequestHandlers.GetAuthenticationCredentialsRequestHandler.<HandleRequestAsync>d__5.MoveNext()

And then it hangs. If I type anything I get a "Password: " prompt and then I have to "Ctrl+c"

[embetten]

@junalmeida first of all, thank you for downloading and running this!

Looking at the error log stack trace, it seems as though the 1.1.0-beta is not being used since NuGetCredentialProvider.Util.EncryptedFile has now been renamed in this version to NuGetCredentialProvider.Util.EncryptedFileWithPermissions. Are you running your nuget restore via the nuget.exe or through dotnet? Depending on how you are running nuget, you may need the netfx version as well as the netcore folder. See the nuget plugin reference for more info.

Can you try to install the 1.1.0-beta version again? Some troubleshooting steps:

1. Double Check our manual install steps.
2. The install script supports passing in version and adding netfx, so you could run this command: iex "& { $(irm https://aka.ms/install-artifacts-credprovider.ps1)} -Version 1.1.0-beta -Force -AddNetfx"
3. Check to see if there are any custom plugin paths set up: https://learn.microsoft.com/en-us/nuget/reference/extensibility/nuget-cross-platform-plugins#plugin-installation-and-discovery as an env varaible on your machine. If so, make sure to update those paths as well.
4. If any further issues are found - you could also try running the .dll in the zip file directly in stand-alone mode.

[junalmeida]

@embetten I was indeed suspecting I was running an old version somehow (even though I have deleted the plugins folder). Thanks for the tip on looking at the stack trace (but I believe it would be useful to see a version info logged to the console?)

I will try again and get back to you!

[junalmeida]

@embetten I just reinstalled beta version, and calling nuget restore on a legacy NetFramework solution, this is what I get:

WARNING:     [CredentialProvider]Warning: Cannot persist Microsoft authentication token cache securely!
WARNING:     [CredentialProvider]Unknown Status: Unexpected
Error: 0xffffffff80040154
Context: Winrt exception was thrown during GetTokenSilently '(pii)'.
Tag: 0x2339e502 (error code -2147221164) (internal error code 590996738)
    [CredentialProvider]DeviceFlow: https://pkgs.dev.azure.com/REDACTED/nuget/v3/index.json
    [CredentialProvider]ATTENTION: User interaction required.

    **********************************************************************

    To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code REDACTED to authenticate.

    **********************************************************************

    [CredentialProvider]VstsCredentialProvider - Acquired bearer token using 'MSAL Device Code'
    [CredentialProvider]VstsCredentialProvider - Attempting to exchange the bearer token for an Azure DevOps session token.

Aside from the warning and errors, it is now proceeding and restoring packages, so I guess it is using the new version I just installed because this behavior is what we expected, right?

[embetten]

Yes, this looks as expected for the fix. Thank you for re-running!

The underlying issue with the cred provider not being able to create/access the MSAL or ADO session token caches still persists. Until that is figured out, silent auth will not work and you will need to authenticate with device code flow each time. Knowing if the cred provider is the only application on the machine not able to access the MSAL cache would allow us to narrow down if this is a cred provider specific issue, a permission issue like we referenced in #448, or missing software on a windows server core machine.

[embetten]

Closing. Able to confirm device code flow is unblocked. We have not been able to reproduce the session token write issue.