Credential Provider Fails with Sign-in frequency Conditional Access Policy

[afscrome]

If your organization has a conditional access policy with a short Sign-in frequency (in my case 10 hours), then the Azure Devops credential provider fails with a 403 error if you haven't logged in to the Azure Devops UI in the time since your token was issued.

[Verbose] [CredentialProvider]VstsCredentialProvider - Exception trying to generate Azure DevOps token: https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json, message: Response status code does not indicate success: 403 (The requested operation is not allowed.)., stack: at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at NuGetCredentialProvider.CredentialProviders.Vsts.VstsSessionTokenClient.CreateSessionTokenAsync(VstsTokenType tokenType, DateTime validTo, CancellationToken cancellationToken) in D:\a_work\1\s\CredentialProvider.Microsoft\CredentialProviders\Vsts\VstsSessionTokenClient.cs:line 101

The exception comes from https://github.com/microsoft/artifacts-credprovider/blob/b66855cc5cb87b5a1c6e925ab84a532ec9c72e8b/CredentialProvider.Microsoft/CredentialProviders/Vsts/VstsCredentialProvider.cs#L161 which makes a call to https://vssps.dev.azure.com/REDACTED/_apis/Token/SessionTokens?tokenType=SelfDescribing which fails with the response

{
    "$id": "1",
    "innerException": null,
    "message": "VS403463: The conditional access policy defined by your Microsoft Entra administrator has failed.",
    "typeName": "Microsoft.VisualStudio.Services.Common.VssServiceException, Microsoft.VisualStudio.Services.Common",
    "typeKey": "VssServiceException",
    "errorCode": 0,
    "eventId": 3000
}

The problem seems to be that whilst the Entra token was obtained interactively, tokenProvider.IsInteractive is false, so the call to the SessionTokens api doesn't extend the session lifetime in Azure Devops. This causes some nasty knock on effects in Visual Studio which gets into an infinite loop of prompting to auth to the credential provider without showing any error.

Since Azure Devops supports Entra auth natively, you could skip this whole problem by using the already obtained bearer token by replacing the above line with something like:

//TODO: Pull deploymentType check out of loop so it's only done once
var deploymentType = await authUtil.GetAzDevDeploymentType(request.Uri);

sessionToken = deploymentType == AzDevDeploymentType.Hosted
    ? bearerToken
    : await vstsSessionTokenProvider.GetAzureDevOpsSessionTokenFromBearerToken(request, bearerToken, tokenProvider.IsInteractive, cancellationToken);

It would also be nice if a 403 error would show more details of the error from the response body - I had to pull the source code and crack open the debugger to find out that conditional access policies were the cause of this error and not the verbose output.

[embetten]

@afscrome can you provide more information about how the cred provider is being called - with nuget.exe, msbuild or dotnet cli? and what MSAL login method was used if it was interactive? what is the exit code (This might be in the nuget logs )? I would have expected NuGet to retry and send isRetry=true, which I would expect to grab fresh tokens regardless of session token cache. We have known work item to improve the cache to check if the token is near expiration, and refresh if so.

Also, this would be really helpful to know if this is experienced outside of visual studio. There are multiple NuGet credproviders shipped with VS and isolating which one, or if it's the interaction of both + nuget, would help narrow down the issue.

There is a known issue for the visual studio credprovider around re-authenticating accounts with certain conditional access policies for folks with multiple accounts in the keychain (where the work around requires reauthenticating in the visual studio keychain before trying to access a feed).

[afscrome]

We experience this issue with both Visual Studio and dotnet CLI. In Visual Studio it shows itself up as an infinite loop of prompting to log in to the artifact feed. In dotnet CLI it shows up as broker prompt which 403s followed by a device code requests which also produces a 403 exception , even though I completed both the device code and broker logins successfully. Dotnet cli seems to provide a loop of Device Code prompts, although it may actually be one per project (I haven't had the patience to keep doing the prompts to see if they stop)

I can also reproduce this raw with the credential provider - both with 1.1 and 1.3.

& $env:userprofile\.nuget\plugins\netcore\CredentialProvider.Microsoft\CredentialProvider.Microsoft.exe -I -V Verbose -U "https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json"

This first prompts me with a Broker request

[Verbose] [CredentialProvider]Running in stand-alone mode [Verbose] [CredentialProvider]Command-line v1.3.0-alpha+b66855cc5cb87b5a1c6e925ab84a532ec9c72e8b: C:\Users\alexanderc.nuget\plugins\netcore\CredentialProvider.Microsoft\CredentialProvider.Microsoft.dll -I -V Verbose -U https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Verbose] [CredentialProvider]Handling auth request, Uri: https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json, IsRetry: True, IsNonInteractive: False, CanShowDialog: True [Verbose] [CredentialProvider]URI: https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Verbose] [CredentialProvider]VstsBuildTaskServiceEndpointCredentialProvider - This credential provider must be run under the Team Build tasks for NuGet with external endpoint credentials. Appropriate environment variable needs to be set. [Verbose] [CredentialProvider]Skipping NuGetCredentialProvider.CredentialProviders.VstsBuildTaskServiceEndpoint.VstsBuildTaskServiceEndpointCredentialProvider, cannot provide credentials for https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Verbose] [CredentialProvider]VstsBuildTaskCredentialProvider - This credential provider must be run under the Team Build tasks for NuGet. Appropriate environment variables must be set. [Verbose] [CredentialProvider]Skipping NuGetCredentialProvider.CredentialProviders.VstsBuildTask.VstsBuildTaskCredentialProvider, cannot provide credentials for https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Verbose] [CredentialProvider]VstsCredentialProvider - Matched well-known Azure DevOps Service hostname: pkgs.dev.azure.com [Verbose] [CredentialProvider]Using NuGetCredentialProvider.CredentialProviders.Vsts.VstsCredentialProvider to try to get credentials for https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json. [Verbose] [CredentialProvider]IsRetry: True [Verbose] [CredentialProvider]Invalidating SessionToken cache for https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Verbose] [CredentialProvider]GET https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Verbose] [CredentialProvider]Found AAD Authority from 401 headers: https://login.windows.net/REDACTED [Verbose] [CredentialProvider]VstsCredentialProvider - Using Entra authority: https://login.windows.net/REDACTED [Verbose] [CredentialProvider]VstsCredentialProvider - Not running bearer token provider 'MSAL Service Principal' [Verbose] [CredentialProvider]VstsCredentialProvider - Not running bearer token provider 'MSAL Managed Identity' [Verbose] [CredentialProvider]VstsCredentialProvider - Attempting to acquire bearer token using provider 'MSAL Silent' [Information] [CredentialProvider]VstsCredentialProvider - Acquired bearer token using 'MSAL Silent' [Information] [CredentialProvider]VstsCredentialProvider - Attempting to exchange the bearer token for an Azure DevOps session token. [Verbose] [CredentialProvider]Requesting a SelfDescribing token valid for duration 04:00:00, valid until 31/10/2024 14:47:11 UTC. Note that the generated token may have different validity than requested. [Verbose] [CredentialProvider]Response: Forbidden [Verbose] [CredentialProvider] ActivityId: 770f3d01-f1bc-4252-871b-ea9240cd1f84 [Verbose] [CredentialProvider]VstsCredentialProvider - Exception trying to generate Azure DevOps token: https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json, message: Response status code does not indicate success: 403 (The requested operation is not allowed.)., stack: at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at NuGetCredentialProvider.CredentialProviders.Vsts.VstsSessionTokenClient.CreateSessionTokenAsync(VstsTokenType tokenType, DateTime validTo, CancellationToken cancellationToken) in D:\a_work\1\s\CredentialProvider.Microsoft\CredentialProviders\Vsts\VstsSessionTokenClient.cs:line 101 at NuGetCredentialProvider.CredentialProviders.Vsts.VstsSessionTokenFromBearerTokenProvider.GetAzureDevOpsSessionTokenFromBearerToken(GetAuthenticationCredentialsRequest request, String bearerToken, Boolean bearerTokenObtainedInteractively, CancellationToken cancellationToken) in D:\a_work\1\s\CredentialProvider.Microsoft\CredentialProviders\Vsts\VstsSessionTokenFromBearerTokenProvider.cs:line 61 at NuGetCredentialProvider.CredentialProviders.Vsts.VstsCredentialProvider.HandleRequestAsync(GetAuthenticationCredentialsRequest request, CancellationToken cancellationToken) in D:\a_work\1\s\CredentialProvider.Microsoft\CredentialProviders\Vsts\VstsCredentialProvider.cs:line 161 [Verbose] [CredentialProvider]VstsCredentialProvider - Attempting to acquire bearer token using provider 'MSAL Windows Integrated Authentication' [Verbose] [CredentialProvider]VstsCredentialProvider - Bearer token provider 'MSAL Windows Integrated Authentication' didn't acquire a token [Verbose] [CredentialProvider]VstsCredentialProvider - Attempting to acquire bearer token using provider 'MSAL Interactive' [Information] [CredentialProvider]VstsCredentialProvider - Acquired bearer token using 'MSAL Interactive' [Information] [CredentialProvider]VstsCredentialProvider - Attempting to exchange the bearer token for an Azure DevOps session token. [Verbose] [CredentialProvider]Requesting a Compact token valid for duration 90.00:00:00, valid until 29/01/2025 10:47:16 UTC. Note that the generated token may have different validity than requested. [Verbose] [CredentialProvider]Response: Forbidden [Verbose] [CredentialProvider] ActivityId: dc724638-7615-4180-9b11-421924f58078 [Verbose] [CredentialProvider]VstsCredentialProvider - Exception trying to generate Azure DevOps token: https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json, message: Response status code does not indicate success: 403 (The requested operation is not allowed.)., stack: at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at NuGetCredentialProvider.CredentialProviders.Vsts.VstsSessionTokenClient.CreateSessionTokenAsync(VstsTokenType tokenType, DateTime validTo, CancellationToken cancellationToken) in D:\a_work\1\s\CredentialProvider.Microsoft\CredentialProviders\Vsts\VstsSessionTokenClient.cs:line 101 at NuGetCredentialProvider.CredentialProviders.Vsts.VstsSessionTokenFromBearerTokenProvider.GetAzureDevOpsSessionTokenFromBearerToken(GetAuthenticationCredentialsRequest request, String bearerToken, Boolean bearerTokenObtainedInteractively, CancellationToken cancellationToken) in D:\a_work\1\s\CredentialProvider.Microsoft\CredentialProviders\Vsts\VstsSessionTokenFromBearerTokenProvider.cs:line 61 at NuGetCredentialProvider.CredentialProviders.Vsts.VstsCredentialProvider.HandleRequestAsync(GetAuthenticationCredentialsRequest request, CancellationToken cancellationToken) in D:\a_work\1\s\CredentialProvider.Microsoft\CredentialProviders\Vsts\VstsCredentialProvider.cs:line 161

Which fails, and then prompts for Device Code

[Verbose] [CredentialProvider]VstsCredentialProvider - Attempting to acquire bearer token using provider 'MSAL Device Code' [Minimal] [CredentialProvider]DeviceFlow: https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Minimal] [CredentialProvider]ATTENTION: User interaction required.

---

To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code REDACTED to authenticate.

---

[Information] [CredentialProvider]VstsCredentialProvider - Acquired bearer token using 'MSAL Device Code' [Information] [CredentialProvider]VstsCredentialProvider - Attempting to exchange the bearer token for an Azure DevOps session token. [Verbose] [CredentialProvider]Requesting a Compact token valid for duration 90.00:00:00, valid until 29/01/2025 10:47:47 UTC. Note that the generated token may have different validity than requested. [Verbose] [CredentialProvider]Response: Forbidden [Verbose] [CredentialProvider] ActivityId: 770f2584-f1bc-4252-871b-ea9240cd1f84 [Verbose] [CredentialProvider]VstsCredentialProvider - Exception trying to generate Azure DevOps token: https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json, message: Response status code does not indicate success: 403 (The requested operation is not allowed.)., stack: at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at NuGetCredentialProvider.CredentialProviders.Vsts.VstsSessionTokenClient.CreateSessionTokenAsync(VstsTokenType tokenType, DateTime validTo, CancellationToken cancellationToken) in D:\a_work\1\s\CredentialProvider.Microsoft\CredentialProviders\Vsts\VstsSessionTokenClient.cs:line 101 at NuGetCredentialProvider.CredentialProviders.Vsts.VstsSessionTokenFromBearerTokenProvider.GetAzureDevOpsSessionTokenFromBearerToken(GetAuthenticationCredentialsRequest request, String bearerToken, Boolean bearerTokenObtainedInteractively, CancellationToken cancellationToken) in D:\a_work\1\s\CredentialProvider.Microsoft\CredentialProviders\Vsts\VstsSessionTokenFromBearerTokenProvider.cs:line 61 at NuGetCredentialProvider.CredentialProviders.Vsts.VstsCredentialProvider.HandleRequestAsync(GetAuthenticationCredentialsRequest request, CancellationToken cancellationToken) in D:\a_work\1\s\CredentialProvider.Microsoft\CredentialProviders\Vsts\VstsCredentialProvider.cs:line 161 [Verbose] [CredentialProvider]VstsCredentialProvider - Could not obtain credentials for https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Verbose] [CredentialProvider]Unable to acquire credentials.

If I do something in the Azure Devops UI in a web browser then repeat the same request to the credential provider exe, everything works perfectly.

[Verbose] [CredentialProvider]Running in stand-alone mode [Verbose] [CredentialProvider]Command-line v1.3.0-alpha+b66855cc5cb87b5a1c6e925ab84a532ec9c72e8b: C:\Users\alexanderc.nuget\plugins\netcore\CredentialProvider.Microsoft\CredentialProvider.Microsoft.dll -I -V Verbose -U https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Verbose] [CredentialProvider]Handling auth request, Uri: https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json, IsRetry: True, IsNonInteractive: False, CanShowDialog: True [Verbose] [CredentialProvider]URI: https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Verbose] [CredentialProvider]VstsBuildTaskServiceEndpointCredentialProvider - This credential provider must be run under the Team Build tasks for NuGet with external endpoint credentials. Appropriate environment variable needs to be set. [Verbose] [CredentialProvider]Skipping NuGetCredentialProvider.CredentialProviders.VstsBuildTaskServiceEndpoint.VstsBuildTaskServiceEndpointCredentialProvider, cannot provide credentials for https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Verbose] [CredentialProvider]VstsBuildTaskCredentialProvider - This credential provider must be run under the Team Build tasks for NuGet. Appropriate environment variables must be set. [Verbose] [CredentialProvider]Skipping NuGetCredentialProvider.CredentialProviders.VstsBuildTask.VstsBuildTaskCredentialProvider, cannot provide credentials for https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Verbose] [CredentialProvider]VstsCredentialProvider - Matched well-known Azure DevOps Service hostname: pkgs.dev.azure.com [Verbose] [CredentialProvider]Using NuGetCredentialProvider.CredentialProviders.Vsts.VstsCredentialProvider to try to get credentials for https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json. [Verbose] [CredentialProvider]IsRetry: True [Verbose] [CredentialProvider]Invalidating SessionToken cache for https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Verbose] [CredentialProvider]GET https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Verbose] [CredentialProvider]Found AAD Authority from 401 headers: https://login.windows.net/REDACTED [Verbose] [CredentialProvider]VstsCredentialProvider - Using Entra authority: https://login.windows.net/REDACTED [Verbose] [CredentialProvider]VstsCredentialProvider - Not running bearer token provider 'MSAL Service Principal' [Verbose] [CredentialProvider]VstsCredentialProvider - Not running bearer token provider 'MSAL Managed Identity' [Verbose] [CredentialProvider]VstsCredentialProvider - Attempting to acquire bearer token using provider 'MSAL Silent' [Information] [CredentialProvider]VstsCredentialProvider - Acquired bearer token using 'MSAL Silent' [Information] [CredentialProvider]VstsCredentialProvider - Attempting to exchange the bearer token for an Azure DevOps session token. [Verbose] [CredentialProvider]Requesting a SelfDescribing token valid for duration 04:00:00, valid until 31/10/2024 15:21:29 UTC. Note that the generated token may have different validity than requested. [Verbose] [CredentialProvider]Response: OK [Verbose] [CredentialProvider] ActivityId: 8a8c371a-66bc-4f1b-897e-fe00dd8eb1e8 [Verbose] [CredentialProvider]VstsCredentialProvider - Created SessionToken for https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Verbose] [CredentialProvider]Caching SessionToken for https://pkgs.dev.azure.com/REDACTED/_packaging/REDACTED/nuget/v3/index.json [Information] [CredentialProvider]Username: VssSessionToken [Information] [CredentialProvider]Password: REDACTED

If I look at the sign in events in the Entra Sign in logs, I see a few errors. In Interactive sign ins, I see For security reasons, user confirmation is required for this request. Please repeat the request allowing user interaction.. Under non-interactive I see The refresh token has expired or is invalid due to sign-in frequency checks by conditional access. The token was issued on {issueDate} and the maximum allowed lifetime for this request is {time}. for Primary Refresh Token requests, and The session is not valid due the following reasons: password expiration or recent password change, SSO Artifact is invalid or expired, session is not fresh enough for application, or a silent sign-in request was sent but the user's session with Azure AD is invalid or has expired. for Device Code Flow. I can share more details from the Entra sign in logs if desired.

For what it's worth, we experience a very similar issue with the git credential provider - if you haven't visited the ADO UI withint eh past 10 hours (very common at the begining of the day), the git credential provider will continually 403 until you open ADO in your UI.

git pull fatal: unable to access 'https://dev.azure.com/REDACTED/REDACTED/_git/REDACTED/': The requested URL returned error: 403

Although we're doing some testing as it looks like we're able to mitigate that issue by forcing the git credential provider to use oauth over PAT tokens, and to use the windows account broker with the following in .gitconfig

[credential]
        azreposCredentialType = oauth
        msauthUseBroker = true

Unfortunately this doesn't work for all our users as we have some applications which require running Visual Studio as admin. My company has separate admin & non admin accounts, and the account broker doesn't work when running as another user.

[embetten]

First and foremost, I recommend turning off the session token cache for your org with these conditional access policies. Setting the env variable NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED to false will help prevent multiple nuget calls and retries.

The issue is that an on-behalf-of call on the ADO server side is failing the conditional access policy. ADO doesn't expose the necessary claims for re-authentication to the downstream resource (see these docs ). However, logging into the ADO UI refreshes the token for the on-behalf-of call and satisfies the downstream claims. This is also why it's needed for GCM.

TL;DR Unfortunately, these kinds of conditional access policies will require users to log on to the ADO UI before using this credential provider. That is, until ADO is able to implement the changes to expose the needed information to the clients and we are able to update the credential provider to use them. Turning off the session token cache will potentially help avoid multiple roundtrips after going to the ADO UI.