reading secrets from KeyVault with Microsoft.Azure.Services.AppAuthentication

[thdotnet]

Hi,

I've just did a console app to read a secret from key vault and it took 24 seconds to retrieve the value. Is it normal? Here's the code:

static void Main(string[] args)
        {
            Console.WriteLine(DateTime.Now);
            Console.WriteLine(GetValue().Result);
            Console.WriteLine(DateTime.Now);
            Console.Read();
        }

        static async Task<string> GetValue()
        {
            AzureServiceTokenProvider azureServiceTokenProvider = new AzureServiceTokenProvider();

            var keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));

            var secret = await keyVaultClient.GetSecretAsync("https://{mykeyvault}.vault.azure.net/secrets/apiKey")
                .ConfigureAwait(false);

            return secret.Value;
        }

and here's the packages.config

<packages>
  <package id="Microsoft.Azure.KeyVault" version="3.0.0" targetFramework="net462" />
  <package id="Microsoft.Azure.KeyVault.WebKey" version="3.0.0" targetFramework="net462" />
  <package id="Microsoft.Azure.Services.AppAuthentication" version="1.0.3" targetFramework="net462" />
  <package id="Microsoft.IdentityModel.Clients.ActiveDirectory" version="3.14.2" targetFramework="net462" />
  <package id="Microsoft.Rest.ClientRuntime" version="2.3.11" targetFramework="net462" />
  <package id="Microsoft.Rest.ClientRuntime.Azure" version="3.3.12" targetFramework="net462" />
  <package id="Newtonsoft.Json" version="6.0.8" targetFramework="net462" />
</packages>

[varunsh-coder]

This is not normal. This may be because you are using Azure CLI for local development, and that has telemetry enabled. I have seen cases where Azure CLI is slow/ hangs if telemetry is enabled.

Can you please disable telemetry and try? Details are here. I believe it is the collect_telemetry option.

Alternatively, you can use Visual Studio 2017 15.6 or above for local development. AzureServiceTokenProvider uses the signed in account in Visual Studio 2017 for local development. https://blogs.msdn.microsoft.com/visualstudio/2017/11/17/managing-secrets-securely-in-the-cloud/

Also, the first time you get a token, these tools will get the token from Azure AD, and then cache it. Moreover, AzureServiceTokenProvider caches the token till it expires. So, the second time you use token, it will be from memory (until it expires).

[thdotnet]

Hi,

I did another test, but this time disabling the log collection and it took 9 seconds.

[thdotnet]

Also, my VS is 2017 version 15.7.2. Do you think it can be related to the datacenter I've chose? It's running in Brazil South. Please let me know if you need more details, maybe more users are facing the same problems.

[nonik0]

I tried a few ways to try to repro the long access token retrieval times you are seeing and was not able to. I tried accessing key vaults in several different regions, changing various Azure CLI options (including disabling logging and telemetry), as well as signing out and back into Azure CLI. In all of these, I was getting a token in around 2-4 seconds. Can you provide more details for your environment?

[thdotnet]

Hi nonik0, are you a MS employee? Maybe we could address this through MVP list or email directly .

[nonik0]

Hi dhdotnet, I am an MS employee. I can email you directly at your GitHub-listed email address, if that works for you?

[thdotnet]

Yes, please

On Mon, 17 Sep 2018 at 20:20 Nick Brown notifications@github.com wrote:

Hi dhdotnet, I am an MS employee. I can email you directly at your GitHib-listed email address, if that works for you?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Azure/azure-sdk-for-net/issues/4645#issuecomment-422202552, or mute the thread https://github.com/notifications/unsubscribe-auth/ABWP2vASrUDU9L3fPyNyXbbF9Hm2PBxfks5ucC5RgaJpZM4V1y6c .

[MisterReally]

Did you find a way to make it more responsive after all?

[paulhickman-a365]

I'm also having these slow connection times with CLI authentication. My INI file just contains:

[cloud]
name = AzureCloud

Could it be due to the app authentication library always trying the other 3 methods first, and one of those timing out. My laptop is joined to a domain but is connected to a different network with no access to domain controllers when I'm running this code, so maybe the AD authentication attempt has to timeout first?

It would be handy if we could tell App Authentication which of 4 supported methods of obtaining an identity we want it to try so it doesn't waste time on futile attempts.

[ghost]

Hello guys,

I am also having this issue.

I have traced the outgoing HTTP requests from the app that is using AzureServiceTokenProvider with Fiddler and found out where is the delay.

There is a outgoing request

GET /metadata/identity/oauth2/token?resource=https://vault.azure.net&api-version=2018-02-01 HTTP/1.1

to

Host: 169.254.169.254

This is most probably the trial for obtaining MSI metadata in case that the provider is running Azure VM.

This request takes a very long time and always ends up 502. Of course, the target host address is private therefore this behavior might differ in different environments. However, it is causing the delay.

I think that this code is responsible:

https://github.com/Azure/azure-sdk-for-net/blob/53b16a9d7954076f9002013846c4c2840cebf92b/src/SdkCommon/AppAuthentication/Azure.Services.AppAuthentication/TokenProviders/MsiAccessTokenProvider.cs

[ghost]

Quick fix to mitigate the issue on a dev machine is to assign the private IP (169.254.169.254) to the loopback interface and setup an empty website in local IIS binded to 169.254.169.254:80 (http):

cmd: netsh int ip add address "Loopback" 169.254.169.254

then

This way, the IIS returns error immediately to the AzureServiceTokenProvider and the delay is significantly reduced.

[nonik0]

Hi all, thanks for reporting your concerns.

If you would like a quicker token retrieval when developing, you have the option of providing a connection string through the AzureServiceTokenProvider’s constructor, or also by setting an environment variable named AzureServicesAuthConnectionString. Using the connection string, you can choose to authenticate with Azure CLI, Visual Studio, Windows Integrated Auth, or a service principal. Please see the documentation here for details on providing a connection string.

Also, for Azure CLI authentication, you can improve performance by disabling telemetry. Details here.

In our trials, the call in MsiAccessTokenProvider for the local development scenario was returning/failing in well under a second for us. Can you confirm how long this takes for you? Here's the code we used:

`

var start = DateTime.Now;

try
{
    var azureServiceTokenProvider = new AzureServiceTokenProvider("RunAs=App;");
    var token = await azureServiceTokenProvider.GetAccessTokenAsync("https://management.azure.com/");
}
catch
{
}

var timespan = DateTime.Now -  start;
Console.WriteLine($"{timespan.TotalSeconds} seconds");`

[paulhickman-a365]

This test takes 23.8 seconds to fail on my machine when it is connected to a network where there are no domain controllers reachable.

It takes 2.1 seconds to fail when on a network where it can reach a domain controller and 1.7 seconds when disconnected from the network entirely.

[ghost]

This test takes 23.8 seconds to fail on my machine when it is connected to a network where there are no domain controllers reachable.

It takes 2.1 seconds to fail when on a network where it can reach a domain controller and 1.7 seconds when disconnected from the network entirely.

I can confirm the execution time for a network without DCs.

• Authentication via Visual Studio
• Windows 10 1803
• Visual Studio 2017 15.8.5
• Microsoft.Azure.Services.AppAuthentication 1.0.3

[mwanchap]

Both solutions seem to work in my case. Adding the environment variable mentioned by @nonik0 feels a bit less hacky to me and has the same result as hijacking the IP addy. Perhaps it's trying the MsiAccessTokenProvider before trying the CLI/VS methods, so either forcing the first to fail immediately or specifying one of the latter methods avoids the delay? Anyway, thanks so much guys, you've just saved my team a whole lot of wasted local dev time by cutting ~25 seconds from solution startup 👍

[navzam]

I'm also seeing calls to GetAccessTokenAsync() take ~24 seconds when running locally and not specifying a connection string. As above, when I run Fiddler, I can see a request to 169.254.169.254/metadata/identity/oauth2/token which hangs for ~22 seconds before it times out with a 502.

If I set the connection string to one of the local development options, then that request isn't made and the whole call takes ~3 seconds.

Is the solution for me to detect whether I'm running locally and always set a connection string based on that? Or is there a way to speed up the local behavior when the connection string isn't specified?

[mwanchap]

@navzam yeah you can speed it up without setting a connection string, just add a local environment variable to windows called "AzureServicesAuthConnectionString" with a value of "RunAs=Developer; DeveloperTool=VisualStudio"

[nonik0]

Hello all,

We have a new preview release where we've made some updates that should yield improvements to local developer access token retrieval times.

Please do try this new release out and let us know if you see improvement and/or have any issues or feedback!

[navzam]

I just tried with 1.2.0-preview3. Running locally and without specifying a connection string, the call took around 7 seconds, which is a big improvement. It looks like the request to the metadata service endpoint is still happening and timing out after 22 seconds, but I get a token before that.

[holwech]

The preview version has significant faster startup time. There seems to be some delay still, but I would say it's at least usable now.

[paulhickman-a365]

Unfortunately, I'm no longer working on a project where I can test this.

[gatsbys]

Hi all, the preview versions improves the token time acquirment.

Before : ~15-20s Now: 4s

Version:

🥇 Thanks!

[martinoss]

Having the same issue, but updating Microsoft.Azure.Services.AppAuthentication didn't change anything for me. Tried v1.2 and 1.3, still have wait times up to 45s. But setting the environment variable like @navzam described worked for me.

[varunsh-coder]

Closing this issue as fix was released in 1.2.1

[CSM91]

Having the same issue, but updating Microsoft.Azure.Services.AppAuthentication didn't change anything for me. Tried v1.2 and 1.3, still have wait times up to 45s. I've set the environment variable as well and still not working. Can someone help pls?

[nonik0]

@CSM91 sorry to hear you're having issues still--could you share the environment variable you are using for AppAuth, i.e. what authentication method you are trying to use?

[CSM91]

Hello @nonik0 I am doing some Xamarin UI tests and I'm using the azure key vault to take a username and password and I am using a macbook macOS 10.15.3. I'm using nuget packages Microsoft.Azure.KeyVault (3.0.5), Microsoft.Azure.Services.AppAuthentication (1.4.0),

[nonik0]

@CSM91, I assume you are using AppAuth to get a token to authenticate to Key Vault to get the username and password? You mentioned in your previous comment that you set an environment variable to try to mitigate this issue and you were still seeing long retrieval times. That environment variable should have been AzureServicesAuthConnectionString to set a specific token provider to speed up token retrieval time--what was the value you tried?

[CSM91]

Hi, I've set in the terminal: export AzureServicesAuthConnectionString="RunAs=Developer;DeveloperTool=VisualStudio" and export AZURE_COLLECT_TELEMETRY="false" and it still took 1 min and 20 seconds

[nonik0]

@CSM91 Hmm, that is odd. I'd like to make absolutely sure that the AzureServiceTokenProvider is picking up that connection string because VS should not be taking that long--so could you either pass that connection string directly to ASTP constructor, or put some typo/garbage in your connection string environment variable and validate you get a connection string error from AppAuth?

[mfprz]

@nonik0 The creation of an AzureServicesAuthConnectionString environment variable solved the problem for me, thank you

[abhiyadav1323]

I tried using the connection string RunAs=Developer; DeveloperTool=VisualStudio and it skipped the second call (as shown in the image below) and directly went on to make the third call. So, the overall time it takes to fetch the access token and then secret is approx 4-6 seconds.

This is the code to initialize the key vault

        static ConfigUtils()
        {
            // https://docs.microsoft.com/en-us/azure/app-service/overview-managed-identity?tabs=dotnet#asal
            var azureServiceTokenProvider = new AzureServiceTokenProvider("RunAs=Developer; DeveloperTool=VisualStudio");
            vaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
        }

This is the code to fetch the secret

       try
       {
           var secret = await vaultClient.GetSecretAsync(key);
           return secret.Value;
       }

My main concern is that it makes the first call (connection to key vault) as soon as it executes the line: var secret = await vaultClient.GetSecretAsync(key); but then it makes the third call in the image above (skips the second call when using connection string: RunAs=Developer; DeveloperTool=VisualStudio) but there is a delay of nearly 3 sec between the two calls.

Is it possible to reduce that delay between calls?