Error: An error occurred while refreshing the key ring. Will try again in 2 minutes.

[apopa92]

Hi,

We have a website that sits on a webfarm (3 servers - Azure VMs behind load balancer).

• .NetCore 3.1, IIS hosted
• Microsoft.AspNetCore.DataProtection.AzureKeyVault 3.1.1
• Microsoft.AspNetCore.DataProtection.AzureStorage 3.1.1
• Microsoft.Azure.Services.AppAuthentication 1.3.1

The VMs have Managed Identities enabled and we are using the following code to enable the DataProtection, store the keys in the AzureStorage and encrypt them with the help of AzureKeyVault.

        var _azureServiceTokenProvider = new AzureServiceTokenProvider();
        var token = _azureServiceTokenProvider.GetAccessTokenAsync("https://storage.azure.com/").GetAwaiter().GetResult();
        var credentials = new StorageCredentials(new TokenCredential(token));
        var storageAccount = new CloudStorageAccount(credentials, "StorageAccountName", "core.windows.net", true);
        var client = storageAccount.CreateCloudBlobClient();
        var container = client.GetContainerReference("StorageContainerName");
        var keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(_azureServiceTokenProvider.KeyVaultTokenCallback));
        services.AddDataProtection()
            .SetApplicationName("ApplicationName")
            .PersistKeysToAzureBlobStorage(container, "StorageBlobName")
            .ProtectKeysWithAzureKeyVault(keyVaultClient, "KeyVaultIdentifier");

When the website starts, it creates the data protection key in the AzureStorage and everything seems to be working without any issues. The errors start pouring in after around 12-16 hours and they keep going, sometimes for a couple of hours, somtimes for a tens of hours.

Message: An error occurred while refreshing the key ring. Will try again in 2 minutes.
Microsoft.Azure.Storage.StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
   at Microsoft.Azure.Storage.Core.Executor.Executor.ExecuteAsync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   at Microsoft.AspNetCore.DataProtection.AzureStorage.AzureBlobXmlRepository.GetLatestDataAsync(ICloudBlob blobRef)
   at Microsoft.AspNetCore.DataProtection.AzureStorage.AzureBlobXmlRepository.GetAllElementsAsync(ICloudBlob blobRef)
   at Microsoft.AspNetCore.DataProtection.AzureStorage.AzureBlobXmlRepository.GetAllElements()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.GetAllKeys()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(DateTimeOffset now, IKey keyJustAdded)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.ICacheableKeyRingProvider.GetCacheableKeyRing(DateTimeOffset now)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefresh)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext) Request Information

[richteambs]

We have a very similar setup and we are experiencing similar problems.

[richteambs]

@apopa92 I think I have got to the bottom of this. The problem is that the access token is expiring (they appear to be valid for one hour) so the next time the DP mechanism tries to refresh the keys, it's using an expired token.

You can fix this by using an overload of the TokenCredential constructor that enables periodic refresh, and implement a method to perform the refresh.

var storageTokenCredential = new TokenCredential(
                accessToken,
                PeriodicStorageTokenRenewer,
                null,
                TimeSpan.FromMinutes(27));

// ...

private async Task<NewTokenAndFrequency> PeriodicStorageTokenRenewer(
            object state,
            CancellationToken cancellationToken)
{
    var accessToken = await GetAccessTokenAsync("https://storage.azure.com/");
    var tokenAndFrequency = new NewTokenAndFrequency { Token = accessToken };
    return tokenAndFrequency;
}

See https://github.com/dotnet/aspnetcore/issues/9012 for some background.

[byronbayer]

We are having the same issue and we are not using managed identities, we are using a SAS token generated with the storage account key that is valid for one hour. The issue doesn't occur until about 15 hours after deployment time.

 public static IServiceCollection AddWebDataProtection(this IServiceCollection services, ResultsAndCertificationConfiguration config, IWebHostEnvironment env)
        {
                services.AddDataProtection()
                        .PersistKeysToAzureBlobStorage(GetDataProtectionBlobTokenUri(config));
            return services;
        }

        private static Uri GetDataProtectionBlobTokenUri(ResultsAndCertificationConfiguration config)
        {
            var cloudStorageAccount = new CloudStorageAccount(new StorageCredentials(config.BlobStorageSettings.AccountName, config.BlobStorageSettings.AccountKey), useHttps: true);
            var blobClient = cloudStorageAccount.CreateCloudBlobClient();
            var container = blobClient.GetContainerReference(config.DataProtectionSettings.ContainerName);
            var blob = container.GetBlockBlobReference(config.DataProtectionSettings.BlobName);

            var sharedAccessPolicy = new SharedAccessBlobPolicy()
            {
                SharedAccessExpiryTime = DateTime.UtcNow.AddHours(1),
                Permissions = SharedAccessBlobPermissions.Read | SharedAccessBlobPermissions.Write | SharedAccessBlobPermissions.Create
            };

            var sasToken = blob.GetSharedAccessSignature(sharedAccessPolicy);
            return new Uri($"{blob.Uri}{sasToken}");
        }