AddDataProtection race condition

[SachaZvetelman]

I'm running multiple instances of a .NET Core 2.1 app behind a load balancer.

When multiple instances are created for the first time, there are frequently race conditions taking place around the DataProtection library that cause problems.

If these instances are created at the same time, the first one will create a new key and persist it to Redis, but the second one will do the same, and the third one will do the same. What you end up with is multiple keys in the key ring that are not known by many of the instances. If each instance is spinned up after a couple of seconds, then this problem doesn't occur.

If you see the logs below, there are three default keys used by different instances.

Given that for some reason the keys are created with an immediate activation date, in many cases, they don't get propagated to all the instances. The problem you'll end up facing is that if you encrypt a cookie with a key that is not available on another instance, you won't be able to decrypt it if you get to one of those instances, getting exceptions like this:

[15:29:57 ERR] An unhandled exception has occurred while executing the request.
System.Security.Cryptography.CryptographicException: The key {8480fbc2-3e02-4b31-b137-dfc300311e24} was not found in the key ring.
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.DangerousUnprotect(Byte[] protectedData, Boolean ignoreRevocationErrors, Boolean& requiresMigration, Boolean& wasRevoked)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect(IDataProtector protector, String protectedData)
at MyApp.Worker.Authentication.WebApp.Filters.ClientSettingsAsyncFilter.SetCobrand(ActionExecutingContext context) in /src/src/MyApp.Worker.Authentication.WebApp/Filters/ClientSettingsAsyncFilter.cs:line 64
at MyApp.Worker.Authentication.WebApp.Filters.ClientSettingsAsyncFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) in /src/src/MyApp.Worker.Authentication.WebApp/Filters/ClientSettingsAsyncFilter.cs:line 40
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
[15:29:57 ERR] An exception was thrown attempting to execute the error handler.
System.Security.Cryptography.CryptographicException: The key {8480fbc2-3e02-4b31-b137-dfc300311e24} was not found in the key ring.
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.DangerousUnprotect(Byte[] protectedData, Boolean ignoreRevocationErrors, Boolean& requiresMigration, Boolean& wasRevoked)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect(IDataProtector protector, String protectedData)
at MyApp.Worker.Authentication.WebApp.Filters.ClientSettingsAsyncFilter.SetCobrand(ActionExecutingContext context) in /src/src/MyApp.Worker.Authentication.WebApp/Filters/ClientSettingsAsyncFilter.cs:line 64
at MyApp.Worker.Authentication.WebApp.Filters.ClientSettingsAsyncFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) in /src/src/MyApp.Worker.Authentication.WebApp/Filters/ClientSettingsAsyncFilter.cs:line 40
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
[15:29:57 ERR] Connection id "0HLGK1LMKD07L", Request id "0HLGK1LMKD07L:00000003": An unhandled exception was thrown by the application.
System.Security.Cryptography.CryptographicException: The key {8480fbc2-3e02-4b31-b137-dfc300311e24} was not found in the key ring.
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.DangerousUnprotect(Byte[] protectedData, Boolean ignoreRevocationErrors, Boolean& requiresMigration, Boolean& wasRevoked)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect(IDataProtector protector, String protectedData)
at MyApp.Worker.Authentication.WebApp.Filters.ClientSettingsAsyncFilter.SetCobrand(ActionExecutingContext context) in /src/src/MyApp.Worker.Authentication.WebApp/Filters/ClientSettingsAsyncFilter.cs:line 64
at MyApp.Worker.Authentication.WebApp.Filters.ClientSettingsAsyncFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) in /src/src/MyApp.Worker.Authentication.WebApp/Filters/ClientSettingsAsyncFilter.cs:line 40
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

I could come up with dirty hacks to avoid this, either at a code level or infrastructure level, but I'd like to know what the recommended approach is and how this can be properly avoided going forward.

• Microsoft.NETCore.App version: 2.1.0
• dotnet -info:

  
  .NET Core SDK (reflecting any global.json):
  Version:   2.1.302
  Commit:    9048955601

Runtime Environment: OS Name: Windows OS Version: 10.0.17134 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\2.1.302\

Host (useful for support): Version: 2.1.2 Commit: 811c3ce6c0

.NET Core SDKs installed: 2.1.104 [C:\Program Files\dotnet\sdk] 2.1.202 [C:\Program Files\dotnet\sdk] 2.1.302 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed: Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Here are the relevant parts of my `Startup.cs` file:

var signingCertificate = new X509Certificate2(identityConfiguration.Credentials.FilePath, identityConfiguration.Credentials.Password, X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.EphemeralKeySet);

var redis = ConnectionMultiplexer.Connect(Configuration.GetConnectionString("Redis"));

services .AddDataProtection() .SetApplicationName(ApplicationName) .PersistKeysToRedis(redis, $"{ApplicationName}-DataProtectionKeys") .ProtectKeysWithCertificate(signingCertificate);

And here are the relevant parts of the logs of 8 instances when they are started:

**Instance 1:**

[15:28:37 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation. [15:28:37 DBG] Policy resolution states that a new key should be added to the key ring. [15:28:37 INF] Creating key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} with creation date 2018-09-06 15:28:37Z, activation date 2018-09-06 15:28:36Z, and expiration date 2018-12-05 15:28:36Z. [15:28:37 DBG] Descriptor deserializer type for key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. [15:28:37 DBG] No key escrow sink found. Not writing key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} to escrow. [15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'. [15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation. [15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}. [15:28:37 DBG] Considering key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} with expiration date 2018-12-05 15:28:36Z as default key. [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'. [15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'. [15:28:37 DBG] Using key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} as the default key. [15:28:37 DBG] Key ring with default key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} was loaded during application startup.

**Instance 2:**

[15:28:36 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation. [15:28:36 DBG] Policy resolution states that a new key should be added to the key ring. [15:28:36 INF] Creating key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef} with creation date 2018-09-06 15:28:36Z, activation date 2018-09-06 15:28:36Z, and expiration date 2018-12-05 15:28:36Z. [15:28:37 DBG] Descriptor deserializer type for key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. [15:28:37 DBG] No key escrow sink found. Not writing key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef} to escrow. [15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'. [15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation. [15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}. [15:28:37 DBG] Found key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef}. [15:28:37 DBG] Considering key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} with expiration date 2018-12-05 15:28:36Z as default key. [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'. [15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'. [15:28:37 DBG] Using key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} as the default key. [15:28:37 DBG] Key ring with default key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} was loaded during application startup.

**Instance 3:**

[15:28:37 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation. [15:28:37 DBG] Policy resolution states that a new key should be added to the key ring. [15:28:37 INF] Creating key {fdb8d6da-d846-429e-a77a-612320851ff4} with creation date 2018-09-06 15:28:37Z, activation date 2018-09-06 15:28:36Z, and expiration date 2018-12-05 15:28:36Z. [15:28:37 DBG] Descriptor deserializer type for key {fdb8d6da-d846-429e-a77a-612320851ff4} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. [15:28:37 DBG] No key escrow sink found. Not writing key {fdb8d6da-d846-429e-a77a-612320851ff4} to escrow. [15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'. [15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation. [15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}. [15:28:37 DBG] Found key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef}. [15:28:37 DBG] Found key {4ab699bc-a747-4452-8b4e-1df6828f7b30}. [15:28:37 DBG] Found key {fdb8d6da-d846-429e-a77a-612320851ff4}. [15:28:37 DBG] Considering key {4ab699bc-a747-4452-8b4e-1df6828f7b30} with expiration date 2018-12-05 15:28:37Z as default key. [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'. [15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'. [15:28:37 DBG] Using key {4ab699bc-a747-4452-8b4e-1df6828f7b30} as the default key. [15:28:37 DBG] Key ring with default key {4ab699bc-a747-4452-8b4e-1df6828f7b30} was loaded during application startup.

**Instance 4:**

[15:28:36 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation. [15:28:37 DBG] Policy resolution states that a new key should be added to the key ring. [15:28:37 INF] Creating key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0} with creation date 2018-09-06 15:28:37Z, activation date 2018-09-06 15:28:36Z, and expiration date 2018-12-05 15:28:36Z. [15:28:37 DBG] Descriptor deserializer type for key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. [15:28:37 DBG] No key escrow sink found. Not writing key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0} to escrow. [15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'. [15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation. [15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}. [15:28:37 DBG] Found key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef}. [15:28:37 DBG] Found key {4ab699bc-a747-4452-8b4e-1df6828f7b30}. [15:28:37 DBG] Found key {fdb8d6da-d846-429e-a77a-612320851ff4}. [15:28:37 DBG] Found key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0}. [15:28:37 DBG] Considering key {4ab699bc-a747-4452-8b4e-1df6828f7b30} with expiration date 2018-12-05 15:28:37Z as default key. [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'. [15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'. [15:28:37 DBG] Using key {4ab699bc-a747-4452-8b4e-1df6828f7b30} as the default key. [15:28:37 DBG] Key ring with default key {4ab699bc-a747-4452-8b4e-1df6828f7b30} was loaded during application startup.

**Instance 5:**

[15:28:37 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation. [15:28:37 DBG] Policy resolution states that a new key should be added to the key ring. [15:28:37 INF] Creating key {dac23a76-9aaa-4dae-be25-ecfcc2ea28a9} with creation date 2018-09-06 15:28:37Z, activation date 2018-09-06 15:28:37Z, and expiration date 2018-12-05 15:28:37Z. [15:28:37 DBG] Descriptor deserializer type for key {dac23a76-9aaa-4dae-be25-ecfcc2ea28a9} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. [15:28:37 DBG] No key escrow sink found. Not writing key {dac23a76-9aaa-4dae-be25-ecfcc2ea28a9} to escrow. [15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'. [15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation. [15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}. [15:28:37 DBG] Found key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef}. [15:28:37 DBG] Found key {4ab699bc-a747-4452-8b4e-1df6828f7b30}. [15:28:37 DBG] Found key {fdb8d6da-d846-429e-a77a-612320851ff4}. [15:28:37 DBG] Found key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0}. [15:28:37 DBG] Found key {dac23a76-9aaa-4dae-be25-ecfcc2ea28a9}. [15:28:37 DBG] Considering key {4ab699bc-a747-4452-8b4e-1df6828f7b30} with expiration date 2018-12-05 15:28:37Z as default key. [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'. [15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'. [15:28:37 DBG] Using key {4ab699bc-a747-4452-8b4e-1df6828f7b30} as the default key. [15:28:37 DBG] Key ring with default key {4ab699bc-a747-4452-8b4e-1df6828f7b30} was loaded during application startup.

**Instance 6:**

[15:28:37 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation. [15:28:37 DBG] Policy resolution states that a new key should be added to the key ring. [15:28:37 INF] Creating key {8480fbc2-3e02-4b31-b137-dfc300311e24} with creation date 2018-09-06 15:28:37Z, activation date 2018-09-06 15:28:37Z, and expiration date 2018-12-05 15:28:37Z. [15:28:37 DBG] Descriptor deserializer type for key {8480fbc2-3e02-4b31-b137-dfc300311e24} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. [15:28:37 DBG] No key escrow sink found. Not writing key {8480fbc2-3e02-4b31-b137-dfc300311e24} to escrow. [15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'. [15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation. [15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}. [15:28:37 DBG] Found key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef}. [15:28:37 DBG] Found key {4ab699bc-a747-4452-8b4e-1df6828f7b30}. [15:28:37 DBG] Found key {fdb8d6da-d846-429e-a77a-612320851ff4}. [15:28:37 DBG] Found key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0}. [15:28:37 DBG] Found key {dac23a76-9aaa-4dae-be25-ecfcc2ea28a9}. [15:28:37 DBG] Found key {8480fbc2-3e02-4b31-b137-dfc300311e24}. [15:28:37 DBG] Considering key {8480fbc2-3e02-4b31-b137-dfc300311e24} with expiration date 2018-12-05 15:28:37Z as default key. [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'. [15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'. [15:28:37 DBG] Using key {8480fbc2-3e02-4b31-b137-dfc300311e24} as the default key. [15:28:37 DBG] Key ring with default key {8480fbc2-3e02-4b31-b137-dfc300311e24} was loaded during application startup.

**Instance 7:**

[15:28:37 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation. [15:28:37 DBG] Policy resolution states that a new key should be added to the key ring. [15:28:37 INF] Creating key {c845406b-d630-43a3-949b-e0387ddfd377} with creation date 2018-09-06 15:28:37Z, activation date 2018-09-06 15:28:37Z, and expiration date 2018-12-05 15:28:37Z. [15:28:37 DBG] Descriptor deserializer type for key {c845406b-d630-43a3-949b-e0387ddfd377} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. [15:28:37 DBG] No key escrow sink found. Not writing key {c845406b-d630-43a3-949b-e0387ddfd377} to escrow. [15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'. [15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation. [15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}. [15:28:37 DBG] Found key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef}. [15:28:37 DBG] Found key {4ab699bc-a747-4452-8b4e-1df6828f7b30}. [15:28:37 DBG] Found key {fdb8d6da-d846-429e-a77a-612320851ff4}. [15:28:37 DBG] Found key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0}. [15:28:37 DBG] Found key {dac23a76-9aaa-4dae-be25-ecfcc2ea28a9}. [15:28:37 DBG] Found key {8480fbc2-3e02-4b31-b137-dfc300311e24}. [15:28:37 DBG] Found key {4f871aa5-0440-4e21-b21f-a40878f2f936}. [15:28:37 DBG] Found key {c845406b-d630-43a3-949b-e0387ddfd377}. [15:28:37 DBG] Considering key {8480fbc2-3e02-4b31-b137-dfc300311e24} with expiration date 2018-12-05 15:28:37Z as default key. [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'. [15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'. [15:28:37 DBG] Using key {8480fbc2-3e02-4b31-b137-dfc300311e24} as the default key. [15:28:37 DBG] Key ring with default key {8480fbc2-3e02-4b31-b137-dfc300311e24} was loaded during application startup.

**Instance 8:**

[15:28:37 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation. [15:28:37 DBG] Policy resolution states that a new key should be added to the key ring. [15:28:37 INF] Creating key {17bd9124-6a55-43d3-97ba-ec2543667a7e} with creation date 2018-09-06 15:28:37Z, activation date 2018-09-06 15:28:37Z, and expiration date 2018-12-05 15:28:37Z. [15:28:37 DBG] Descriptor deserializer type for key {17bd9124-6a55-43d3-97ba-ec2543667a7e} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. [15:28:37 DBG] No key escrow sink found. Not writing key {17bd9124-6a55-43d3-97ba-ec2543667a7e} to escrow. [15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'. [15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation. [15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}. [15:28:37 DBG] Found key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef}. [15:28:37 DBG] Found key {4ab699bc-a747-4452-8b4e-1df6828f7b30}. [15:28:37 DBG] Found key {fdb8d6da-d846-429e-a77a-612320851ff4}. [15:28:37 DBG] Found key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0}. [15:28:37 DBG] Found key {dac23a76-9aaa-4dae-be25-ecfcc2ea28a9}. [15:28:37 DBG] Found key {8480fbc2-3e02-4b31-b137-dfc300311e24}. [15:28:37 DBG] Found key {4f871aa5-0440-4e21-b21f-a40878f2f936}. [15:28:37 DBG] Found key {c845406b-d630-43a3-949b-e0387ddfd377}. [15:28:37 DBG] Found key {17bd9124-6a55-43d3-97ba-ec2543667a7e}. [15:28:37 DBG] Considering key {8480fbc2-3e02-4b31-b137-dfc300311e24} with expiration date 2018-12-05 15:28:37Z as default key. [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60 [15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'. [15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'. [15:28:37 DBG] Using key {8480fbc2-3e02-4b31-b137-dfc300311e24} as the default key. [15:28:37 DBG] Key ring with default key {8480fbc2-3e02-4b31-b137-dfc300311e24} was loaded during application startup.

[Eilon]

Ping @blowdart

[blowdart]

This is not unexpected, you'll have to stagger the instances. We can't go back and reread the key every time we see a new key ID because that's a DoS waiting to happen, all an attacker would have to do is send a new key id, with random data after it, and then everything stops while the keyring is refreshed.

@GrabYourPitchforks would a refresh within in the first X seconds of a new instance coming up help fix this problem, and mitigate the DoS?

[SachaZvetelman]

@blowdart, I agree with you. However, I raised the issue (if it's an issue at all...?) because maybe there's a better solution instead of staggering the instances or coming up with another solution external to this library.

[davidfowl]

We could try to solve the issue with redis specifically by using lua script. Thats atomic and basically takes a server lock. It's probably the easiest thing to do here.

[GrabYourPitchforks]

Sorry, somehow never got the notification for this. My suggestion would be to change the logic at the beginning of https://github.com/aspnet/DataProtection/blob/88a191f0f348a1eae467a906048e6adcac5f9cc3/src/Microsoft.AspNetCore.DataProtection/KeyManagement/DefaultKeyResolver.cs#L74. The new proposed logic would be:

1. Get the list of keys from the repository.
2. Filter the list to keys whose activation date is on or before now - _maxServerToServerClockSkew.
3. From the list in Step (2), find all keys whose creation date is on or before now - _keyPropagationWindow, marking the key with the most recent creation date as the "preferred" key.
4. If no preferred key is found from Step (3), then to back to the list from Step (2), finding all keys whose creation date is on or after now - _keyPropagationWindow, marking the key with the earliest creation date as the "preferred" key.
5. Run through the remainder of the "is revoked" / "is expired" / etc. logic already in the method.

This won't eliminate the race condition entirely (since it's possible the server clocks could be off by a second or two) but it should significantly narrow the window in which it could happen.

[oliverkane]

If anyone is having issues with this in a Kubernetes environment, may I suggest using an Init Container. This can be a super simple app, which only has the job of starting up, creating the initial key, and then terminating.

This doesn't solve the root cause, but it should significantly decrease the window of opportunity. I think combined with some other techniques above, that would all-but eliminate this bug.

[GrabYourPitchforks]

Another alternative is a play on what @blowdart suggested. When the DataProtection layer first comes online, if it immediately had to generate and persist a new key due to no existing key being acceptable, it could set a background timer to auto-refresh its keyring. (The refresh logic should only refresh; it should not add a new key.) The value _maxServerToServerClockSkew would be an appropriate delay period before running the refresh logic. It means that there could be an initial window after app start where the servers don't agree on a key, but at a time no later than t = _maxServerToServerClockSkew all servers should agree on the default key.

[blowdart]

@Eilon Frankly we should look at doing this properly, in code, rather than tell customers to deploy in an unnatural way. This isn't the first instance of this reported, and we keep pushing it back. I'd like to push it forward, as a bug into 3.0. Thoughts?

[GrabYourPitchforks]

I think in order to solve this properly you'd need transactions of some kind. Basically, you'd want the behavior to be: (a) check the persistence mechanism for the preferred key; (b) if no preferred key is found then generate a new one; (c) attempt to store the new key into the persistence mechanism only if the store itself hasn't been modified since the initial check in step A, otherwise go back to step A and loop.

This is guaranteed not to wind up in an endless loop because if the loop has to iterate for any reason, it means that some web server somewhere was able to make forward progress in the generation algorithm.

[brockallen]

I think in order to solve this properly you'd need transactions of some kind.

I doubt that will work for the network share filesystem approach.

I implemented something similar in a project where I create the first key, write it, then wait a configurable amount of time (presumably waiting for other nodes to write their keys), then reload from the store.

[davidfowl]

@GrabYourPitchforks Redis has that. This is why I suggested using lua script. It works and is transactional (because it locks the database). Should we just investigate that approach.

[NinoFloris]

Using https://redis.io/topics/transactions would work. Otherwise storing the keyring as separate redis keys (instead of list as it is right now) would allow you to use https://redis.io/commands/getset

[GrabYourPitchforks]

So there are a few ways forward:

a) Make a change to each of the repository implementations to take advantage of transactional capabilities of the underlying data store. This should work but would be some amount of code churn and complexity.

b) Make the change described in https://github.com/aspnet/AspNetCore/issues/3514#issuecomment-472204177 without changing any of the repository implementations. This should significantly reduce (but not eliminate) the race condition on first launch, and within 5 minutes (or whatever configurable time you want) of app launch the race condition should be eliminated entirely.

[davidfowl]

I think we should use what the underlying store provides (and start with just redis since most of the issues seem to be people using redis) but if b) is cheaper it seems like a fair thing to try and see if people still hit the issue.

[dorlandode]

This is definitely an issue using the file system provider. I attempted using a network share, and experienced several keys being generated with multiple app instances.

If the Ef core provider implemented a table lock, that would solve the problem for me.

[GrabYourPitchforks]

I posted at https://github.com/aspnet/AspNetCore/issues/3975#issuecomment-503775408 a potential solution for the file system provider.

[hoffmanntravis]

Would a simple file semaphore / mutex in the UNC directory solve this problem (for file based DPAPI storage)? That is, when generating a new key, create a file mutex in the shared directory that all phoning in applications check first, and wait until a given timeout for the lock to be removed or proceed with key generation. The application in the process of generating a key would delete the semaphore when done generating the key. If the timeout of the phoning in application is NOT exceeded when the lock is removed (suggesting generation is complete) it would instead change it's behavior to check if the newly generated key is acceptable. A cleanup step on exit would likely be needed so old mutex files would not create issues. You could also put a timestamp in the mutex file itself. Apologies if this comment is short sighted in some way.

"This is not unexpected, you'll have to stagger the instances. We can't go back and reread the key every time we see a new key ID because that's a DoS waiting to happen, all an attacker would have to do is send a new key id, with random data after it, and then everything stops while the keyring is refreshed."

@blowdart Not clear if this approach sidesteps that problem or not.

Also I see you merged the 5 minute fix there, didn't see that at first. I'll leave this here in case it's useful, and will check if our version is up to date.

[HaoK]

@blowdart are we ok with the 5 minute fix long term, or is there some more work that we want to consider here?

[blowdart]

More I think. We need to have a major rethink for parts of data protection anyway, which will happen in 6, so leave for now

[ghost]

Thanks for contacting us. We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[blowdart]

We believe that 3.1 has protection against this, we just forgot to close the issue.

There are some questions around how redis works, and if it's suitable for data protection as it may not make the atomicity we require.

[ghost]

Thanks for contacting us. We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[ghost]

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.

[adityamandaleeka]

This is being covered by the meta issue (https://github.com/dotnet/aspnetcore/issues/36157)

[WolfspiritM]

We have two pods running with redis persisted dataprotection keys. It worked fine for a while but it seems like last night both pods simulationously created the key and it caused our application to stop working correctly. We're using .NET 5 so this is still an issue.

[joshsten]

For the kubernetes cases.....

Is it possible to provide a "seed" or a key such that the encryption key would be calculated as the same for several pods coming up at once?

I'm thinking if we base the key off the build ID or some other common state that regularly changes it would be sufficient turnover and that the race doesn't really matter if everyone is carrying the same seed for the key as all of them would use the same projected key. Would coexists with timed turn over too only applying for situations like this.

An additional indirect option I considered which we don't like for the window of multiple versions is setting pod budgets to have k8s pods rollover one at a time.

[davidfowl]

A better k8s enabled version of this would push the key rotation to an operator component that injects the updated config. Most of this would be pushed out of the runtime itself.

cc @ReubenBond

[nolanbradshaw]

Is there an active workaround for this being used in production? I can reproduce this behavior in .NET 6 with multiple containers coming up at the same time

[AnthonyDewhirst]

@davidfowl Can you give an example of this please? (the operator component that is) I read previously that this would be fixed in .net 6, but I, as others have said, are still seeing this issue. We are not using redis either, but azure storage.

A detailed ms document on official solutions to this, if the framework can't do it, would be great.

[dharmeshtailor]

We just migrated to Asp.Net Core MVC. We have instance deployed on multiple servers and using AWS SM Key Manager and this issue just happened for us. Both the servers start at similar times and created 2 keys.

All our Anti Forgery validation failed when requests went cross servers.

[JoeMarkov]

Trying to fix the race condition inside the Data Protection API is hard.

Isn't then the real solution to just realize that providing the key for the Data Protection API is in the same problem domain, as HTTPS certificates (lets encrypt...) or JWT token signing keys?

They all need to be rotated on a regular basis and we always usually let someone else do it for us. Instead of having services trying to create their own keys (DPAPI), and some keys are provided externally, I think it's better to have as a policy that all keys are provided externally.

I really think the Data Protection API documentation should bring this up as a solution.

Perhaps then, the Data Protection API could be adjusted, so that it is easier to provide a set of keys from like Azure Key Vault or similar services?

[WolfspiritM]

Isn't then the real solution to just realize that providing the key for the Data Protection API is in the same problem domain, as HTTPS certificates (lets encrypt...) or JWT token signing keys?

While it might be a good idea to provide a manual way to assign data protection keys, I think there are two big differences compared to certificates here.

First, certificates usually are signed by a CA. This is outside of the control of .NET, so it makes sense that they have to be provided manually.

However, I think the main difference is that data protection isn't only about encrypting cookies or other "short-lived" data. It doesn't just require the current and the last key for rotation, but to decrypt older encrypted data, for example, in a database, it also requires all of the past encryption keys that have ever been used. So, as long as you don't reencrypt everything that is at rest, it would be a challenge to provide all previous keys manually somehow.

[JoeMarkov]

@WolfspiritM I agree, you need to manage the keyring, not a single encryption key. We have the same issues with OpenID-Connect and JWT token signing keys, where you might have set of both active and inactive keys.

So my point is that we should use the same approach in DAPI as we use for key mangement in services like IdentityServer similar. Let some external service or application managing the key-ring and let our ASP.NET Web-apps and APIs just read a read-only keyring from somewhere.

I think this approach should be mentioned in the documentation of the DPAPI.

[DavidGHyde]

I have been seeing this very, very intermittently on my Production site for quite a while now. It took a while to find this post and then some more time to actually get the application logs together to, I think, confirm it. Killing the pods one at a time and allowing them to restart fixes the issue.

Most of the time this happens after a deployment when the application restarts, but very occasionally I have seen this happen to the application when it has been stable for several days. I also observe a consistent baseline of a few cryptographic exceptions with a key not being found in a keyring per day without any apparent customer issue. Can anyone comment as to whether this behaviour is consistent with this issue?

What is the preferred workaround at this time?

[amcasey]

@DavidGHyde Sorry for the late reply - this wasn't the main tracking issue for Data Protection races. There are multiple races in this area, especially prior to 9.0, and most effort to date has been focused on making these exceptions rarer (vs eliminating them).

An approach we've been exploring for addressing the root cause has been to have a separate component that is responsible for all key generation and disabling key generation in app instances. If there's only one writer, there's no race. Unfortunately, the approach isn't mature enough for us to have a doc or a sample at this point, but let me know if you have questions.

[amcasey]

I'm going to close this issue. Please post additional feedback in #36157 so we can track it centrally and don't drop any.