System.Security.Cryptography.CryptographicException: The payload was invalid

[pmbaokhanh]

Which version of Duende IdentityServer are you using? Duende 6.1.5 Which version of .NET are you using? 6.0 Describe the bug

We have our Duende identity server running in k8s with 2 instances. Recently, after a deployment, we suddenly hit exceptions when clients try to refresh tokens with below stack trace

dbug: Duende.IdentityServer.EntityFramework.Stores.PersistedGrantStore[0]
      5D1AA9475989FFADB2FDB427872B15C8083CBFC6DE6A92003652A3535330E4CA found in database: True
trce: Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector[5]
      Performing unprotect operation to key {02692f47-964a-4ad2-ac42-f2a142642001} with purposes ('/app', 'PersistentGrantSerializer').
fail: Duende.IdentityServer.Stores.DefaultRefreshTokenStore[0]
      Failed to deserialize JSON from grant store.
      System.Security.Cryptography.CryptographicException: The payload was invalid. For more information go to http://aka.ms/dataprotectionwarning
         at Microsoft.AspNetCore.DataProtection.Managed.ManagedAuthenticatedEncryptor.Decrypt(ArraySegment`1 protectedPayload, ArraySegment`1 additionalAuthenticatedData)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
         at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect(IDataProtector protector, String protectedData)
         at Duende.IdentityServer.Stores.Serialization.PersistentGrantSerializer.Deserialize[T](String json) in /_/src/Storage/Stores/Serialization/PersistentGrantSerializer.cs:line 108
         at Duende.IdentityServer.Stores.DefaultGrantStore`1.GetItemByHashedKeyAsync(String hashedKey) in /_/src/IdentityServer/Stores/Default/DefaultGrantStore.cs:line 146
[04:09:06 ERR] Failed to deserialize JSON from grant store.
System.Security.Cryptography.CryptographicException: The payload was invalid. For more information go to http://aka.ms/dataprotectionwarning
   at Microsoft.AspNetCore.DataProtection.Managed.ManagedAuthenticatedEncryptor.Decrypt(ArraySegment`1 protectedPayload, ArraySegment`1 additionalAuthenticatedData)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
   at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect(IDataProtector protector, String protectedData)
   at Duende.IdentityServer.Stores.Serialization.PersistentGrantSerializer.Deserialize[T](String json) in /_/src/Storage/Stores/Serialization/PersistentGrantSerializer.cs:line 108
   at Duende.IdentityServer.Stores.DefaultGrantStore`1.GetItemByHashedKeyAsync(String hashedKey) in /_/src/IdentityServer/Stores/Default/DefaultGrantStore.cs:line 146
dbug: Duende.IdentityServer.Stores.DefaultRefreshTokenStore[0]
      refresh_token grant with value: 43893D46C338C7706873D85BAA7D0EC3117676B5C1CF2E58EAE67435F3B2F0F7-1 not found in store.
warn: Drawboard.IdentityServer.Services.CustomRefreshTokenService[0]
      Invalid refresh token

We are not sure what is causing it as there are no changes in the code. We checked and ensure that there is not change in our signing key and rollover signing key. We can see that the refresh token exist in the database (as indicated in the stack trace) and have not consumed.

Could you provide some guidance please?

Thanks a lot,

Minh Phan

[RolandGuijt]

This seems to be a data protection issue. Can you please follow our written guide about that and let us know if the tips in there helped?

[pmbaokhanh]

Thanks for your reply. I looked at the guide. We are almost doing what is recommended. However, there is one thing stand out for us that is:

• We are using .net 6
• We don't have application name set

  services.AddDataProtection()
              .PersistKeysToStackExchangeRedis(() => multiplexer.GetDatabase().WithKeyPrefix(redisPrefix), RedisDataProtectionKey);

• And we changed the path in our docker from WORKDIR /home/site/wwwroot to WORKDIR /app

Do you think that is what could cause this issue?

And if it is what is causing the issue. Does it mean that the issue should resolve itself after all of our user re-login? If I apply the application name now, it will cause another wave of forcing user to login right? Considering the deployment was out for about 1 week. For application name, will simply set it as "IdentityServer" work, or I have to use the path like this guide https://learn.microsoft.com/en-us/aspnet/core/security/data-protection/configuration/overview?view=aspnetcore-6.0#setapplicationname

Thanks Roland

[RolandGuijt]

Yes, I think that is the issue. Data protection prevents apps from accessing each others encrypted data by using a string value to discriminate. If an application name isn't set it will use the content root path to construct it.

IdentityServer relies heavily on the data protection feature. All grants e.g. are encrypted with it. So when the string is changed the current values for the application won't be readable anymore. That doesn't necessarily mean that all current sessions are ended on the spot. If clients are using a self-contained cookie, i.e. if the server side sessions feature of IdentityServer isn't active, the session will continue as long as the cookie is valid and there is no interaction with the identity provider (like refreshing access tokens).

I would set the value to a static string that uniquely identifies the application. Maybe something like "[companyname]-identityprovider".

[pmbaokhanh]

Thanks a not for your confirmation Roland. I will close this issue.