Invalid grant only during published version of client

Which version of Duende IdentityServer are you using? 7.04 Which version of .NET are you using? .net 8 Describe the bug

Getting an invalid grant when trying to use my app when its published, it works great locally under debug

To Reproduce

  string keyServiceClientSecret = Environment.GetEnvironmentVariable("KeyServiceClientSecret");
  ClientSecretCredential credential = new(
                Environment.GetEnvironmentVariable("KeyServiceTenantId"),
                 Environment.GetEnvironmentVariable("KeyServiceClientId"),
                keyServiceClientSecret);

  string dataSource = builder.Configuration["AppSettings:UcpmDataSource"];
  builder.Services.AddDataProtection()
      .PersistKeysToAzureBlobStorage("DefaultEndpointsProtocol=https;AccountName=ucpmcloudfiles;AccountKey=flyCsbFmpHOGusrvEHNAEFKcy5xWBGQPGE4GMlLkXkXNYG6iJ5Iy9uhHzPdKwcJQiUUw2nQj5BFbNATYl0G7Rg==;EndpointSuffix=core.windows.net",
      "protection",
      $"identity-{dataSource}")
      .ProtectKeysWithAzureKeyVault(
          new Uri(builder.Configuration["DataProtection:ProtectionKeyForKeys"]),
          credential);
  SecretClient secretClient = new(
         new Uri(builder.Configuration["KeyVault:RootUri"]),
         credential);

  Azure.Response<KeyVaultSecret> secretResponse = secretClient.GetSecret(
      builder.Configuration["KeyVault:CertificateName"]);

  X509Certificate2 signingCertificate = new(
    Convert.FromBase64String(secretResponse.Value.Value),
    (string)null,
    X509KeyStorageFlags.MachineKeySet);

  builder.Services.AddIdentityServer(options =>
  {
      // https://docs.duendesoftware.com/identityserver/v6/fundamentals/resources/api_scopes#authorization-based-on-scopes
      options.EmitStaticAudienceClaim = true;
      options.EmitScopesAsSpaceDelimitedStringInJwt = true;
      options.Events.RaiseFailureEvents = true;
      options.LicenseKey = "our license key (UCPM, LLC)";
      //options.ServerSideSessions. = false;
      //options.ena

  })
   .AddProfileService<AgentLocalUserProfileService>()
  .AddClientStore<ClientStore>()
  .AddCorsPolicyService<CorsPolicyService>()
  .AddResourceStore<ResourceStore>()
  .AddInMemoryCaching()
  .AddSigningKeyStore<SigningKeyStore>()
  .AddClientStoreCache<ClientStore>()
  .AddCorsPolicyCache<CorsPolicyService>()
  .AddResourceStoreCache<ResourceStore>()
  .AddIdentityProviderStoreCache<IdentityProviderStore>()
  .AddSigningCredential(signingCertificate)
  .AddServerSideSessions<ServerSideSessionStore>()
  .AddPushedAuthorizationRequestStore<PushedAuthorizationRequestStore>();

  builder.Services.AddTransient<IPersistedGrantStore, PersistedGrantStore>();
  builder.Services.AddTransient<IDeviceFlowStore, DeviceFlowStore>();

Expected behavior

Expect logins to work

Log output/exception with stacktrace

info: Duende.IdentityServer.Hosting.IdentityServerMiddleware[0]
      Invoking IdentityServer endpoint: Duende.IdentityServer.Endpoints.TokenEndpoint for /connect/token
fail: Duende.IdentityServer.Stores.DefaultAuthorizationCodeStore[0]
      Failed to deserialize JSON from grant store.
      System.Security.Cryptography.CryptographicException: The payload was invalid. For more information go to https://aka.ms/aspnet/dataprotectionwarning
         at Microsoft.AspNetCore.DataProtection.Cng.CbcAuthenticatedEncryptor.DecryptImpl(Byte* pbCiphertext, UInt32 cbCiphertext, Byte* pbAdditionalAuthenticatedData, UInt32 cbAdditionalAuthenticatedData)
         at Microsoft.AspNetCore.DataProtection.Cng.Internal.CngAuthenticatedEncryptorBase.Decrypt(ArraySegment`1 ciphertext, ArraySegment`1 additionalAuthenticatedData)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
         at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect(IDataProtector protector, String protectedData)
         at Duende.IdentityServer.Stores.Serialization.PersistentGrantSerializer.Deserialize[T](String json) in /_/src/Storage/Stores/Serialization/PersistentGrantSerializer.cs:line 103
         at Duende.IdentityServer.Stores.DefaultGrantStore`1.GetItemByHashedKeyAsync(String hashedKey)
fail: Duende.IdentityServer.Validation.TokenRequestValidator[0]
      Invalid authorization code { code = 4844A86E83400763862AEFE946746FAD1883F7E468635D08DEB751B4909E31A6-1 }, details: {
        "ClientId": "policytracker",
        "ClientName": "PolicyTracker",
        "GrantType": "authorization_code",
        "AuthorizationCode": "****A6-1",
        "RefreshToken": "********",
        "Raw": {
          "client_id": "policytracker",
          "client_secret": "***REDACTED***",
          "code": "***REDACTED***",
          "grant_type": "authorization_code",
          "redirect_uri": "https://policy-tracker-client-stage.azurewebsites.net/signin-oidc",
          "code_verifier": "vaWcfCoDvVuF8OzsgIB5F8LiMo8bSftYse8-2AA9yYc"
        }
      }
info: Duende.IdentityServer.Events.DefaultEventService[0]
      {
        "ClientId": "policytracker",
        "ClientName": "PolicyTracker",
        "Endpoint": "Token",
        "GrantType": "authorization_code",
        "Error": "invalid_grant",
        "Category": "Token",
        "Name": "Token Issued Failure",
        "EventType": "Failure",
        "Id": 2001,
        "ActivityId": "80001c2c-0014-ea00-b63f-84710c7967bb",
        "TimeStamp": "2024-05-14T17:49:00.9682702",
        "ProcessId": 14612,
        "LocalIpAddress": "192.168.8.24:443",
        "RemoteIpAddress": "192.168.10.99"
      }

Additional context

My identity server is load balanced across two servers, my api is also balanced across two servers, Client is hosted in Azure

DuendeSoftware / Support

Invalid grant only during published version of client #1257