Signing certificate invalid on production in docker container

[Thodor12]

Confirm you've already contributed to this project or that you sponsor it

• [X] I confirm I'm a sponsor or a contributor

Version

3.x

Question

I've deployed my application using docker to a server, I've generated certificates using OpenSSL and passed them as a volume to my docker container.

When the application starts up I get an exception in the log saying:

root-nextlevel-auth-1  | warn: Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver[12]
root-nextlevel-auth-1  |       Key {5698643d-b0a7-4c77-b4f7-03c93ed2d78f} is ineligible to be the default key because its CreateEncryptor method failed.
root-nextlevel-auth-1  |       System.Security.Cryptography.CryptographicException: Symmetric algorithm is not specified.
root-nextlevel-auth-1  |          at System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey(EncryptedData , String )
root-nextlevel-auth-1  |          at System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument()
root-nextlevel-auth-1  |          at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.Decrypt(XElement encryptedElement)
root-nextlevel-auth-1  |          at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
root-nextlevel-auth-1  |          at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)
root-nextlevel-auth-1  |          at Microsoft.AspNetCore.DataProtection.KeyManagement.DeferredKey.<>c__DisplayClass1_0.<GetLazyDescriptorDelegate>b__0()
root-nextlevel-auth-1  |          at System.Lazy`1.ViaFactory(LazyThreadSafetyMode )
root-nextlevel-auth-1  |          at System.Lazy`1.ExecutionAndPublication(LazyHelper , Boolean )
root-nextlevel-auth-1  |          at System.Lazy`1.CreateValue()
root-nextlevel-auth-1  |          at System.Lazy`1.get_Value()
root-nextlevel-auth-1  |          at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.get_Descriptor()
root-nextlevel-auth-1  |          at Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.CngGcmAuthenticatedEncryptorFactory.CreateEncryptorInstance(IKey key)
root-nextlevel-auth-1  |          at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.CreateEncryptor()
root-nextlevel-auth-1  |          at Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver.CanCreateAuthenticatedEncryptor(IKey key)

This is an exception from the data protection stack, but if I disable this certificate on data protection I get about the same error from OpenIddict because they utilize the same certificate.

This is the code how I set up data protection and OpenIddict:

using var store = new X509Store(StoreName.My, StoreLocation.CurrentUser, OpenFlags.ReadWrite);

var dataProtectionCertificate = new X509Certificate2(
  configuration.GetValue<string>("Core:DataProtectionCertificatePath"),
  configuration.GetValue<string>("Core:DataProtectionCertificatePassword"));

var signingCertificate = new X509Certificate2(
  configuration.GetValue<string>("OpenId:SigningKeyCerfificatePath"),
  configuration.GetValue<string>("OpenId:SigningKeyCertificatePassword"));

store.Add(dataProtectionCertificate);
store.Add(signingCertificate);

services.AddDataProtection()
  .SetApplicationName(hidden)
  .PersistKeysToDbContext<ApplicationDbContext>()
  .ProtectKeysWithCertificate(dataProtectionCertificate);

----------------
// OpenIddict server options builder
options.UseDataProtection();
options.AddEncryptionCredentials(encryptionKey);
options.AddSigningCertificate(signingCertificate);

The 2 certificates are (for now) the exact same certificate generated using OpenSSL like so:

openssl genrsa -aes256 -out app_key.pem 2048
openssl req -key app_key.pem -new -x509 -days 36500 -out app_cert.crt
openssl pkcs12 -export -in app_cert.crt -inkey app_key.pem -out app_cert.pfx

Like I said the application is hosted on docker using the aspnet docker image, so this is running on Debian 11. The host system is Ubuntu 20.04.

What do I need to do to get valid certificates (because with the exact same certificates downloaded to my PC this exact same scenario works completely fine)

[kevinchalet]

What do I need to do to get valid certificates (because with the exact same certificates downloaded to my PC this exact same scenario works completely fine)

Can you try generating the certificates without OpenSSL? There's a snippet in the OpenIddict docs that uses the (excellent) .NET CertificateRequest API: https://documentation.openiddict.com/configuration/encryption-and-signing-credentials.html#registering-a-certificate-recommended-for-production-ready-scenarios

[Thodor12]

That did not give me a whole lot of progress, I have had to remove the data protection stuff because it errored with the exact same problem, I do not know which flags to attach to that. I've changed the code to:

        var encryptionCertificate = GetCertificate(
            configuration.GetValue<string>("OpenId:EncryptionCerfificatePath"),
            configuration.GetValue<string>("OpenId:EncryptionCertificatePassword"),
            X509KeyUsageFlags.KeyEncipherment);

        var signingCertificate = GetCertificate(
            configuration.GetValue<string>("OpenId:SigningKeyCerfificatePath"),
            configuration.GetValue<string>("OpenId:SigningKeyCertificatePassword"),
            X509KeyUsageFlags.DigitalSignature);

    private static X509Certificate2 GetCertificate(string fileName, string password, X509KeyUsageFlags keyUsages)
    {
        if (File.Exists(fileName))
        {
            return new X509Certificate2(fileName, password);
        }

        using var algorithm = RSA.Create(2048);

        var subject = new X500DistinguishedName("CN=IPM Knaagdierbeheersing");
        var request = new CertificateRequest(subject, algorithm, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
        request.CertificateExtensions.Add(new X509KeyUsageExtension(keyUsages, true));

        var certificate = request.CreateSelfSigned(DateTimeOffset.UtcNow, DateTimeOffset.UtcNow.AddYears(2));

        File.WriteAllBytes(fileName, certificate.Export(X509ContentType.Pfx, password));

        return certificate;
    }

However I am getting the exception:

root-nextlevel-auth-1  |       Connection id "0HMKR4IDSIJR4", Request id "0HMKR4IDSIJR4:0000000D": An unhandled exception was thrown by the application.
root-nextlevel-auth-1  |       System.InvalidOperationException: The exception handler configured on ExceptionHandlerOptions produced a 404 status response. This InvalidOperationException containing the original exception was thrown since this is often due to a misconfigured ExceptionHandlingPath. If the exception handler is expected to return 404 status responses then set AllowStatusCode404Response to true.
root-nextlevel-auth-1  |        ---> System.InvalidOperationException: The signing credentials algorithm is not valid.
root-nextlevel-auth-1  |          at OpenIddict.Server.OpenIddictServerHandlers.AttachTokenDigests.HandleAsync(ProcessSignInContext context)
root-nextlevel-auth-1  |          at OpenIddict.Server.OpenIddictServerDispatcher.DispatchAsync[TContext](TContext context)
root-nextlevel-auth-1  |          at OpenIddict.Server.OpenIddictServerDispatcher.DispatchAsync[TContext](TContext context)
root-nextlevel-auth-1  |          at OpenIddict.Server.OpenIddictServerHandlers.Exchange.HandleTokenRequest.HandleAsync(ProcessRequestContext context)
root-nextlevel-auth-1  |          at OpenIddict.Server.OpenIddictServerDispatcher.DispatchAsync[TContext](TContext context)
root-nextlevel-auth-1  |          at OpenIddict.Server.OpenIddictServerDispatcher.DispatchAsync[TContext](TContext context)
root-nextlevel-auth-1  |          at OpenIddict.Server.AspNetCore.OpenIddictServerAspNetCoreHandler.HandleRequestAsync()
root-nextlevel-auth-1  |          at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
root-nextlevel-auth-1  |          at Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware.InvokeInternal(HttpContext context)
root-nextlevel-auth-1  |          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
root-nextlevel-auth-1  |          --- End of inner exception stack trace ---
root-nextlevel-auth-1  |          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)
root-nextlevel-auth-1  |          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
root-nextlevel-auth-1  |          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

I think underwater this is still for the same reason as the other exception

[kevinchalet]

How do you register the certificate? Using the Add*Certificate(X509Certificate2) overload?

Can you please try to run CryptoConfig.CreateFromName("SHA256") and see what it returns on Docker?

[Thodor12]

I use the

                 options.AddEncryptionCertificate(encryptionCertificate);
                 options.AddSigningCertificate(signingCertificate);

overloads.

That cryptoconfig returns but that is not in the docker.

So how would I log that instance, if it's an instance of SHA256Managed inside Docker is it good enough or are you looking for a specific property that should be set?

[Thodor12]

Inside Docker on my server this prints an empty line (when just print x by using Console.WriteLine) so I'm assuming it's NULL

[kevinchalet]

So how would I log that instance, if it's an instance of SHA256Managed inside Docker is it good enough or are you looking for a specific property that should be set?

I only need to know whether null is returned or not and what's the actual type if an instance is returned. Something like Console.WriteLine(CryptoConfig.CreateFromName("SHA256")?.GetType().FullName) should be enough 😃

[kevinchalet]

Inside Docker on my server this prints an empty line (when just print x by using Console.WriteLine) so I'm assuming it's NULL

It's going to be a fun one to debug 🤣

I'll need more information before involving someone from the .NET team:

• Do you use assembly trimming?
• What .NET version do you target?

[Thodor12]

I have no clue what assembly trimming means but if you refer to this then yes:

RUN dotnet publish "NextLevel.Auth.csproj" -c Release -o /app/publish \
    -r linux-x64 \
    --self-contained true \
    -p:PublishTrimmed=true

I target .NET 6.

Full DockerFile for reference:

#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.

FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["NextLevel.Auth/NextLevel.Auth.csproj", "NextLevel.Auth/"]
COPY ["Common.Logic/Common.Logic.csproj", "Common.Logic/"]
COPY ["NextLevel.Logic/NextLevel.Logic.csproj", "NextLevel.Logic/"]
COPY ["Common.Resources/Common.Resources.csproj", "Common.Resources/"]
COPY ["NextLevel.Data/NextLevel.Data.csproj", "NextLevel.Data/"]
COPY ["NextLevel.Models/NextLevel.Models.csproj", "NextLevel.Models/"]
RUN dotnet restore "NextLevel.Auth/NextLevel.Auth.csproj"
COPY . .
WORKDIR "/src/NextLevel.Auth"
RUN dotnet build "NextLevel.Auth.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "NextLevel.Auth.csproj" -c Release -o /app/publish \
    -r linux-x64 \
    --self-contained true \
    -p:PublishTrimmed=true

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "NextLevel.Auth.dll"]

[kevinchalet]

I have no clue what assembly trimming means but if you refer to this then yes:

Can you try without -p:PublishTrimmed=true? It's likely the root cause of your pain 😄

[Thodor12]

Let's hope the images fit on my registry because that makes them quite a bit bigger now

[Thodor12]

Alright, short of the fact that my production app attempts to talk to localhost now that did it 🤣 Next time I'll just leave the docker file they give me as-is and not follow some janky tutorial on how to make it (they told me to add that line).

Thank you so much though, I've been stuck on this for a while and neither StackOverflow nor the C# Discord server were really any help!

[kevinchalet]

Thank you so much though, I've been stuck on this for a while and neither StackOverflow nor the C# Discord server were really any help!

My pleasure 😄

So, as I suspected, the root cause is that CryptoConfig.CreateFromName() (used by OpenIddict to resolve hash algorithms in a dynamic and configurable way) is likely not trimming-friendly so when the linker kicks in, the SHA256 type (and its actual implementations) is considered "unused" and removed from its assembly (and thus cannot be resolved by CryptoConfig.CreateFromName()).

@bartonjs hey. Can you please confirm I got this right? If so, what's the recommended action(s) in this case? I suspect moving to the static SHA*.Create() methods is what you'll recommend, but I really liked the fact you could use different SHA* implementations via CryptoConfig.AddOID() and AFAICT, it's not something you can do with the algorithm-specific static methods?

[bartonjs]

You're correct, which is why CryptoConfig.CreateFromName is painted with [RequiresUnreferencedCode("The default algorithm implementations might be removed, use strong type references like 'RSA.Create()' instead.")].

It might work to just have strong references somewhere else in your code; but in the BCL we never call into CryptoConfig, just use our own switchafalls (e.g. https://github.com/dotnet/runtime/blob/2f3fcca06499725469b6ae021be960e08d808c75/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/HMACCommon.cs#L57-L69).

[kevinchalet]

You're correct, which is why CryptoConfig.CreateFromName is painted with [RequiresUnreferencedCode("The default algorithm implementations might be removed, use strong type references like 'RSA.Create()' instead.")].

Makes total sense. While OpenIddict itself doesn't enable trimming support on its assemblies (yet), enabling it at the app-level actually affects it directly given the linker treats all assemblies as a whole, so yeah, not unreasonable 😄

It might work to just have strong references somewhere else in your code;

I guess a good approach preserving the ability to swap implementations while making it trimming friendly would be to use something like CryptoConfig.CreateFromName("SHA256") as HashAlgorithm ?? SHA256.Create(). CryptoConfig.CreateFromName("SHA256") shouldn't return null due to the direct type reference to SHA256.Create() but it covers marginal cases where an invalid type is registered via CryptoConfig.AddOID().

but in the BCL we never call into CryptoConfig, just use our own switchafalls

In @Thodor12's original post, he mentioned a similar issue with ASP.NET Core Data Protection when using key encryption at rest with certificates. I took a look at the underlying code and EncryptedXml.GetDecryptionKey() uses it to build SymmetricAlgorithm instances in at least one branch (not sure it's actually part of the BCL, as the scope isn't as clear as it used to be 😃): https://github.com/dotnet/runtime/blob/9d6396deb02161f5ee47af72ccac52c2e1bae458/src/libraries/System.Security.Cryptography.Xml/src/System/Security/Cryptography/Xml/EncryptedXml.cs#L375

[bartonjs]

Yeah, I almost said "aside from SignedXml and EncryptedXml". They're legacy components that we basically don't touch.

[kevinchalet]

Makes sense, thanks for your insights 😃

I don't see in the docs that trimming might break ASP.NET Core Data Protection when using key protection via a certificate. Should that be mentioned somewhere?

[bartonjs]

I think it hasn't come up automatically yet because we haven't finished trimmer analysis on System.Security.Cryptography.Xml. That's not to say that wherever the data protector lives wouldn't benefit from a nudge to let them know they might want to draft a "how to make data protector happy in this scenario" doc.

[kevinchalet]

Improvements are tracked by https://github.com/openiddict/openiddict-core/issues/1555.