SSL cert upload fails: Certificate invalid, or password is incorrect

[christoe]

Please provide us with the following information:

This issue is a: (mark with an x)

• [ ] bug report -> please search issues before submitting
• [ ] documentation issue or request
• [x] regression (a behavior that used to work and stopped in a new release)

Issue description

Adding/uploading a SSL certificate to a containerapps environment fails, both in UI and CLI: azure.cli.core.azclierror.CLIInternalError: (InvalidCertificate) The certificate wildcard.stageargeo.no is invalid or password is incorrect.

I have tried both using a wildcard certificate and a non-wildcard SNI cert (as I suspected a regression of issue #229) but both fail on upload (UI shows successful validation, but error message after clicking the "Add" button. I have tried creating a bundle with and without password, but both variants fail. Recent comment in #229 also suggest more people having the same issue.

Steps to reproduce

1. Request Let's Encrypt certificate
2. openssl pkcs12 -export -out certbundle.pfx -inkey privkey.pem -in fullchain.pem to create cert bundle with full chain
3. az containerapp ssl upload --certificate-file certbundle.pfx --environment <aca-env-name> --hostname <hostname> --certificate-name <certname> -g <resourcegroupname> -n <appname>

Expected behavior [What you expected to happen.] Successful add of certificate and option to map cert to container apps. I have previously been able to add a certificate bundle of the same type using the portal.

Actual behavior [What actually happened.] Error message: The certificate is invalid or password is incorrect.

Screenshots
n/a

Additional context

Issue occurs both in the CLI and the Portal. Enabled debug and looked at http responses, but they give no further clues to as what is happening.

[reidav]

I had this issue when the domain was added somewhere else such as in a container app, or container environment. Did you have a chance to look after the api response (DevToolbar) when you click on "Add" ? Sometimes, you have more details and the error is more explicit.

   "content": {
                "isHostnameAlreadyVerified": false,
                "customDomainVerificationTest": "Passed",
                "hasConflictOnManagedEnvironment": true,
                "conflictWithEnvironmentCustomDomain": true
    }

[christoe]

Thanks for your reply @reidav, however I'm not able to see the fields you have in your example - only error, code and message. The response in my case looks like this:

{
    "responses": [
        {
            "name": "b3c4e268-751a-4bfb-895c-d2207453e03b",
            "httpStatusCode": 400,
            "headers": {
                "Pragma": "no-cache",
                "x-ms-ratelimit-remaining-subscription-resource-requests": "99",
                "api-supported-versions": "2022-01-01-preview, 2022-03-01, 2022-06-01-preview, 2022-10-01, 2022-11-01-preview",
                "x-ms-request-id": "0fc882ce-1d91-4ee2-a946-d5c49f235a36",
                "x-ms-correlation-request-id": "b7425b3f-015e-4606-9e17-8debc1cf27ad",
                "x-ms-routing-request-id": "SWEDENSOUTH:20221130T125003Z:0fc882ce-1d91-4ee2-a946-d5c49f235a36",
                "Strict-Transport-Security": "max-age=31536000; includeSubDomains",
                "X-Content-Type-Options": "nosniff",
                "Cache-Control": "no-cache",
                "Date": "Wed, 30 Nov 2022 12:50:03 GMT",
                "Server": "Microsoft-IIS/10.0",
                "X-Powered-By": "ASP.NET"
            },
            "content": {
                "error": {
                    "code": "InvalidCertificate",
                    "message": "The certificate <cert-name> is invalid or password is incorrect."
                },
                "code": "InvalidCertificate",
                "message": "The certificate <cert-name> is invalid or password is incorrect."
            },
            "contentLength": 243
        }
    ]
}

[reidav]

It should be something else then ... i've done it like 4 days ago, here is the steps i've followed maybe it could help you.

1. Let's encrypt request (wildcard)

   certbot certonly --manual \
              --preferred-challenges=dns \
              --email foo@foo.xyz \
              --server https://acme-v02.api.letsencrypt.org/directory \
              --agree-tos \
              --manual-public-ip-logging-ok \
              --config-dir . \
              --work-dir . \
              --logs-dir . \
              -d '*.foo.xyz'

2. Cert bundle, added 'querty' as password

   openssl pkcs12 -export \
       -out ~/live/foo.xyz/cert.pfx \
       -inkey ~/live/foo.xyz/privkey.pem \
       -in ~/live/foo.xyz/cert.pem \
       -certfile ~/live/foo.xyz/chain.pem

3. Base64 encoding of pfx

   base64 ~/live/foo.xyz/cert.pfx

4. Added a bicep section with base64 in certificate value

   resource caenv 'Microsoft.App/managedEnvironments@2022-06-01-preview' = {
   name: containerAppEnvName
   location: location
   properties: {
   appLogsConfiguration: {
     destination: 'log-analytics'
     logAnalyticsConfiguration: {
       customerId: law.properties.customerId
       sharedKey: law.listKeys().primarySharedKey
     }
   }
   customDomainConfiguration: {
     certificatePassword: 'querty'
     certificateValue: '<<base64>>'
     dnsSuffix: 'foo.xyz'
   }
   }

[christoe]

I tried this method now as well, and it's producing pretty much the same error message:

 {
   "error": {
     "code": "InvalidCertificate",
     "message": "The certificate k8se-additional-domain-cert is invalid or password is incorrect."
   },
   "code": "InvalidCertificate",
   "message": "The certificate k8se-additional-domain-cert is invalid or password is incorrect."
 }

I'm able to open and validate the pfx-file and password using openssl pkcs12 -in certbundle.pfx -nokeys. No issues there. I was able the add a wildcard certificate using the same method about a month back or so in another ACA environment, but I cannot add this new certificate to that old environment now either.

Is there any way to get more debug information out of the system? Do anyone with insight into the code know what's being validated as part of the upload?

[reidav]

Can you try to load your certificate in a console app or something using the following code ? Are you having a 'CryptographicException' ?

var certBytes = Convert.FromBase64String(certValue);
X509Certificate2 cert = new X509Certificate2(certBytes, password, X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.EphemeralKeySet);

[christoe]

No, I'm able to load the certificate using that code, and read properties from the cert object. I had to remove the EphemeralKeySet flag though, as it seems its not supported on my platform (MacOS/Apple M1/.Net Core).

[michaelkira]

@christoe Can you try re-export your pfx using windows mmc certificate console? First install your pfx in windows, and then find your cert in mmc console Right click the target cert and choose export with private key [Important]Choose 3DES-SHA1 here, and use exported pfx and retry.

[christoe]

@michaelkira: What do you know!? It actually worked. Importing my openssl-generated pfx-file into MMC and then exporting it again made it upload successfully in ACA.

It was however your Important note that led me to find the cause of my issue; I was also able to upload an OpenSSL generated certificate if I use the 3DES/SHA1: openssl pkcs12 -export -out certbundle-des3sha1.pfx -inkey privkey.pem -in fullchain.pem -certfile cert.pem -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -macalg SHA1.

Why is 3DES allowed as it's being deprecated next month, while AES-256-CBC/SHA256 is not (as was the case with my failing cert)?

[michaelkira]

@michaelkira: What do you know!? It actually worked. Importing my openssl-generated pfx-file into MMC and then exporting it again made it upload successfully in ACA.

It was however your Important note that led me to find the cause of my issue; I was also able to upload an OpenSSL generated certificate if I use the 3DES/SHA1: openssl pkcs12 -export -out certbundle-des3sha1.pfx -inkey privkey.pem -in fullchain.pem -certfile cert.pem -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -macalg SHA1.

Why is 3DES allowed as it's being deprecated next month, while AES-256-CBC/SHA256 is not (as was the case with my failing cert)?

I suppose there is some middleware/cert sdk by default does not turn on AES support. We are investigating.

[SophCarp]

Hey @michaelkira Thanks for looking into this. Please report back the findings of your investigation. If there are any action items (such as for documentation) please let me know. Thanks!

[donprecious]

This method worked for me ,

openssl pkcs12 -export -out outputcert.pfx -inkey cert.key -in cert.pem -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -macalg SHA1

@christoe thanks

[donprecious]

This method worked for me ,

openssl pkcs12 -export -out outputcert.pfx -inkey cert.key -in cert.pem -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -macalg SHA1

@christoe thanks

[SophCarp]

@donprecious thanks for sharing the workaround! We'll add it to the docs until the underlying bug is fixed.

[christoe]

@SophCarp: Please look into if there are any better ciphers supported by ACA before updating the documentation. SHA1-3DES was the one I found works, but it's deprecated and should not be used.

[cemerick]

@michaelkira: What do you know!? It actually worked. Importing my openssl-generated pfx-file into MMC and then exporting it again made it upload successfully in ACA.

This class of issue has been percolating for a while, but oddly doesn't seem to affect everyone? (e.g. authors of posts like https://blog.devgenius.io/azure-container-apps-aca-and-cloudflare-e63e16ae1c64 apparently breeze through the process without difficulty?)

After reading https://stackoverflow.com/questions/29952874/azure-ssl-certificate last night (where people suggest round-tripping the .pfx cert through IIS, or through the local windows cert store via powershell), I was able to get the same effect by uploading the cert to an Azure Key Vault, then exporting it again as another .pfx. Might be a useful workaround for those that don't want to fuss with openssl too much. :wave:

[pbesbergler]

After some initial issues creating a working certificate I eventualy was able to create a PEM file using openssl which I am able to upload through the portal without any issues. But if I try to upload the same base64 encoded PEM using a bicep template I still get this "The certificate EnvironmentCustomDomainCertificate is invalid or password is incorrect." error. I tried multiple different base64 encoding options (e.g. openssl base64 -in cert.pem -out cert.pem.b64) or doing the base64 encoding in the bicep as below:

resource containerAppEnvironment_certificates 'Microsoft.App/managedEnvironments/certificates@2022-10-01' = { parent: containerAppEnvironment name: dnsSuffixName location: location properties: { value: any(base64(cert)) } }

Does anyone as any ideas on who to upload a certificate for a dns suffix using a bicep template?

[cemerick]

At some point, both the portal and the ARM/bicep deployment route began accepting .pem files. The cert we have in place now is just a simple concatenation of the private key and certificate, without any openssl/IIS/key vault shenanigans, and it rolls right in using this as a child resource of our app environments:

  resource certificate 'certificates' = {
    name: 'cloudflare2'
    location: location
    properties: {
      password: ''
      value: loadFileAsBase64('cloudflare2.pem')
    }
  }

[waynebrantley]

It took 3 weeks with help from support to generate the correct CSR (so it will produce elliptical curve ssl, etc). Then another week to figure out this SHA1 thing above - so his is a bit of a mess - but we do finally have a working solution!

Here is how you create the correct CSR to get the type of cert ACA requires.

openssl ecparam -out wildcard.key -name prime256v1 -genkey
openssl req -new -key wildcard.key -out wildcard.csr -sha256
openssl pkcs12 -export -out wildcard.pfx -inkey wildcard.key -in STAR_domain_com.crt -certfile intermedatecert1.crt -certfile intermedatecert1.crt 

Of course the last command creates a PFX that ACA will not accept - unless you follow the above about import/export from MMC or potentially add the -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -macalg SHA1 options - to generate a with a different encryption (evidentially depreciated)

I guess this is what preview is all about!

Hopefully they will make this use a non-depreciated encryption, provide docs on exactly what it requires for SSL (or just make it much friendlier with larger support of what it will accept) before it exits preview!

[nauar]

For the ones having issues with the certificate upload: I have had the following issue:

"Failed to add new private key certificate: 'certificate-name'. The certificate 'certificate-name' is invalid or password is incorrect."

If you created the .pfx fille with OpenSSL version 3, it is not working. I re-created the same certificate with OpenSSL version 1 and it worked well using the same command and parameters.

[csaba-kovacs]

For the ones having issues with the certificate upload: I have had the following issue:

"Failed to add new private key certificate: 'certificate-name'. The certificate 'certificate-name' is invalid or password is incorrect."

If you created the .pfx fille with OpenSSL version 3, it is not working. I re-created the same certificate with OpenSSL version 1 and it worked well using the same command and parameters.

Thanks, this one worked for me. Went back to 1.1