ordering issue of chains

[moparlakci]

Dude below you have an ordering issue

create one CA chain file

cat "$intermediate_CA_file" "$root_CA_file" > "$chain_file"

should be

cat "$root_CA_file" "$intermediate_CA_file"  > "$chain_file"

Then it will work fine ;)

[VojtechMyslivec]

Hello,

I am not sure it should be the way you mentioned here. Common order of a certificate chain is from the leaf to the root. Any subsequent certificate should "sign"/"verify" the one before.

In fact, this order was introduced in #18 about 5 years ago and no one has reported this issue since then.

If your Zimbra installation report some error, would you be so kind and share an error message here?

Vojtech

[moparlakci]

I’ve had some issues with verify command. It gave me unable_to_get_issuer_certificate error

Then I found this page https://wiki.zimbra.com/wiki/Fix_depth_lookup:unable_to_get_issuer_certificate

As you can see the ROOT comes first and then comes the intermediate certificates.

When I changed the ordering of the chain, it all worked back perfectly.

[VojtechMyslivec]

What Zimbra version do you use?

The article you linked is from 2015 and marked as WIP, so it's hard to tell if it is up-to date source :confused:

[moparlakci]

v8.7 I know it’s from 2015, but seems to be logical to put root first

[sjbronner]

@moparlakci In over 15 years of managing SSL and TLS certificates, I have only ever been confused as to what certificate should go where by one product: zimbra. It requires the inclusion of the trusted root certificate in the certificate chain, even though that is neither necessary nor good practice. To make things worse, the instructions are often misleading or contain straight-out errors. The correct order is to put the intermediate certificate or certificates first, in order so that the first is signed by the second and so on. The trusted root certificate goes last.

As to your depth-error, and for more information about Let's Encrypt certificates, the following articles might be helpful:

https://wiki.zimbra.com/wiki/Installing_a_RapidSSL_Commercial_Certificate (see the comment about errors at the bottom)

https://wiki.zimbra.com/wiki/Installing_a_LetsEncrypt_SSL_Certificate

If you would like help figuring out why the correct order of certificates was causing errors for you, please post the exact sequence of commands you ran along with their output.

[VojtechMyslivec]

@sjbronner thanks you very much for the info

inclusion of the trusted root certificate in the certificate chain, even though that is neither necessary nor good practice

exactly :+1:

The correct order is to put the intermediate certificate or certificates first, in order so that the first is signed by the second and so on. The trusted root certificate goes last.

I agree with that as well. Some of these wiki articles seems to be misleading, as you noted.

[VojtechMyslivec]

Closing this as it seems the discussion is over and the issue wasn't confirmed

[hubert3]

Confirming I needed exactly the change @moparlakci suggested above in order for zmcertmgr verifycrt / deploycrt to succeed

Without it, I was getting:

** Verifying '0000_cert.pem' against 'chain.pem'
ERROR: Unable to validate certificate chain: C = US, O = Internet Security Research Group, CN = ISRG Root X1

My Zimbra 8.8.15_GA_4232 (build 20220204072400) on Ubuntu 16.04 needs the chain to be written in this order for verifycrt / deploycrt to work.

[VojtechMyslivec]

Thanks for your comment.

This is really weird. Some installations works as expected while some others needs the reversed order of the certs :shrug:

[azadpsg]

I get this error could you guide me how to resolve this issue?

certbot: error: unrecognized arguments: --preferred-chain ISRG Root X1
letsencrypt-zimbra.sh: error: The certificate cannot be obtained with '/usr/bin/certbot' tool.

zimbra version : Release 8.8.15.GA.3869.UBUNTU18.64 UBUNTU18_64 FOSS edition, Patch 8.8.15_P11.

[VojtechMyslivec]

hi @azadpsg

I guess your problem is not related with "ordering issue of chains", which is discussed in this thread.

Your problem is the same as discussed in #78: your certbot client is too old and you need to update it to at least version 1.6.

[defkev]

Since i just ran into the same problem at a client i couldn't (and still can't) make sense of unless the clocks run backwards at Zimbra:

...
ERROR: Unable to validate certificate chain: C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 2 depth lookup: unable to get issuer certificate
error commercial.crt: verification failed

Zimbra expects:

1. Self-signed ISRG Root X1 -> https://letsencrypt.org/certs/isrgrootx1.pem
2. ISRG Root X1 signed R3 -> https://letsencrypt.org/certs/lets-encrypt-r3.pem

so:

cat isrgrootx1.pem lets-encrypt-r3.pem > commercial_ca.crt

which indeed is the wrong way around!

Chain goes intermediate(s) -> root, not root -> intermediate(s) otherwise your tree stands on its head.

chain.pem (from certbot)

1. ISRG Root X1 signed R3 -> https://letsencrypt.org/certs/lets-encrypt-r3.pem
2. Cross-signed ISRG Root X1 -> https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem

which is the proper order.

[zimbra@mail commercial]$ zmcertmgr verifycrt comm
** Verifying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' against '/opt/zimbra/ssl/zimbra/commercial/commercial.key'
Certificate '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' and private key '/opt/zimbra/ssl/zimbra/commercial/commercial.key' match.
** Verifying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' against '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt'
Valid certificate chain: /opt/zimbra/ssl/zimbra/commercial/commercial.crt: OK

This is zmcertmgr on Zimbra Release 8.8.15_GA_3869.RHEL7_64_20190917004220 RHEL7_64 FOSS edition, Patch 8.8.15_P40.

[VojtechMyslivec]

I prepared the branch reverse-cert-order with revert of efc0b921 commit.

Sadly, I don't have a Zimbra instance to test it now so I would leave it in the open branch for a while to not break other deployments where it works in the original order.

[defkev]

Looks like LE is about to get rid of the cross-signed X1, effectively shortening the chain of trust, so this should fix itself eventually:

https://letsencrypt.org/2023/07/10/cross-sign-expiration.html

[dhcmega]

Maintainer's note

Question in this post is moved to #83. Original text (which is not related to this issues) is below :arrow_heading_down:

---

Hi I'm trying to figure this out. I tried with the original and the reverted one:

#cat "$intermediate_CA_file" "$root_CA_file" > "$chain_file"
cat "$root_CA_file" "$intermediate_CA_file"  > "$chain_file"

Neither works:

** Verifying '/tmp/tmp.xuHC12YvmQ/0000_cert.pem' against '/opt/zimbra/ssl/zimbra/commercial/commercial.key'
Certificate '/tmp/tmp.xuHC12YvmQ/0000_cert.pem' and private key '/opt/zimbra/ssl/zimbra/commercial/commercial.key' match.
** Verifying '/tmp/tmp.xuHC12YvmQ/0000_cert.pem' against '/tmp/tmp.xuHC12YvmQ/0001_chain.pem'
ERROR: Unable to validate certificate chain: C = US, O = Let's Encrypt, CN = R10
error 2 at 1 depth lookup: unable to get issuer certificate
error /tmp/tmp.xuHC12YvmQ/0000_cert.pem: verification failed

On top of this, the currents certs expires tomorrow, great....

8.8.15.GA.4581.FOSS

Thanks!