search

matteocorti / check_ssl_cert

A shell script (that can be used as a Nagios/Icinga plugin) to check an SSL/TLS connection.
GNU General Public License v3.0
368 stars 132 forks source link

OCSP validation fails half the time when openssl ocsp -timeout is set #376

Closed moritz-makandra closed 2 years ago

moritz-makandra commented 2 years ago

When I run the the script to check the certs on our servers with OCSP enabled i get an CITICAL with the message OCSP error (-v for details)

The version is 2.25.0

My parameters are:

check_ssl_cert -H $HOSTNAME -d -v

I found this in the debug log. I tried the openssl command standalone. When i remove the -timeout parameter the command works fine 

[DBG] executing /usr/bin/openssl ocsp -timeout "120" -no_nonce -issuer /tmp/SJxKdr -cert /tmp/Xn6ngq  -url http://r3.o.lencr.org  -header HOST=r3.o.lencr.org
[DBG] OCSP: response = Error querying OCSP responder
[DBG] OCSP: response = 140602108248512:error:0200206F:system library:connect:Connection refused:../crypto/bio/bss_conn.c:189:hostname=r3.o.lencr.org service=80
[DBG] OCSP: response = 140602108248512:error:2007306E:BIO routines:conn_state:nbio connect error:../crypto/bio/bss_conn.c:193:

The certificate chain

-------------------------------------------------------------------------------
 Certificate 1
-------------------------------------------------------------------------------
subject=CN = REDACTED
X509v3 Subject Alternative Name: 
    DNS:REDACTED
issuer=C = US, O = Let's Encrypt, CN = R3
notBefore=Mar 15 11:27:05 2022 GMT
notAfter=Jun 13 11:27:04 2022 GMT
SHA1 Fingerprint=REDACTED
Public-Key: id-ecPublicKey (384 bit)
      OCSP: http://r3.o.lencr.org

> https://crt.sh/?q=REDACTED

-------------------------------------------------------------------------------
 Certificate 2
-------------------------------------------------------------------------------
subject=C = US, O = Let's Encrypt, CN = R3
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
notBefore=Sep  4 00:00:00 2020 GMT
notAfter=Sep 15 16:00:00 2025 GMT
SHA1 Fingerprint=A0:53:37:5B:FE:84:E8:B7:48:78:2C:7C:EE:15:82:7A:6A:F5:A4:05
Public-Key: rsaEncryption (2048 bit)
       CRL: http://x1.c.lencr.org/

> https://crt.sh/?q=A0:53:37:5B:FE:84:E8:B7:48:78:2C:7C:EE:15:82:7A:6A:F5:A4:05

-------------------------------------------------------------------------------
 Certificate 3
-------------------------------------------------------------------------------
subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
notBefore=Jan 20 19:14:03 2021 GMT
notAfter=Sep 30 18:14:03 2024 GMT
SHA1 Fingerprint=93:3C:6D:DE:E9:5C:9C:41:A4:0F:9F:50:49:3D:82:BE:03:AD:87:BF
Public-Key: rsaEncryption (4096 bit)
       CRL: http://crl.identrust.com/DSTROOTCAX3CRL.crl

> https://crt.sh/?q=93:3C:6D:DE:E9:5C:9C:41:A4:0F:9F:50:49:3D:82:BE:03:AD:87:BF

The full debug output

[DBG] Checking OCSP status of element 1
[DBG] temporary file /tmp/Xn6ngq created
[DBG] Storing the chain element in /tmp/Xn6ngq
[DBG] Checking revocation via OCSP
[DBG] extracting cert attribute issuer_hash
[DBG] Issuer hash: 8d33f237
[DBG] extracting cert attribute issuer_uri
[DBG] Chain element issuer URIs: http://r3.i.lencr.org/
[DBG] checking issuer URIs: http://r3.i.lencr.org/
[DBG] OCSP: fetching issuer certificate http://r3.i.lencr.org/ to /tmp/SJxKdr
[DBG] executing with timeout (120s): /usr/bin/curl    --silent --location \"http://r3.i.lencr.org/\" > /tmp/SJxKdr
[DBG]   start time = 1650642810
[DBG] /usr/bin/timeout 120 /bin/sh -c "/usr/bin/curl    --silent --location \"http://r3.i.lencr.org/\" > /tmp/SJxKdr"
[DBG]   end time = 1650642810
[DBG]   new timeout = 120
[DBG] OCSP: issuer certificate type (1): data
[DBG] OCSP: issuer certificate type (2): data
[DBG] OCSP: converting issuer certificate from DER to PEM
[DBG] OCSP: issuer certificate type (3): PEM certificate
[DBG] extracting cert attribute oscp_uri
[DBG] OCSP: URIs = http://r3.o.lencr.org
[DBG] OCSP: URI = http://r3.o.lencr.org
[DBG] OCSP: host = r3.o.lencr.org
[DBG] openssl ocsp supports the -header option
[DBG] /usr/bin/openssl ocsp -header requires 'key=value'
[DBG] executing /usr/bin/openssl ocsp -timeout "120" -no_nonce -issuer /tmp/SJxKdr -cert /tmp/Xn6ngq  -url http://r3.o.lencr.org  -header HOST=r3.o.lencr.org
[DBG] OCSP: response = Error querying OCSP responder
[DBG] OCSP: response = 140602108248512:error:0200206F:system library:connect:Connection refused:../crypto/bio/bss_conn.c:189:hostname=r3.o.lencr.org service=80
[DBG] OCSP: response = 140602108248512:error:2007306E:BIO routines:conn_state:nbio connect error:../crypto/bio/bss_conn.c:193:
[DBG] OCSP: not good. HTTP_PROXY = 
[DBG] executing /usr/bin/openssl ocsp -timeout "120" -no_nonce -issuer "/tmp/SJxKdr" -cert "/tmp/Xn6ngq" -url "http://r3.o.lencr.org" "" 2>&1
CRITICAL error: OCSP error (-v for details)
[DBG] CRITICAL >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[DBG] prepend_critical_message: new message    = OCSP error (-v for details)
[DBG] prepend_critical_message: CRITICAL_MSG   = 
[DBG] prepend_critical_message: ALL_MSG 1      = 
[DBG] prepend_critical_message: MSG 2          = SSL_CERT CRITICAL $SERVERNAME: OCSP error (-v for details)
[DBG] prepend_critical_message: CRITICAL_MSG 2 = SSL_CERT CRITICAL $SERVERNAME: OCSP error (-v for details)
[DBG] prepend_critical_message: ALL_MSG 2      = 
[DBG]     SSL_CERT CRITICAL $HOSTNAME: OCSP error (-v for details)
[DBG] CRITICAL <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
matteocorti commented 2 years ago

The error message is not the best, but it seems that you can not connect to r3.o.lencr.org on port 80.

I could improve it, if I could reproduce the error.

Please post the full log (I don't even know which system you are using).

The hostname would be really nice to have to reproduce the problem. If you really want to keep it secret, please send it to me privately.

matteocorti commented 2 years ago

I committed a new version, which should generate a little bit more output about the error.

moritz-makandra commented 2 years ago

The error message is not the best, but it seems that you can not connect to r3.o.lencr.org on port 80.

This is the Problem. While connecting to r3.o.lencr.org works fine over IPv4, IPv6 seems broken.

Thank you for pointing me into the right direction.

matteocorti commented 2 years ago

You could try to force IPv4 with the -4 flag. I have several hosts with Let's Encrypt and it seems to work with IPv6

moritz-makandra commented 2 years ago

Unfortunately this doesn't generate reproducible result sometimes the check fails when -4 is set and sometimes when -6. I think this could be caused by some sort of rate limiting or DDoS protection at Akamai

matteocorti commented 2 years ago

I am trying to test but the -6 and -4 are not really working. openssl s_client -6 is even connecting to an host with IPv4 only (see https://github.com/openssl/openssl/issues/18173)

moritz-makandra commented 2 years ago

I tried reproducing this problem on a different internet access, without success. I think this is not caused by any bug in check_ssl_cert.

Being able to distinct between IPv4 and IPv6 would be great. Hopefully the Issue in OpenSSL gets addressed.

Thank you for your support and the great tool.

matteocorti commented 2 years ago

Ok, then I'll close the issue. If something pops up, please let me know.