search

matteocorti / check_ssl_cert

A shell script (that can be used as a Nagios/Icinga plugin) to check an SSL/TLS connection.
GNU General Public License v3.0
360 stars 132 forks source link

Sporadic Error checking SSL Labs: curl returned 56, see 'man curl' for details #493

Closed Constey closed 7 months ago

Constey commented 7 months ago

Describe the bug

I've a sporadic issue, that the ssllabs verification is not working and giving this as output: Error checking SSL Labs: curl returned 56, see 'man curl' for details We're running this command from checkmk, so I dont have a nice debug log. When i run the command from hand, it looks good always. May we check to often? (60s) Is there a way to do a debug logging to file via command while having the "normal" ouput? Then I can try to collect more details if the issue occours.

To Reproduce

check_ssl_cert -H $HOSTADDRESS$ -r /etc/ssl/certs/ -L A+ -w 10 -c 7

Expected behavior

Check should response true or false ;)

System (please complete the following information):

[DBG] check_ssl_cert version: 2.76.0
[DBG] System info: Linux monitoring 5.15.0-1053-azure #61~20.04.1-Ubuntu SMP Tue Nov 21 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[DBG] /etc/os-release:
[DBG]   NAME="Ubuntu"
[DBG]   VERSION="20.04.6 LTS (Focal Fossa)"
[DBG]   ID=ubuntu
[DBG]   ID_LIKE=debian
[DBG]   PRETTY_NAME="Ubuntu 20.04.6 LTS"
[DBG]   VERSION_ID="20.04"
[DBG]   HOME_URL="https://www.ubuntu.com/"
[DBG]   SUPPORT_URL="https://help.ubuntu.com/"
[DBG]   BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
[DBG]   PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
[DBG]   VERSION_CODENAME=focal
[DBG]   UBUNTU_CODENAME=focal
[DBG] User: root
[DBG] Shell: /bin/bash
[DBG]   GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)
[DBG]   Copyright (C) 2019 Free Software Foundation, Inc.
[DBG]   License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
[DBG]
[DBG]   This is free software; you are free to change and redistribute it.
[DBG]   There is NO WARRANTY, to the extent permitted by law.
[DBG] grep: /bin/grep
[DBG]   grep (GNU grep) 3.4
[DBG]   Copyright (C) 2020 Free Software Foundation, Inc.
[DBG]   License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
[DBG]   This is free software: you are free to change and redistribute it.
[DBG]   There is NO WARRANTY, to the extent permitted by law.
[DBG]
[DBG]   Written by Mike Haertel and others; see
[DBG]   <https://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.
[DBG] hostname: /bin/hostname
[DBG] $PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
[DBG] Command line arguments: -d -v
[DBG]   TMPDIR = /tmp
[DBG] Required HTTP headers:
[DBG] Unrequired HTTP headers:
[DBG] cleaning up temporary files
SSL_CERT UNKNOWN: No host specified
matteocorti commented 7 months ago

You can try with the --debug-file file Write the debug messages to file option. Without some output or an example it will be difficult...

Constey commented 7 months ago

Okay i've got lucky and catched this one:

 /opt/omd/sites/slave/local/lib/nagios/plugins/check_ssl_cert -H www.domain-bkk.de -r /etc/ssl/certs/ -L A+ -w 14 -c 7 -vv  --debug-file /tmp/check_ssl.debug.log
expect not available
timeout available (/usr/bin/timeout)
Found GNU date with timestamp support: enabling date computations
Downloading certificate to /tmp
checking TLS renegotiation
TLS renegotiation OK
Parsing the x509 certificate file
The certificate for this site contains a Subject Alternative Name extension
Certificate element 1 (www.domain-bkk.de) is valid for 70 days
OCSP check for element 1 OK
Certificate element 2 (R3) is valid for 644 days
OCSP check for element 2 OK
Certificate element 3 (ISRG Root X1) is valid for 295 days
OCSP check for element 3 OK
Error checking SSL Labs: curl returned 35, see 'man curl' for details|days_chain_elem1=70;14;7;; days_chain_elem2=644;14;7;; days_chain_elem3=295;14;7;;

the debug log looks like:

cat /tmp/check_ssl.debug.log
Sun 10 Dec 2023 05:56:01 PM CET
Command line arguments: -v
  TMPDIR = /tmp
Required HTTP headers:
Unrequired HTTP headers:
curl binary needed. SSL Labs = A+, OCSP = 1, CURL = , IGNORE_CONNECTION_STATE=, FILE_URI=
curl binary not specified
curl available: /usr/bin/curl
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets
Proxy settings (after):
  http_proxy  =
  https_proxy =
  HTTP_PROXY  =
  HTTPS_PROXY =
  s_client    =
  curl        =
Checking if the host is listed in /etc/hosts
Host not found in /etc/hosts: checking DNS
Checking if the host (www.domain-bkk.de) exists
HOST = www.domain-bkk.de
SNI                 =
HOST_NAME           = www.domain-bkk.de
HOST_ADDR           = www.domain-bkk.de
NAMES_TO_BE_CHECKED = __HOST__
Checking if www.domain-bkk.de is an IP address
www.domain-bkk.de is not an IP address
HOST_IS_IP.         = 0
Checking if www.domain-bkk.de is an IP address
www.domain-bkk.de is not an IP address
Adding www.domain-bkk.de to NAMES_TO_BE_CHECKED
NAMES_TO_BE_CHECKED = www.domain-bkk.de
Root CA option = -CApath /etc/ssl/certs/
-c specified: 7
-w specified: 14
Executing comparison '1209600 < 604800' (precision 0)
  bc result = 0
  returning 1
ROOT_CA = -CApath /etc/ssl/certs/
mktemp available: /bin/mktemp
file version: file-5.38
magic file from /etc/magic:/usr/share/misc/magic
nmap binary not specified
nmap available: /usr/bin/nmap
Checking IPs: host www.domain-bkk.de
perl available: /usr/bin/perl
date available: /bin/date
checking date version
date computation type: GNU
'/usr/bin/openssl s_client' supports '-servername': using -servername www.domain-bkk.de
Proxy settings (before):
  http_proxy  =
  https_proxy =
  HTTP_PROXY  =
  HTTPS_PROXY =
'/usr/bin/openssl s_client' supports '-name': using monitoring-slave
'/usr/bin/openssl s_client' supports '-xmpphost': using -xmpphost www.domain-bkk.de
HOST_HEADER = www.domain-bkk.de
Testing connection with www.domain-bkk.de:443
Executing: '/usr/bin/nmap  --unprivileged -Pn -p 443 www.domain-bkk.de'
/bin/grep -q "443.*open"
Sanity checks: OK
temporary file /tmp/sXPOr1 created
temporary file /tmp/NSsDYz created
temporary file /tmp/KHlAdX created
temporary file /tmp/Rfg3kH created
temporary file /tmp/Gtr9pG created
temporary file /tmp/I4K3JU created
temporary file /tmp/1DTQdG created
Temporary files created
www.domain-bkk.de is not an IP address
fetch_certificate: PROTOCOL =
exec_with_timeout: TIMEOUT=120, CURRENT_TIMEOUT=120, ELAPSED=0
exec_with_timeout printf 'HEAD / HTTP/1.1
Host: www.domain-bkk.de
User-Agent: check_ssl_cert/2.76.0
Connection: close

' | /usr/bin/openssl s_client     -crlf  -connect www.domain-bkk.de:443 -servername www.domain-bkk.de   -showcerts -verify 6 -CApath /etc/ssl/certs/       2> /tmp/NSsDYz 1> /tmp/sXPOr1
executing with timeout (120s): printf 'HEAD / HTTP/1.1
Host: www.domain-bkk.de
User-Agent: check_ssl_cert/2.76.0
Connection: close

' | /usr/bin/openssl s_client     -crlf  -connect www.domain-bkk.de:443 -servername www.domain-bkk.de   -showcerts -verify 6 -CApath /etc/ssl/certs/       2> /tmp/NSsDYz 1> /tmp/sXPOr1
/usr/bin/timeout 120 /bin/sh -c "printf 'HEAD / HTTP/1.1
Host: www.domain-bkk.de
User-Agent: check_ssl_cert/2.76.0
Connection: close

' | /usr/bin/openssl s_client     -crlf  -connect www.domain-bkk.de:443 -servername www.domain-bkk.de   -showcerts -verify 6 -CApath /etc/ssl/certs/       2> /tmp/NSsDYz 1> /tmp/sXPOr1"
Return value of the command = 0
Negotiated protocol:
openssl_version 3.0.0
Checking if OpenSSL version is at least 3.0.0 ( '3' '0' '0' ':0' )
openssl version: OpenSSL 1.1.1f  31 Mar 2020
Current version 1.1.1f ( '1' '1' '1' 'f:102' )
checking TLS renegotiation
exec_with_timeout: TIMEOUT=120, CURRENT_TIMEOUT=119, ELAPSED=1
exec_with_timeout printf 'R
' | /usr/bin/openssl s_client   -crlf -connect www.domain-bkk.de:443 -servername www.domain-bkk.de   2>&1 | /bin/grep -F -q err
executing with timeout (119s): printf 'R
' | /usr/bin/openssl s_client   -crlf -connect www.domain-bkk.de:443 -servername www.domain-bkk.de   2>&1 | /bin/grep -F -q err
/usr/bin/timeout 119 /bin/sh -c "printf 'R
' | /usr/bin/openssl s_client   -crlf -connect www.domain-bkk.de:443 -servername www.domain-bkk.de   2>&1 | /bin/grep -F -q err"
extracting cert attribute enddate
extracting cert attribute startdate
extracting cert attribute cn
extracting cert attribute subject
SUBJECT = subject=CN = www.domain-bkk.de
extracting cert attribute serial
SERIAL = 0371873635C71EF9FDEEE91ADFC24781E3AD
extracting cert attribute version
X509_VERSION = 3 (0x2)
extracting cert attribute fingerprint
FINGERPRINT = 97:5B:AC:EF:05:15:3E:B0:5F:82:6E:50:1D:74:E9:74:B2:85:47:2A
Checking if x509 supports the -ext option
extracting cert attribute keyUsage
Certificate purpose is not defined as critical
extracting cert attribute oscp_uri_single
extracting cert attribute oscp_uri
OCSP_URI = http://r3.o.lencr.org
Extracting issuers
  Number of certificates in the chain: 3
Checking certificate chain
    extracting issuer for element 1
extracting cert attribute issuer
ELEMENT_ISSUER=Let's Encrypt
ELEMENT_ISSUER=R3
ISSUERS=Let's Encrypt
ISSUERS=R3
    extracting issuer for element 2
extracting cert attribute issuer
ELEMENT_ISSUER=Internet Security Research Group
ELEMENT_ISSUER=ISRG Root X1
ISSUERS=Let's Encrypt
ISSUERS=R3
ISSUERS=Internet Security Research Group
ISSUERS=ISRG Root X1
    extracting issuer for element 3
extracting cert attribute issuer
ELEMENT_ISSUER=Digital Signature Trust Co.
ELEMENT_ISSUER=DST Root CA X3
ISSUERS=Let's Encrypt
ISSUERS=R3
ISSUERS=Internet Security Research Group
ISSUERS=ISRG Root X1
ISSUERS=Digital Signature Trust Co.
ISSUERS=DST Root CA X3
Certificate chain check finished
ISSUERS =
Let's Encrypt
R3
Internet Security Research Group
ISRG Root X1
Digital Signature Trust Co.
DST Root CA X3
extracting cert attribute issuer_uri_single
extracting cert attribute issuer_uri
extracting cert attribute pub_key_algo
extracting cert attribute sig_algo
extracting cert attribute subjectAlternativeName
subjectAlternativeName = ruv-bkk.de www.domain-bkk.de
Check the common name and alternative names
CN                       = www.domain-bkk.de
ALTNAMES                 = 1
NAMES_TO_BE_CHECKED      = www.domain-bkk.de
  checking 'www.domain-bkk.de'
    common name
      checking if (3) www.domain-bkk.de matches ^www.domain-bkk.de$
        www.domain-bkk.de matches ^www.domain-bkk.de$
    alternative names
      check altname: ruv-bkk.de
      check altname: www.domain-bkk.de
      www.domain-bkk.de matches ^www.domain-bkk.de$
 CN check finished
Checking expiration date
Number of certificates in CA chain: 3
------------------------------------------------------------------------------
-- Checking element 1
extracting cert attribute cn
Checking expiration date of element 1 (www.domain-bkk.de)
extracting cert attribute enddate
Validity date on cert element 1 (www.domain-bkk.de) is Feb 19 01:30:13 2024 GMT
Date computations: GNU
Computing number of hours until 'Feb 19 01:30:13 2024 GMT' with GNU
Computing '(1708306213-1702227362)/3600' (precision 0)
Hours until Feb 19 01:30:13 2024 GMT: 1688
Computing '1688/24' (precision 0)
Computing '1688 * 3600' (precision 0)
Adding line to prometheus days output: cert_days_chain_elem{cn="www.domain-bkk.de", element="1"} 70
  valid for 70 days
executing: /usr/bin/openssl x509 -noout -checkend 0 on cert element 1 (www.domain-bkk.de)
executing: /usr/bin/openssl x509 -noout -checkend 604800 on cert element 1 (www.domain-bkk.de)
executing: /usr/bin/openssl x509 -noout -checkend 1209600 on cert element 1
Adding line to prometheus validity output: cert_valid_chain_elem{cn="www.domain-bkk.de", element="1"} 0
------------------------------------------------------------------------------
Checking OCSP status of element 1
temporary file /tmp/Nm6nSq created
Storing the chain element in /tmp/Nm6nSq
Checking revocation via OCSP
extracting cert attribute issuer_hash
Issuer hash: 8d33f237
extracting cert attribute issuer_uri
Chain element issuer URIs: http://r3.i.lencr.org/
checking issuer URIs: http://r3.i.lencr.org/
OCSP: fetching issuer certificate http://r3.i.lencr.org/ to /tmp/I4K3JU
exec_with_timeout: TIMEOUT=120, CURRENT_TIMEOUT=119, ELAPSED=1
exec_with_timeout /usr/bin/curl    --silent --user-agent 'check_ssl_cert/2.76.0' --location \"http://r3.i.lencr.org/\" > /tmp/I4K3JU
executing with timeout (119s): /usr/bin/curl    --silent --user-agent 'check_ssl_cert/2.76.0' --location \"http://r3.i.lencr.org/\" > /tmp/I4K3JU
/usr/bin/timeout 119 /bin/sh -c "/usr/bin/curl    --silent --user-agent 'check_ssl_cert/2.76.0' --location \"http://r3.i.lencr.org/\" > /tmp/I4K3JU"
OCSP: issuer certificate type (1): data
OCSP: issuer certificate type (2): data
OCSP: converting issuer certificate from DER to PEM
OCSP: issuer certificate type (3): PEM certificate
extracting cert attribute oscp_uri
OCSP: URIs = http://r3.o.lencr.org
OCSP: URI = http://r3.o.lencr.org
OCSP: host = r3.o.lencr.org
openssl ocsp supports the -header option
/usr/bin/openssl ocsp -header requires 'key=value'
executing (5) /usr/bin/openssl ocsp -timeout "119" -no_nonce -issuer /tmp/I4K3JU -cert /tmp/Nm6nSq  -url http://r3.o.lencr.org  -header HOST=r3.o.lencr.org
OCSP: response = Response verify OK
OCSP: response = /tmp/Nm6nSq: good
OCSP: response =        This Update: Dec  9 12:00:00 2023 GMT
OCSP: response =        Next Update: Dec 16 11:59:58 2023 GMT
------------------------------------------------------------------------------
-- Checking element 2
extracting cert attribute cn
Checking expiration date of element 2 (R3)
extracting cert attribute enddate
Validity date on cert element 2 (R3) is Sep 15 16:00:00 2025 GMT
Date computations: GNU
Computing number of hours until 'Sep 15 16:00:00 2025 GMT' with GNU
Computing '(1757952000-1702227362)/3600' (precision 0)
Hours until Sep 15 16:00:00 2025 GMT: 15479
Computing '15479/24' (precision 0)
Computing '15479 * 3600' (precision 0)
Adding line to prometheus days output: cert_days_chain_elem{cn="www.domain-bkk.de", element="2"} 644
  valid for 644 days
Executing comparison '15479 < 1688' (precision 0)
  bc result = 0
  returning 1
Executing comparison '644 < 70' (precision 0)
  bc result = 0
  returning 1
executing: /usr/bin/openssl x509 -noout -checkend 0 on cert element 2 (R3)
executing: /usr/bin/openssl x509 -noout -checkend 604800 on cert element 2 (R3)
executing: /usr/bin/openssl x509 -noout -checkend 1209600 on cert element 2
Adding line to prometheus validity output: cert_valid_chain_elem{cn="www.domain-bkk.de", element="2"} 0
------------------------------------------------------------------------------
Checking OCSP status of element 2
temporary file /tmp/EPZAGn created
Storing the chain element in /tmp/EPZAGn
Checking revocation via OCSP
extracting cert attribute issuer_hash
Issuer hash: 4042bcee
extracting cert attribute issuer_uri
Chain element issuer URIs: http://x1.i.lencr.org/
checking issuer URIs: http://x1.i.lencr.org/
OCSP: fetching issuer certificate http://x1.i.lencr.org/ to /tmp/I4K3JU
exec_with_timeout: TIMEOUT=120, CURRENT_TIMEOUT=119, ELAPSED=1
exec_with_timeout /usr/bin/curl    --silent --user-agent 'check_ssl_cert/2.76.0' --location \"http://x1.i.lencr.org/\" > /tmp/I4K3JU
executing with timeout (119s): /usr/bin/curl    --silent --user-agent 'check_ssl_cert/2.76.0' --location \"http://x1.i.lencr.org/\" > /tmp/I4K3JU
/usr/bin/timeout 119 /bin/sh -c "/usr/bin/curl    --silent --user-agent 'check_ssl_cert/2.76.0' --location \"http://x1.i.lencr.org/\" > /tmp/I4K3JU"
OCSP: issuer certificate type (1): data
OCSP: issuer certificate type (2): data
OCSP: converting issuer certificate from DER to PEM
OCSP: issuer certificate type (3): PEM certificate
extracting cert attribute oscp_uri
OCSP: URIs =
------------------------------------------------------------------------------
-- Checking element 3
extracting cert attribute cn
Checking expiration date of element 3 (ISRG Root X1)
extracting cert attribute enddate
Validity date on cert element 3 (ISRG Root X1) is Sep 30 18:14:03 2024 GMT
Date computations: GNU
Computing number of hours until 'Sep 30 18:14:03 2024 GMT' with GNU
Computing '(1727720043-1702227363)/3600' (precision 0)
Hours until Sep 30 18:14:03 2024 GMT: 7081
Computing '7081/24' (precision 0)
Computing '7081 * 3600' (precision 0)
Adding line to prometheus days output: cert_days_chain_elem{cn="www.domain-bkk.de", element="3"} 295
  valid for 295 days
Executing comparison '7081 < 1688' (precision 0)
  bc result = 0
  returning 1
Executing comparison '295 < 70' (precision 0)
  bc result = 0
  returning 1
executing: /usr/bin/openssl x509 -noout -checkend 0 on cert element 3 (ISRG Root X1)
executing: /usr/bin/openssl x509 -noout -checkend 604800 on cert element 3 (ISRG Root X1)
executing: /usr/bin/openssl x509 -noout -checkend 1209600 on cert element 3
Adding line to prometheus validity output: cert_valid_chain_elem{cn="www.domain-bkk.de", element="3"} 0
------------------------------------------------------------------------------
Checking OCSP status of element 3
temporary file /tmp/2CWql7 created
Storing the chain element in /tmp/2CWql7
Checking revocation via OCSP
extracting cert attribute issuer_hash
Issuer hash: 2e5ac55d
extracting cert attribute issuer_uri
Chain element issuer URIs: http://apps.identrust.com/roots/dstrootcax3.p7c
checking issuer URIs: http://apps.identrust.com/roots/dstrootcax3.p7c
OCSP: fetching issuer certificate http://apps.identrust.com/roots/dstrootcax3.p7c to /tmp/I4K3JU
exec_with_timeout: TIMEOUT=120, CURRENT_TIMEOUT=118, ELAPSED=2
exec_with_timeout /usr/bin/curl    --silent --user-agent 'check_ssl_cert/2.76.0' --location \"http://apps.identrust.com/roots/dstrootcax3.p7c\" > /tmp/I4K3JU
executing with timeout (118s): /usr/bin/curl    --silent --user-agent 'check_ssl_cert/2.76.0' --location \"http://apps.identrust.com/roots/dstrootcax3.p7c\" > /tmp/I4K3JU
/usr/bin/timeout 118 /bin/sh -c "/usr/bin/curl    --silent --user-agent 'check_ssl_cert/2.76.0' --location \"http://apps.identrust.com/roots/dstrootcax3.p7c\" > /tmp/I4K3JU"
OCSP: issuer certificate type (1): data
OCSP: converting issuer certificate from PKCS #7 to PEM
OCSP: issuer certificate type (2): ASCII text
OCSP: issuer certificate type (3): ASCII text
extracting cert attribute oscp_uri
OCSP: URIs =
------------------------------------------------------------------------------
temporary file /tmp/3KSjEU created
Storing the SSL Labs JSON output to /tmp/3KSjEU
http_proxy  =
HTTPS_PROXY =
executing /usr/bin/curl    --silent "https://api.ssllabs.com/api/v2/analyze?host=www.domain-bkk.de"
exec_with_timeout: TIMEOUT=120, CURRENT_TIMEOUT=118, ELAPSED=2
exec_with_timeout /usr/bin/curl    --silent \"https://api.ssllabs.com/api/v2/analyze?host=www.domain-bkk.de\" > /tmp/3KSjEU
executing with timeout (118s): /usr/bin/curl    --silent \"https://api.ssllabs.com/api/v2/analyze?host=www.domain-bkk.de\" > /tmp/3KSjEU
/usr/bin/timeout 118 /bin/sh -c "/usr/bin/curl    --silent \"https://api.ssllabs.com/api/v2/analyze?host=www.domain-bkk.de\" > /tmp/3KSjEU"
curl return code = 35
curl returned 35: /usr/bin/curl    --silent "https://api.ssllabs.com/api/v2/analyze?host=www.domain-bkk.de"
cleaning up temporary files
 /tmp/
XPOr1 /tmp/NS
DYz /tmp/KHlAdX /tmp/Rfg3kH /tmp/Gtr9pG /tmp/I4K3JU /tmp/1DTQdG /tmp/Nm6nSq /tmp/EPZAGn /tmp/2CWql7 /tmp/3KSjEU
exiting with CRITICAL
ALL_MSG =
number of errors = 0

is this helpful? I'm not seeing something obvious in the debug log :-(

matteocorti commented 7 months ago

The error is SSL/TLS handshake connecting to ssllabs.com (I will commit a new version with a better description).

Anyway, there is not much that can be done on the plugin side. You can always use --ignore-ssl-labs-errors to continue with the checks ignoring SSL Labs.