SSL_CERT UNKNOWN www.xxx.com: cannot find program: hostname

[RodLew94]

Running from the command line the script runs fine. Running from Nagios it shows the following: 'SSL_CERT UNKNOWN www.xxx.com: cannot find program: hostname'

check_ssl_cert version 2.78.0

Centos7 / Nagios Core v4.4.9

runuser -u nagios -- /usr/lib64/nagios/plugins/check_ssl_cert -H www.xxx.com -w 29 -c 10 -P https

SSL_CERT WARN xxx.com:443: x509 certificate element 1 (xxx.com) will expire in 19 day(s) on Dec 30 23:59:59 2023 GMT |days_chain_elem1=19;29;10;; days_chain_elem2=523;29;10;; days_chain_elem3=1847;29;10;;

[matteocorti]

hostname is not in the PATH.

[miken32]

Hmm not so sure about that resolution. I started getting this error in Icinga after a server IP change and reboot; it had been working for years prior. All my other monitors are working fine. I updated the script error message to unknown "cannot find program: $1 ($PATH)" and the path is indeed correct:

SSL_CERT UNKNOWN example.ca: cannot find program: hostname (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)

Still digging into it...

[matteocorti]

And if you add command -v? I usually suppress the output

[miken32]

There is no output from command -v on either stderr or stdout. When I update the error line to unknown "cannot find program: $1 ($PATH) >$(command -v "$1" 2>&1)<" it returns

SSL_CERT UNKNOWN example.ca: cannot find program: hostname (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin) ><

I also tried command -v -p with no luck. Here's a debug log:

Converting 5 days into seconds by shell function
Converted 5 days into seconds: 432000
Converting 15 days into seconds by shell function
Converted 15 days into seconds: 1296000
check_ssl_cert version: 2.81.1
System info: Linux marceline 5.14.0-284.18.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jun 29 17:06:27 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
/etc/os-release:
User: icinga
Fri 31 May 2024 05:23:31 PM EDT
Shell: /sbin/nologin
grep: /usr/bin/grep
hostname: 
$PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Command line arguments: --debug --debug-file /tmp/ssl_debug.txt -H example.ca -c 5 -p 5061 -w 15
  TMPDIR = /tmp
Required HTTP headers:   
Unrequired HTTP headers: 
curl binary needed. SSL Labs = , OCSP = 1, CURL = , IGNORE_CONNECTION_STATE=, FILE_URI=
curl binary not specified
curl available: /usr/bin/curl
curl 7.76.1 (x86_64-redhat-linux-gnu) libcurl/7.76.1 OpenSSL/3.0.7 zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.1 (+libidn2/2.3.0) libssh/0.10.4/openssl/zlib nghttp2/1.43.0
Release-Date: 2021-04-14
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets
Proxy settings (after):
  http_proxy  = 
  https_proxy = 
  HTTP_PROXY  = 
  HTTPS_PROXY = 
  s_client    =  
  curl        =  
Checking if the host is listed in /etc/hosts
Host not found in /etc/hosts: checking DNS
Checking if the host (example.ca) exists
HOST = example.ca
SNI                 = 
HOST_NAME           = example.ca
HOST_ADDR           = example.ca
NAMES_TO_BE_CHECKED = __HOST__
Checking if example.ca is an IP address
example.ca is not an IP address
HOST_IS_IP.         = 0
Checking if example.ca is an IP address
example.ca is not an IP address
Adding example.ca to NAMES_TO_BE_CHECKED
NAMES_TO_BE_CHECKED = example.ca
-c specified: 5
-w specified: 15
cleaning up temporary files

[miken32]

Re "Converting 5 days into seconds by shell function" I do have Perl installed on this system as well. Not sure why it's able to find /usr/bin/grep without any problem...

[matteocorti]

I see that $SHELL is /sbin/nologin. What happens if you run the script with /bin/sh /YOURPATH/check_ssl_cert?

[miken32]

Same results, it still inherits the icinga user's environment. Though the shell (or lack of) shouldn't be a problem since it still has the correct path, according to the debug output (and the fact that it can find /usr/bin/command to run.)

[matteocorti]

It should not use /usr/bin/command as command is a shell built (defined by POSIX). And I am using command instead of type since type is not POSIX (and could therefore not be there)

[miken32]

Ah ok I thought that was a Bash-only builtin. I did give it a try with both /usr/bin/command -v and /usr/bin/type -P as well with the same results.

I'm at a loss as to what could be stopping this from working. All I did was update IP address by editing /etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network, and then restarted the server. Had been working for years prior to that.

[miken32]

Found the culprit. SELinux was blocking it.

type=AVC msg=audit(1717172179.168:35910): avc:  denied  { getattr } for  pid=711087 comm="check_ssl_cert" path="/usr/bin/hostname" dev="dm-0" ino=17000184 scontext=system_u:system_r:icinga2_t:s0 tcontext=system_u:object_r:hostname_exec_t:s0 tclass=file permissive=0

Just had to reapply contexts to the plugin directory restorecon -rF /usr/lib64/nagios/. No idea how that changed spontaneously; possibly a package update from months ago that didn't get applied until a reboot.