Librenms can't ping host - but container can

[loopism]

I have an odd issue where, when adding a host that is not on the local LAN subnet via the web UI or command-line, LibreNMS tells me that it can't ping the host. However I can ping and snmpwalk the same host from the container, and DNS resolution works as well.

If I force LibreNMS to perform no ping or snmp checks the host is added, but no data is polled as the host is always considered down. Interestingly running a snmp debug shows that LibreNMS is able to get data from the host.

root@librenms:/# cat /etc/resolv.conf 
nameserver      8.8.8.8
nameserver      8.8.4.4
root@librenms:/# ping <myhost>
PING <myhost> (104.x.x.x) 56(84) bytes of data.
64 bytes from <myhost> (104.x.x.x): icmp_seq=1 ttl=51 time=147 ms
64 bytes from <myhost> (104.x.x.x): icmp_seq=1 ttl=115 time=147 ms (DUP!)
64 bytes from <myhost> (104.x.x.x): icmp_seq=2 ttl=51 time=150 ms
64 bytes from <myhost> (104.x.x.x): icmp_seq=2 ttl=115 time=150 ms (DUP!)
^C
--- <myhost> ping statistics ---
3 packets transmitted, 2 received, +2 duplicates, 33% packet loss, time 2003ms
rtt min/avg/max/mdev = 147.889/149.158/150.466/1.256 ms
root@librenms:/# snmpwalk -v2c -c community <myhost> .1
iso.3.6.1.2.1.1.1.0 = STRING: "Linux zip 2.6.32-042stab120.20 #1 SMP Fri Mar 10 16:52:50 MSK 2017 
x86_64"
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.8072.3.2.10
iso.3.6.1.2.1.1.3.0 = Timeticks: (16133619) 1 day, 20:48:56.19
iso.3.6.1.2.1.1.4.0 = STRING: "R"
iso.3.6.1.2.1.1.5.0 = STRING: "zip"

[jarischaefer]

Perhaps it is similar to https://github.com/jarischaefer/docker-librenms/issues/6? Could you please post the output of SHOW VARIABLES LIKE 'sql_mode'?

[loopism]

Initially it was :

MariaDB [(none)]> SHOW VARIABLES LIKE 'sql_mode';
+---------------+--------------------------------------------+
| Variable_name | Value                                      |
+---------------+--------------------------------------------+
| sql_mode      | NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
+---------------+--------------------------------------------+
1 row in set (0.00 sec)

I've since set sql_mode = "" in my.cnf but still can not add the host:

MariaDB [(none)]> SHOW VARIABLES LIKE 'sql_mode';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| sql_mode      |       |
+---------------+-------+
1 row in set (0.00 sec)

root@librenms:/opt/librenms# ./addhost.php <myhost> <mycommunity> v2c
Could not ping <myhost>

[jarischaefer]

What happens if you use fping inside the container? Could you post the output of running ls -al /usr/bin/fping in the container and docker info on the host?

[loopism]

root@librenms:/# ls -al /usr/bin/fping
-rwsr-xr-x 1 root root 31632 Oct 27  2015 /usr/bin/fping
root@librenms:/# su - librenms
$ /usr/bin/fping <myhost>
<myhost> is alive

tank# docker info
Containers: 8
Images: 317
Storage Driver: aufs
 Root Dir: /volume1/@docker/aufs
 Backing Filesystem: extfs
 Dirs: 333
 Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.10.35
Operating System: <unknown>
CPUs: 2
Total Memory: 3.864 GiB
Name: tank
ID: HYKY:V2HE:TKPG:5EIU:WM4P:JBTG:FGOH:THFC:STY3:KMTY:UTGU:MMGM
WARNING: No memory limit support
WARNING: No swap limit support

This container is running on a standard PC running Synology DSM (XPenology)

As a comparison, I ran up a container running Observium and it is able to add that host. The database is used by both Librenms and Observium

[jarischaefer]

The database is used by both Librenms and Observium

The MySQL instance or database?

As far as I can tell from the code, it uses fping's exit code and loss to determine reachability. I suggest replacing /usr/bin/fping with the following snippet:

#!/bin/sh

echo $@ > /tmp/debug.txt

Next, try adding the device again. Afterwards, you should find the arguments passed to fping in /tmp/debug.txt. Try running fping manually using these arguments.

[loopism]

The MySQL instance or database?

Instance - each have their own database:

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| librenms           |
| mysql              |
| observium          |
| performance_schema |
+--------------------+
5 rows in set (0.02 sec)

I replaced /usr/bin/fping as suggested, but nothing was logged in /tmp/debug.txt when I tried adding the host, it's like addhost.php isn't even trying to ping and just getting hung up somewhere.

I can override the ping and snmp requirement and add the host that way, then go in and turn off ICMP polling for the host

[jarischaefer]

Could you try setting up another LibreNMS and SNMPd instance from scratch (new database)? If you are familiar with PHP, you could temporarily modify the relevant scripts inside the container for debugging purposes.