/usr/bin/fping Cannot allocate memory

[zkryakgul]

SUMMARY

I'm using zabbix-docker in my production environment. Recently I realized that error messages in the zabbix-server service's logs. When I check my server memory status I see there is so many available memory. So what causes this errors?

Zabbix-server container log:

zabbix-docker-zabbix-server-1  |    274:20220916:085948.598 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_274.pinger;: [12] Cannot allocate memory
zabbix-docker-zabbix-server-1  |    274:20220916:085950.599 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_274.pinger;: [12] Cannot allocate memory
zabbix-docker-zabbix-server-1  |    274:20220916:085952.600 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_274.pinger;: [12] Cannot allocate memory
zabbix-docker-zabbix-server-1  |    274:20220916:085954.601 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_274.pinger;: [12] Cannot allocate memory

OS / ENVIRONMENT / Used docker-compose files

OS: Ubuntu 20.04 RAM: 64GB RAM Usage: 17GB Docker version: Docker version 19.03.11, build 42e35e61f3 Compose version: v2.10.2 Template file: docker-compose_v3_ubuntu_pgsql_latest.yaml

CONFIGURATION

.env_srv

# ZBX_LISTENIP=
# ZBX_LISTENBACKLOG=
# ZBX_HISTORYSTORAGEURL=http://elasticsearch:9200/ # Available since 3.4.5
# ZBX_HISTORYSTORAGETYPES=uint,dbl,str,log,text # Available since 3.4.5
# ZBX_ALLOWUNSUPPORTEDDBVERSIONS=0 # Available since 6.0.0
# ZBX_DBTLSCONNECT=required # Available since 5.0.0
# ZBX_DBTLSCAFILE=/run/secrets/root-ca.pem # Available since 5.0.0
# ZBX_DBTLSCERTFILE=/run/secrets/client-cert.pem # Available since 5.0.0
# ZBX_DBTLSKEYFILE=/run/secrets/client-key.pem # Available since 5.0.0
# ZBX_DBTLSCIPHER= # Available since 5.0.0
# ZBX_DBTLSCIPHER13= # Available since 5.0.0
# ZBX_AUTOHANODENAME=fqdn # Allowed values: fqdn, hostname. Available since 6.0.0
# ZBX_HANODENAME= # Available since 6.0.0
# ZBX_AUTONODEADDRESS=fqdn # Allowed values: fqdn, hostname. Available since 6.0.0
# ZBX_NODEADDRESSPORT=10051 # Allowed to use with ZBX_AUTONODEADDRESS variable only. Available since 6.0.0
# ZBX_NODEADDRESS=localhost:10051 # Available since 6.0.0
# ZBX_DEBUGLEVEL=3
ZBX_STARTPOLLERS=10
# ZBX_IPMIPOLLERS=0
# ZBX_STARTPREPROCESSORS=3 # Available since 3.4.0
# ZBX_STARTPOLLERSUNREACHABLE=1
# ZBX_STARTTRAPPERS=5
# ZBX_STARTPINGERS=1
# ZBX_STARTDISCOVERERS=1
# ZBX_STARTHTTPPOLLERS=1
# ZBX_STARTHISTORYPOLLERS=5 # Available since 5.4.0
# ZBX_STARTTIMERS=1
# ZBX_STARTESCALATORS=1
# ZBX_STARTALERTERS=3 # Available since 3.4.0
# ZBX_STARTLLDPROCESSORS=2
ZBX_JAVAGATEWAY_ENABLE=true
# ZBX_JAVAGATEWAY=zabbix-java-gateway
# ZBX_JAVAGATEWAYPORT=10052
ZBX_STARTJAVAPOLLERS=5
# ZBX_STARTVMWARECOLLECTORS=0
# ZBX_VMWAREFREQUENCY=60
# ZBX_VMWAREPERFFREQUENCY=60
# ZBX_VMWARECACHESIZE=8M
# ZBX_VMWARETIMEOUT=10
ZBX_ENABLE_SNMP_TRAPS=true
# ZBX_SOURCEIP=
# ZBX_HOUSEKEEPINGFREQUENCY=1
# ZBX_MAXHOUSEKEEPERDELETE=5000
# ZBX_PROBLEMHOUSEKEEPINGFREQUENCY=60
# ZBX_SENDERFREQUENCY=30
ZBX_CACHESIZE=8G
# ZBX_CACHEUPDATEFREQUENCY=60
# ZBX_STARTDBSYNCERS=4
# ZBX_HISTORYCACHESIZE=16M
ZBX_HISTORYINDEXCACHESIZE=512M
# ZBX_HISTORYSTORAGEDATEINDEX=0
ZBX_TRENDCACHESIZE=32M
ZBX_TRENDFUNCTIONCACHESIZE=32M
ZBX_VALUECACHESIZE=400M
# ZBX_TIMEOUT=4
# ZBX_TRAPPERTIMEOUT=300
# ZBX_UNREACHABLEPERIOD=45
# ZBX_UNAVAILABLEDELAY=60
# ZBX_UNREACHABLEDELAY=15
# ZBX_LOGSLOWQUERIES=3000
# ZBX_EXPORTFILESIZE=
# ZBX_STARTPROXYPOLLERS=1
# ZBX_PROXYCONFIGFREQUENCY=3600
# ZBX_PROXYDATAFREQUENCY=1
# ZBX_LOADMODULE="dummy1.so,dummy2.so,dummy10.so"
# ZBX_TLSCAFILE=
# ZBX_TLSCRLFILE=
# ZBX_TLSCERTFILE=
# ZBX_TLSKEYFILE=
# ZBX_VAULTDBPATH=
# ZBX_VAULTURL=https://127.0.0.1:8200
# VAULT_TOKEN=
# ZBX_STARTREPORTWRITERS=0
# ZBX_WEBSERVICEURL=http://zabbix-web-service:10053/report
# ZBX_SERVICEMANAGERSYNCFREQUENCY=60

[dotneft]

I recommend you check allocated memory for the container.

[zkryakgul]

I checked the memory usage of container, Its using only few GB of the 64 GB memory. And the container has no memory limit.

[dotneft]

if you execute the same command (just ping a host using the same command and ping list of hosts from a file) in the container, do you see any issue?

[phlibi]

How has this been resolved? I'm facing the same problem on one Docker installation, but not on another. And apparently this does not affect all configured hosts/IPs, just some of them.

EDIT: Running the fping command using docker exec -it zabbix-docker_zabbix-server_1 bash works just fine. And running free both inside the container and directly on the server gives about 72G available memory, so this certainly isn't the issue. A restart of the containers did not help.

Just in case: This Zabbix instance was upgraded from 5.4 to 6.2 and prior to the upgrade, this problem did not occur. Is there a configuration change that needs to be implemented somewhere?

And I've just seen that I've been mistaken: The icmppingsec is broken for all hosts, but not all triggers have fired, so I've not seen the others.

It looks like this also happened for other people, but wasn't resolved with a fix there either: https://support.zabbix.com/browse/ZBX-21704

What can I do to pin down and fix this issue?

[zkryakgul]

We realized that our host machine had held a lot of broken packages. So we migrate our Zabbix setup to another stable machine. After that issue was resolved.