ARK status awk issue with zabbix-agent2

jurim76 commented 4 years ago

Debian default mawk doesn't support zfs.arcstats - "Value of type "string" is not suitable for value type "Numeric (unsigned)". Value "awk: program limit exceeded: maximum number of fields size=32767 FILENAME="/proc/spl/kstat/zfs/arcstats" FNR=48 NR=48"

ARC stats from /proc/spl/kstat/zfs/arcstats UserParameter=zfs.arcstats[*],awk '/^$1/ {printf $$3;}' /proc/spl/kstat/zfs/arcstats

Gawk has no such limitation, but prints nothing while running "gawk '/^misses/ {printf $$3;}' /proc/spl/kstat/zfs/arcstats" (with two dollar signs) "gawk '/^misses/ {printf $3;}' /proc/spl/kstat/zfs/arcstats" returns 1021618956 integer as expected (with one dollar sign)

Proxmox 5 with Debian 9 stretch, gawk 4.1.4+dfsg-1, mawk 1.3.3-17+b3

AceSlash commented 4 years ago

This is very strange because this userparameter was never changed and is several years old. It worked fine on Debian 9.

I will have to install a Debian 9 to test it but I suppose the issue is somehow specific to Proxmox since it didn't happen on Debian 9 before.

stefanux commented 4 years ago

this is no proxmox issue, this error exists on vanilla debian 9 (and on debian 10).

actually i take that back, it works perfectly with default mawk. thanks a lot for the template!

AceSlash commented 4 years ago

I don't see the issue on any Debian 10 and I also tested it on Debian 9, it works.

Could you please do this:

uname -a
cat /etc/os-release 
sudo -u zabbix zabbix_agentd -t zfs.arcstats[hits]
sudo -u zabbix zabbix_agentd -t zfs.arcstats[miss]
dpkg -l|grep awk

For reference, this is what I have on one of my last Debian 9:

root@srv-201:~ # uname -a
Linux srv-201 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1 (2018-04-29) x86_64 GNU/Linux
root@srv-201:~ # cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@srv-201:~ # sudo -u zabbix zabbix_agentd -t zfs.arcstats[hits]
zfs.arcstats[hits]                            [t|203642662586]
root@srv-201:~ # sudo -u zabbix zabbix_agentd -t zfs.arcstats[miss]
zfs.arcstats[miss]                            [t|40278493990]
root@srv-201:~ # dpkg -l|grep awk
ii  mawk                                1.3.3-17+b3                             amd64        a pattern scanning and text processing language

And on a Debian 10:

root@srv-328:~ # uname -a
Linux srv-328 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux
root@srv-328:~ # cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@srv-328:~ # sudo -u zabbix zabbix_agentd -t zfs.arcstats[hits]
zfs.arcstats[hits]                            [t|290778]
root@srv-328:~ # sudo -u zabbix zabbix_agentd -t zfs.arcstats[miss]
zfs.arcstats[miss]                            [t|33]
root@srv-328:~ # dpkg -l|grep awk
ii  mawk                           1.3.3-17+b3                  amd64        a pattern scanning and text processing language

I'm monitoring quite a lot of Debian so I'm confident that it works. I would like to reproduce your issue but I don't see how.

AceSlash commented 4 years ago

Actually, could you try to replace awk by the absolute path to mawk?

 UserParameter=zfs.arcstats[*],/usr/bin/mawk '/^$1/ {printf $$3;}' /proc/spl/kstat/zfs/arcstats

Also check your alternative config:

update-alternatives --list awk

The issue may just be that gawk takes over mawk, if that's the case, using mawk directly would fix this, but it may not be possible for bsd systems to use that.

jurim76 commented 4 years ago

@AceSlash take a look at awk print "$$"

lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 9.13 (stretch) Release: 9.13 Codename: stretch

update-alternatives --list awk /usr/bin/mawk

1) awk print with double "$$" /usr/bin/mawk '/^mfu_size/ {printf $$3;}' /proc/spl/kstat/zfs/arcstats mawk: program limit exceeded: maximum number of fields size=32767 FILENAME="/proc/spl/kstat/zfs/arcstats" FNR=54 NR=54

awk '/^mfu_size/ {printf $$3;}' /proc/spl/kstat/zfs/arcstats

mawk: program limit exceeded: maximum number of fields size=32767 FILENAME="/proc/spl/kstat/zfs/arcstats" FNR=54 NR=54

2) awk print with single "$" /usr/bin/mawk '/^mfu_size/ {printf $3;}' /proc/spl/kstat/zfs/arcstats 3979592704

awk '/^mfu_size/ {printf $3;}' /proc/spl/kstat/zfs/arcstats

3979592704

AceSlash commented 4 years ago

The double $$ is used by the zabbix agent to actually only use a single $. It's because zabbix uses the single $ for its own macro.

So UserParameter=zfs.arcstats[*],awk '/^$1/ {printf $$3;}' /proc/spl/kstat/zfs/arcstats doesn't actually execute awk with double $$. It does this instead:

awk '/^[ZABBIX MACRO RESULT 1]/ {printf $3;}' /proc/spl/kstat/zfs/arcstats

You will find the relevant documentation here: https://www.zabbix.com/documentation/current/manual/config/items/userparameters#flexible_user_parameters

jurim76 commented 4 years ago

The double $$ is used by the zabbix agent to actually only use a single $. It's because zabbix uses the single $ for its own macro.

Ok, got it. Double $$ is valid for legacy zabbix-agent, but I'm using zabbix-agent2 and double $$ is not required in this case.

AceSlash commented 4 years ago

Okay, that's interesting! I'm still using the legacy agent everywhere. If that's the source of the error, I'll have to create an other file for zabbix-agent2. I will have to install it on at least one server and test it first though.

gador commented 3 years ago

Just wanted to chime in: I also have the exact same problem (Debian 10 with agent2) which got solved by removing the double "$" sign.

pwfels commented 3 years ago

Just wanted to chime in: I also have the exact same problem (Debian 10 with agent2) which got solved by removing the double "$" sign.

same here! Thanks for help!

mjanssens commented 3 years ago

Just to let you know. I corrected the conf file to single $. It was running fine, zabbix-agent2, version 5.0.7 I just upgraded to version 5.0.8 and my arc stats didn't give any data. So I set it back to $$ and arc stats is working again.

Edit: I changed all $$ back to the state of conf file in this repo, now the vdev state and errors data is working again too.

jurim76 commented 3 years ago

Seems that zabbix team reversed old behaviour with double $$ in recent zabbix-agent2 releases

Cosium / zabbix_zfs-on-linux

ARK status awk issue with zabbix-agent2 #17