check_vmware_esx.pl triggered the segmentation fault on CentOS 7.4

Hi @BaldMansMojo

Thanks for the help last time.

I have been experiencing this issue for some time since keep my centos 7 update to 7.4. Could it because the older version vmware perl vdk 5.x (https://github.com/BaldMansMojo/check_vmware_esx/issues/91) i am using ?

Do you know a way to trace the root cause ?

I am reading https://www.cyberciti.biz/tips/segmentation-fault-on-linux-unix.html and hope to come up with more details where are those segfault coming from.

I can see many nagios process running like following from ps output.

 perl /usr/lib64/nagios/plugins/contrib/check_vmware_esx

ssllib info

[me@nagios01 servers]$ rpm -qa |grep openssl-lib
openssl-libs-1.0.2k-8.el7.x86_64
[me@nagios01 servers]$ find /usr/lib/ -iname 'libssl*'
/usr/lib/vmware-vcli/lib32/libssl.so.0.9.8
[me@nagios01 servers]$ find /usr/lib64/ -iname 'libssl*'
/usr/lib64/libssl3.so
/usr/lib64/libssl.so.10
/usr/lib64/libssl.so.1.0.2k
/usr/lib64/pkgconfig/libssl.pc
/usr/lib64/libssl.so
[me@nagios01 servers]$

dmesg output

[me@nagios031~]$ uname -r
3.10.0-693.2.2.el7.x86_64
[me@inagios01 ~]$ dmesg
<snipped>
[1380763.505799]  nfs_acl lockd grace tcp_htcp sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi drm mptscsih mptbase ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
[1380763.505802] CPU: 7 PID: 25758 Comm: postqueue Tainted: G             L ------------   3.10.0-693.2.2.el7.x86_64 #1
[1380763.505803] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
[1380763.505804] task: ffff880233b15ee0 ti: ffff88022b238000 task.ti: ffff88022b238000
[1380763.505808] RIP: 0010:[<ffffffff816ab865>]  [<ffffffff816ab865>] _raw_spin_unlock_irqrestore+0x15/0x20
[1380763.505809] RSP: 0018:ffff88022b23b7e0  EFLAGS: 00000246
[1380763.505810] RAX: 0000000000000246 RBX: ffffffff811e20b8 RCX: ffffc90000c6c000
[1380763.505811] RDX: 00000000f00d0001 RSI: 0000000000000246 RDI: 0000000000000246
[1380763.505811] RBP: ffff88022b23b7e0 R08: ffffffffc00550c0 R09: 0000000000000000
[1380763.505812] R10: ffff88021c3e1410 R11: 0000000000000000 R12: ffff88023ffd8008
[1380763.505813] R13: 0000000000000000 R14: 0000000000000002 R15: 000000002b23b804
[1380763.505814] FS:  00007fca20c00840(0000) GS:ffff88023fdc0000(0000) knlGS:0000000000000000
[1380763.505815] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1380763.505816] CR2: 00007fca1e7b28e0 CR3: 00000001baeba000 CR4: 00000000000007e0
[1380763.505849] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1380763.505886] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1380763.505886] Stack:
[1380763.505890]  ffff88022b23b808 ffffffffc005452e ffff88021c3e1350 ffff880233630000
[1380763.505892]  ffff880233630000 ffff88022b23b828 ffffffff815816fe ffff88021c3e134c
[1380763.505893]  ffff880229612800 ffff88022b23b850 ffffffff816a6119 ffff880233630000
[1380763.505893] Call Trace:
[1380763.505914]  [<ffffffffc005452e>] vmxnet3_get_stats64+0x3e/0x1a0 [vmxnet3]
[1380763.505923]  [<ffffffff815816fe>] dev_get_stats+0x6e/0x150
[1380763.505931]  [<ffffffff816a6119>] rtnl_fill_stats+0x46/0x126
[1380763.505946]  [<ffffffff8159b080>] rtnl_fill_ifinfo+0x540/0xbb0
[1380763.505954]  [<ffffffff811a31b8>] ? zone_statistics+0x88/0xa0
[1380763.505961]  [<ffffffff8134c1b2>] ? nla_parse+0x32/0x120
[1380763.505963]  [<ffffffff8159d97e>] rtnl_dump_ifinfo+0x16e/0x260
[1380763.505970]  [<ffffffff815bb0e7>] netlink_dump+0x117/0x2e0
[1380763.505973]  [<ffffffff815bba8b>] __netlink_dump_start+0x1ab/0x230
[1380763.505974]  [<ffffffff8159d810>] ? rtnetlink_put_metrics+0x1b0/0x1b0
[1380763.505977]  [<ffffffff81599cb0>] rtnetlink_rcv_msg+0x160/0x270
[1380763.505978]  [<ffffffff8159d810>] ? rtnetlink_put_metrics+0x1b0/0x1b0
[1380763.505980]  [<ffffffff81599b50>] ? rtnetlink_rcv+0x30/0x30
[1380763.505982]  [<ffffffff815bd949>] netlink_rcv_skb+0xa9/0xc0
[1380763.505984]  [<ffffffff81599b48>] rtnetlink_rcv+0x28/0x30
[1380763.505986]  [<ffffffff815bd032>] netlink_unicast+0xf2/0x1b0
[1380763.505988]  [<ffffffff815bd40f>] netlink_sendmsg+0x31f/0x6a0
[1380763.505998]  [<ffffffff8156a580>] sock_sendmsg+0xb0/0xf0
[1380763.506001]  [<ffffffff816aba0e>] ? _raw_spin_unlock_bh+0x1e/0x20
[1380763.506003]  [<ffffffff8156e998>] ? release_sock+0x118/0x170
[1380763.506005]  [<ffffffff8156a731>] SYSC_sendto+0x121/0x1c0
[1380763.506008]  [<ffffffff81567340>] ? sock_alloc_file+0xa0/0x140
[1380763.506014]  [<ffffffff81220335>] ? __fd_install+0x25/0x60
[1380763.506017]  [<ffffffff8156b2ce>] SyS_sendto+0xe/0x10
[1380763.506019]  [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
[1380763.506035] Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48
[1380763.870958]  ffff8802247d3e68 ffffffff816a8fad ffff880233bfdee0 ffff8802247d3fd8
[1380763.884470]  ffff8802247d3fd8 ffff8802247d3fd8 ffff880233bfdee0 ffff880233bfdee0
[1380763.886890] Call Trace:
[1380763.889734]  [<ffffffff816a8fad>] __schedule+0x39d/0x8b0
[1380763.890227]  [<ffffffff816a94e9>] schedule+0x29/0x70
[1380763.890974]  [<ffffffff816a84c6>] do_nanosleep+0x86/0xf0
[1380763.891716]  [<ffffffff810b55cb>] hrtimer_nanosleep+0xbb/0x180
[1380763.892383]  [<ffffffff810b4370>] ? hrtimer_get_res+0x50/0x50
[1380763.894975]  [<ffffffff816a84b2>] ? do_nanosleep+0x72/0xf0
[1380763.895658]  [<ffffffff810b570e>] SyS_nanosleep+0x7e/0x90
[1380763.896087]  [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
[1380763.896675] Code: 8b 36 66 66 66 66 90 65 48 8b 34 25 00 ce 00 00 66 66 66 66 90 41 c7 45 28 00 00 00 00 48 89 df c6 07 00 66 66 66 90 fb 66 66 90 <66> 66 90 65 48 8b 04 25 00 ce 00 00 48 8b 98 78 01 00 00 48 85
[2015044.566909] perl[16982]: segfault at 18 ip 00007faf72b173b0 sp 00007ffec8b72e58 error 4 in libssl.so.1.0.2k[7faf72ad2000+67000]
[2126744.586768] perl[21150]: segfault at 18 ip 00007f6b1900c3b0 sp 00007ffd601e9ed8 error 4 in libssl.so.1.0.2k[7f6b18fc7000+67000]
[2261587.237014] sched: RT throttling activated
[2261587.566415] perl[27754]: segfault at 18 ip 00007f601b1383b0 sp 00007fff44fa0938 error 4 in libssl.so.1.0.2k[7f601b0f3000+67000]
[3290673.469103] perl[26240]: segfault at 18 ip 00007fde6d9153b0 sp 00007ffd2819b9a8 error 4 in libssl.so.1.0.2k[7fde6d8d0000+67000]
[3999156.810761] perl[25018]: segfault at 18 ip 00007fa5fed5a3b0 sp 00007ffd1c340b78 error 4 in libssl.so.1.0.2k[7fa5fed15000+67000]
[me@nagios01 ~]$

BaldMansMojo / check_vmware_esx

check_vmware_esx.pl triggered the segmentation fault on CentOS 7.4 #133