search

Cacti / cacti

Cacti ™
http://www.cacti.net
GNU General Public License v2.0
1.65k stars 406 forks source link

Monitoring Host CPU's does not always work on Windows endpoints #5492

Closed Yomark1 closed 1 year ago

Yomark1 commented 1 year ago

Describe the bug

Cacti 1.2.25: ss_host_cpu.php CPU graphs not updating anymore Script server CPU graphs aren't updated anymore.

To Reproduce

Steps to reproduce the behavior:

Expected behavior

Working graphs like on older releases

Screenshots

cactissue

Desktop (please complete the following information)

Smartphone (please complete the following information)

Additional context

Yomark1 commented 1 year ago

For the record, I'm using Windows server 2022 as a cacti host. I did fresh installs of 1.2.25 and 1.2.24 on several freshly deployed Windows server 2022 Virtual Machines to validate the issue.

TheWitness commented 1 year ago

Post your cacti log. Put the device into debug mode. What plugins are installed?

Yomark1 commented 1 year ago

Debug log posted on cacti forum:

2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] WARNING: Invalid Response(s), Errors[2] Device[525] Thread[1] DS[7557, 7558]
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] DEBUG: Device[525] HT[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] Total Time: 5.5 Seconds
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7634] TT[254.25] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_workersidle, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".9, value: 4
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7633] TT[220.50] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_workersbusy, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".8, value: 1
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7632] TT[199.86] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_sb_write, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".13, value: 1
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7631] TT[48.55] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_sb_wait, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".10, value: 4
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7630] TT[0.55] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_sb_start, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".11, value: 0
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7629] TT[127.82] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_sb_read, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".12, value: 0
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7628] TT[98.34] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_sb_openslot, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".20, value: 145
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7627] TT[71.28] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_sb_log, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".17, value: 0
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7626] TT[38.50] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_sb_keepalive, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".14, value: 0
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7625] TT[0.48] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_sb_idle, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".19, value: 0
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7624] TT[147.88] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_sb_finish, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".18, value: 0
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7623] TT[78.90] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_sb_dns, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".15, value: 0
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7622] TT[52.14] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_sb_close, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".16, value: 0
2023-09-18 15:00:08 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7621] TT[35.09] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_reqpersec, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".5, value: .003433
2023-09-18 15:00:07 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7620] TT[0.48] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_accesses, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".1, value: 20226
2023-09-18 15:00:07 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7619] TT[116.10] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_workersbusy, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".8, value: 1
2023-09-18 15:00:07 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7618] TT[89.48] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_accesses, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".1, value: 20226
2023-09-18 15:00:07 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7617] TT[60.86] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_cpu, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".3, value: .00348188
2023-09-18 15:00:07 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7616] TT[33.22] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_bytespersec, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".6, value: 10.7575
2023-09-18 15:00:07 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7615] TT[0.55] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_bytes, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".2, value: 63379456
2023-09-18 15:00:07 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7614] TT[121.29] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_bytesperreq, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".7, value: 3133.56
2023-09-18 15:00:07 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7613] TT[93.91] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_bytes, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".2, value: 63379456
2023-09-18 15:00:07 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7612] TT[70.54] SNMP: v2: srvowncloudtest.dommel.local, dsname: apache_accesses, oid: NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".1, value: 20226
2023-09-18 15:00:07 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7560] TT[51.41] SNMP: v2: srvowncloudtest.dommel.local, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.2, value: 2072752694
2023-09-18 15:00:07 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7560] TT[31.37] SNMP: v2: srvowncloudtest.dommel.local, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.2, value: 6849952577
2023-09-18 15:00:07 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7558] TT[979.96] SS[3] SERVER: C:\Apache24\htdocs\cacti\scripts\ss_host_cpu.php ss_host_cpu "srvowncloudtest.dommel.local" "525" "2:161:1000:1:5:public::::::" "get" "usage" "1", output: U
2023-09-18 15:00:06 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7557] TT[1882.23] SS[1] SERVER: C:\Apache24\htdocs\cacti\scripts\ss_host_cpu.php ss_host_cpu "srvowncloudtest.dommel.local" "525" "2:161:1000:1:5:public::::::" "get" "usage" "0", output: U
2023-09-18 15:00:04 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7556] TT[1374.73] SS[3] SERVER: C:\Apache24\htdocs\cacti\scripts\ss_host_disk.php ss_host_disk "srvowncloudtest.dommel.local" "525" "2:161:1000:1:5:public::::::" "get" "used" "31", output: 24333463552
2023-09-18 15:00:03 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7556] TT[20.70] SS[0] SERVER: C:\Apache24\htdocs\cacti\scripts\ss_host_disk.php ss_host_disk "srvowncloudtest.dommel.local" "525" "2:161:1000:1:5:public::::::" "get" "total" "31", output: 54570520576
2023-09-18 15:00:03 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7555] TT[23.68] SS[1] SERVER: C:\Apache24\htdocs\cacti\scripts\ss_host_disk.php ss_host_disk "srvowncloudtest.dommel.local" "525" "2:161:1000:1:5:public::::::" "get" "used" "1", output: 1968664576
2023-09-18 15:00:03 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DS[7555] TT[27.46] SS[2] SERVER: C:\Apache24\htdocs\cacti\scripts\ss_host_disk.php ss_host_disk "srvowncloudtest.dommel.local" "525" "2:161:1000:1:5:public::::::" "get" "total" "1", output: 3137822720
2023-09-18 15:00:03 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] NOTE: There are '31' Polling Items for this Device
2023-09-18 15:00:03 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DQ[9] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 192600 < output: 222500)
2023-09-18 15:00:03 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DQ[8] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 192600 < output: 222500)
2023-09-18 15:00:03 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DQ[1] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 192600 < output: 222500)
2023-09-18 15:00:03 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DQ[1] Extended Uptime Result: 222500, Is Numeric: 1
2023-09-18 15:00:03 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] HT[1] DQ[1] Legacy Uptime Result: 222551, Is Numeric: 1
2023-09-18 15:00:03 - SPINE: Poller[1] PID[310] PT[42950289312] DEBUG: Device[525] HT[1] RECACHE: Processing 3 items in the auto reindex cache for 'srvowncloudtest.dommel.local'
2023-09-18 15:00:03 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] Updating Full System Information Table
2023-09-18 15:00:03 - SPINE: Poller[1] PID[310] PT[42950289312] Device[525] DEBUG: Entering SNMP Ping
2023-09-18 15:00:03 - SPINE: Poller[1] PID[310] PT[42950289312] DEBUG: Device[525] HT[1] In Poller, About to Start Polling

Only thold plugin is installed.

Running it manually from command-line:

C:\Users\Administrator>c:\php\php.exe c:\Apache24\htdocs\cacti\script_server.php
PHP Script Server has Started - Parent is cmd
 C:\Apache24\htdocs\cacti\scripts\ss_host_cpu.php ss_host_cpu "srvowncloudtest.dommel.local" "525" "2:161:1000:1:5:public::::::" "get" "usage" "0"
-t:  (Sub-id not found: (top) -> -t)
U

Error "-t: (Sub-id not found: (top) -> -t)" isn't always shown. Sometimes it just returns "U"

TheWitness commented 1 year ago

It looks as though this has something to do with the SNMP agent. What type of host are you polling there?

See this article. There are many more of course.

https://www.mail-archive.com/net-snmp-users@lists.sourceforge.net/msg16196.html

Yomark1 commented 1 year ago

Hi TheWithness. I already found those posts, because I did some searching on the internet prior to creating the post on the cacti forum and this issue on github.

Like I said in the Cacti forum post, I'm polling all kinds of hosts. Windows(2012-2022), Linux(Ubuntu, CentOS), FreeBSD(pfSense), etc... all those have the same issue.

Take note of the thing that I can easily reproduce it on a freshly installed cacti host: on 1.2.24 it works. When I upgrade to 1.2.25 it breaks. Downgrade to 1.2.24 it works again. Did tests with php7 and php8. .24 works, .25 breaks.
(Maybe an important note, I use Windows as a host for Cacti)

There must be some issue with the changes made from .24 to .25

Here is a screenshot from another test host(srvwin2022test5) I upgraded from .24 to .25 to demonstrate the issue: afbeelding

The srvowncloudtest i'm polling is a server with Ubuntu Linux on it, and as you can see it fails directly after the update and works fine before.

Yomark1 commented 1 year ago

I put back lib\snmp.php from 1.2.24 onto 1.2.25 and now it works. so there is some kind of issue in there.

TheWitness commented 1 year ago

Do you have php-snmp installed?

Yomark1 commented 1 year ago

Yes, from phpinfo():

snmp NET-SNMP Support enabled NET-SNMP Version 5.7.3

Yomark1 commented 1 year ago

About net-snmp binaries"...

Production Server(I upgraded it some time ago when these problems started): C:\net-snmp\bin>snmpwalk -V NET-SNMP version: 5.9.3

On the test server(from screenshot above): C:\net-snmp\bin>snmpwalk.exe -V NET-SNMP version: 5.5

Edit: I'm wondering...nobody has these same issues like me?

Short recap: I can easily reproduce it. Fresh Windows server(This installer: https://forums.cacti.net/viewtopic.php?t=14946 ) . Works fine. Upgrade up and including 1.2.24 all works fine. Then after 1.2.25 the only thing(and I'm polling LOTS different SNMP stuff on production server) that fails is ss_host_cpu.php graphs(query's work fine). And that is then fixed after putting lib\snmp.php from 1.2.24 back (nothing else breaks from putting that old file back as it seems).

TheWitness commented 1 year ago

Ah, Windows, so much fun. Disable php-snmp and see if the released binary works. Also, what PHP version are you using? I'm guessing the latest.

Yomark1 commented 1 year ago

Issue is happening on 7.4.27, 8.1.16 and 8.1.21.

Disabled snmp php extension. Restarted apache to verify if the extension is indeed disabled (it is).

No difference in this regard.

Works with lib/snmp.php from .24, fails with /lib/snmp.php from .25. Also from the command line: afbeelding

TheWitness commented 1 year ago

Well, first of all, let's call it right...

[root@vmhost5 cacti]# php -q script_server.php
PHP Script Server has Started - Parent is cmd
/var/www/html/cacti/scripts/ss_host_cpu.php ss_host_cpu "192.168.11.105" "2" "2:161:2000:1:10:public::::::" "get" "usage" "4000"
1.0000
quit
[root@vmhost5 cacti]#
TheWitness commented 1 year ago

Okay, I was able to reproduce after setting up a Windows Cacti server.

TheWitness commented 1 year ago

@Yomark1, update to the latest 1.2.x branch of lib/snmp.php and the issue will be resolved. Thanks for being patient. It's hard for me to install on Windows any more. Like 23 years of bad memories ;). But it's okay.

Yomark1 commented 1 year ago 
@Yomark1, update to the latest 1.2.x branch of `lib/snmp.php` and the issue will be resolved. Thanks for being patient. It's hard for me to install on Windows any more. Like 23 years of bad memories ;). But it's okay.

Great! Thank you for your help. Sorry I had to put you through this. :) We're using Cacti for a long time. we created a lot of WMI/vbscript scripts to monitor al kinds of windows stuff back in the day when WMI stuff/monitoring from Linux was not great. This is why we still use Windows.

Edit: Seems to work!

Downloaded file from github. Had to change these lines though: if ($config['php_snmp_support']) { include_once(CACTI_PATH_INCLUDE . '/vendor/phpsnmp/extension.php'); } else { include_once(CACTI_PATH_INCLUDE . '/vendor/phpsnmp/classSNMP.php'); } to if ($config['php_snmp_support']) { include_once($config['include_path'] . '/vendor/phpsnmp/extension.php'); } else { include_once($config['include_path'] . '/vendor/phpsnmp/classSNMP.php'); }

But maybe that is due to Github stuff

TheWitness commented 1 year ago

Hehe, yea the WMI stuff kind of does not work from Linux the way I wanted it to. I created the WMI plugin, but for the life of me, I don't know how to make the darn thing open up permissions to use the old protocol in Windows 10 and now 11 any longer. The old Linux binary is using the old v1 protocol. This is why we are going to make Telegraf an option. Soon I hope.

netniV commented 1 year ago

Just for reference, the CACTI_PATH_INCLUDE constant is how we are doing things going forward from 1.3.x. It does mean that if you see any of that in 1.2.x, it won't work. Because the scripts are shared between the systems, they may need to do a bit of hybrid work at seeing which they should be using.