search

Cacti / cacti

Cacti ™
http://www.cacti.net
GNU General Public License v2.0
1.64k stars 405 forks source link

1.2.18 Remote Pollers #4350

Closed eschoeller closed 3 years ago

eschoeller commented 3 years ago

Remote pollers are not running well under 1.2.18. Seeing several issues, most notably pollers are exceeding their runtime and I have no way to increase the log verbosity on the remote pollers. When I login to the remote poller and attempt to increase the log_verbosity in the UI, it changes the logging verbosity on the main poller. Behavior we've seen before. So I went into the remote poller's database and directly changed the log_verbosity ... still no dice the poller won't log any detailed information, so I can't really figure out what's wrong with the pollers and why they keep timing out.

I also have one remote poller that keeps spewing out this every run:

2021/07/19 15:43:27 - SPINE: Poller[6] PID[46135] ERROR: SQL Failed! Error:'1364', Message:'Field 'monitor_text' doesn't have a default value', SQL Fragment:'INSERT INTO host (id, snmp_sysDescr, snmp_sysObjectID, snmp_sysUpTimeInstance, snmp_sysContact, snmp_sysName, snmp_sysLocation, status, status_event_count, status_fail_date, status_rec_date, status_last_error, min_time, max_time, cur_time, avg_time, polling_time, total_polls, failed_polls, availability, last_updated) VALUES  (269, 'APC Web/SNMP Management Card (MB:v4.1.1 PF:v3.7.5 PN:apc_hw03_aos_375.bin AF1:v3.7.5 AN1:apc_hw03_acrp_375.bin
...
bmfmancini commented 3 years ago

Are you running the monitor plugin ? Looks like it might be giving you those errors I remember seeing something similar in our lab

On Mon, Jul 19, 2021 at 5:45 PM Eric Schoeller @.***> wrote:

Remote pollers are not running well under 1.2.18. Seeing several issues, most notably pollers are exceeding their runtime and I have no way to increase the log verbosity on the remote pollers. When I login to the remote poller and attempt to increase the log_verbosity in the UI, it changes the logging verbosity on the main poller. Behavior we've seen before. So I went into the remote poller's database and directly changed the log_verbosity ... still no dice the poller won't log any detailed information, so I can't really figure out what's wrong with the pollers and why they keep timing out.

I also have one remote poller that keeps spewing out this every run:

2021/07/19 15:43:27 - SPINE: Poller[6] PID[46135] ERROR: SQL Failed! Error:'1364', Message:'Field 'monitor_text' doesn't have a default value', SQL Fragment:'INSERT INTO host (id, snmp_sysDescr, snmp_sysObjectID, snmp_sysUpTimeInstance, snmp_sysContact, snmp_sysName, snmp_sysLocation, status, status_event_count, status_fail_date, status_rec_date, status_last_error, min_time, max_time, cur_time, avg_time, polling_time, total_polls, failed_polls, availability, last_updated) VALUES (269, 'APC Web/SNMP Management Card (MB:v4.1.1 PF:v3.7.5 PN:apc_hw03_aos_375.bin AF1:v3.7.5 AN1:apc_hw03_acrp_375.bin ...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Cacti/cacti/issues/4350, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADGEXTE2WVSI7P54LFUTH73TYSMIPANCNFSM5AUQ4VHQ .

-- Thank you

Sean Mancini,(Six Sigma LBBIT®, ITIL,CEA-IT®,SCRUM SMPC®) Owner/Principal Engineer www.seanmancini.com

“Companies spend millions of dollars on firewalls, encryption, and secure access devices, and it’s money wasted because none of these measures address the weakest link in the security chain.”

– Kevin Mitnick

eschoeller commented 3 years ago

Nope, no plugins loaded over here: image

I'm going to pull all the devices off that poller and then add them back in in groups or just keep trying random troubleshooting ideas ... but I could really use some help figuring out how to get the logging verbosity increased on remote pollers. Seems like no matter what I do they're stuck on LOW. Thinking I might have to hook up strace to spine to see if there's anything I can glean from that.

netniV commented 3 years ago

The problem is that most of the settings are being pushed from the main poller. So, you have to update it on the main which updates all the pollers too. It would be an enhancement request to implement an override for that on the pollers.

eschoeller commented 3 years ago

Yeah, here's what I've tried:

  1. Set the log verbosity on the main poller -> main poller now logs at that verbosity. remote pollers do not.
  2. Set the log verbosity on the remote poller through the UI -> no change. Main and Remote poller still use regular verbosity
  3. Set log_verbosity manually in the database on the remote poller -> no change. remote poller still logs at regular verbosity

I figured work-around #3 would have done the trick ... but it didn't. So the remote pollers are having a lot of trouble with logging.

eschoeller commented 3 years ago

Any help on this? I'm really running into some problems figuring out what's wrong with these remote pollers since I can't get any logging out of them.

bmfmancini commented 3 years ago

I can take a peek if you want? Maybe a skype session or something?

On Thu., Aug. 5, 2021, 21:23 Eric Schoeller, @.***> wrote:

Any help on this? I'm really running into some problems figuring out what's wrong with these remote pollers since I can't get any logging out of them.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Cacti/cacti/issues/4350#issuecomment-893933552, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADGEXTFQFPMGNG27EYTZQVLT3M2QTANCNFSM5AUQ4VHQ .

eschoeller commented 3 years ago

Sure when would work for you?

bmfmancini commented 3 years ago

What timezone are you in?

On Thu., Aug. 5, 2021, 21:27 Eric Schoeller, @.***> wrote:

Sure when would work for you?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Cacti/cacti/issues/4350#issuecomment-893934961, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADGEXTDKMYGRMLMP5CHHZALT3M27JANCNFSM5AUQ4VHQ .

eschoeller commented 3 years ago

US/Mountain

bmfmancini commented 3 years ago

Shoot me a email we will set it up

On Thu., Aug. 5, 2021, 21:28 Eric Schoeller, @.***> wrote:

US/Mountain

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Cacti/cacti/issues/4350#issuecomment-893935418, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADGEXTBWA5FV7GQ3GS4FGM3T3M3D7ANCNFSM5AUQ4VHQ .

eschoeller commented 3 years ago

Will do!

eschoeller commented 3 years ago

OK, I actually figured out the problem. Stupid configuration error on my end. The remote pollers (spine.conf) didn't have access to the main poller database.