search

newrelic / nri-perfmon

Windows Perfmon / WMI On-Host Integration for New Relic Infrastructure
Other
14 stars 18 forks source link

Excessive amount of nri-perfmon processes running #25

Closed hpapazov closed 3 years ago

hpapazov commented 4 years ago

Hi, Is it normal to have 20+ nri-perfmon processes running on a given host? Recently we experienced memory issues with couple of our clients' web server hosts. While investigating, our support team discovers 20+ nri-perfmon running there. They told us that it looked like the processes were hung and continually spawned off a new process.

nri-perfmon version: 0.5.1/nri-perfmon-release-x64.zip

Thanks!

sschwartzman commented 4 years ago

That's a new one! Can you send over a couple things:

  1. The config file for counters you're using (default file is config.json)
  2. Look in Windows Event Viewer for any events from nri-perfmon (that should be the "source" name). If you find any, please use the "copy" button in them and paste into a text doc.

We may also want to run nri-perfmon in verbose mode at command line for a minute and send over its output. That's explained here. I'd say this is a next step though, the current config and log messages are first step.

hpapazov commented 4 years ago

Our support team did not find anything in the Windows Event Viewer. This is the config: { "counterlist": [ { "provider": "Counters", "category": "SMBClientShares", "counters": [ { "counter": "*" } ] } ] }

And this is the command `name: com.newrelic.perfmon description: Perfmon On-Host Integration protocol_version: 1 os: windows commands: metrics: command:

sschwartzman commented 4 years ago

@hpapazov I don't see anything glaring, can you try the latest version of nri-perfmon to see if it happens again? I added some cleanup of old data after each cycle, which may help in case the service is quietly running out of memory.

https://github.com/newrelic/nri-perfmon/releases/tag/0.5.2-alpha

Also can you try verbose mode on one host to see what it outputs? https://github.com/newrelic/nri-perfmon#verbose-logging-mode

sschwartzman commented 3 years ago

Any update @hpapazov ?

Also to note, the latest release is now this one: https://github.com/newrelic/nri-perfmon/releases/tag/0.6.1 I suggest trying it and letting me know if you still see the issue.

hpapazov commented 3 years ago

Sorry @sschwartzman, our operations team did not have the chance to try the above. For the time being they just disabled it. I am sorry, at this point I do not have any additional information that might be helpful.