search

nickfloyd / newrelic-perfmon-plugin

The Perfmon Plugin for the New Relic Plugins - https://newrelic.com/plugins
MIT License
11 stars 22 forks source link

100% cpu and ruby processes spawning up #9

Open ghost opened 11 years ago

ghost commented 11 years ago

Hi there,

We currently had an online session with Scott Jordan and David Yu where we discussed several issues and questions we have with New Relic.

We are eager to get a way to monitor perf. counters and we have therefore used your plugin for a few servers. But we saw some unwanted cpu usage when we deployed the rest of the agents to the app. farm. Both the ruby process and also perf. counter executable went into 100% cpu for good. We see that typeperf.exe uses a lot of cpu when collecting counters but only for a few seconds, this time is was permanent.

Is there a way to optimize this plugin so it can run with less cpu usage. 100% for X servers is a bit too much for us.

Last I want to know if this plugin will be a native feature in New Relic? Performance counters is highly wanted by any Windows Customer.

Thanks, Mads Eneqvist, e-conomic international a/s Copenhagen, Denmark

sschwartzman commented 11 years ago

Hi Mads,

It is being discussed as a native feature, but not yet on any roadmap for release. If I hear otherwise, I'll keep you posted.

About the CPU utilization, I noticed the same thing and so I cut down significantly on the number of simultaneous threads the plugin will spin up. It is currently capped at 5, whereas it used to hover around 20. Have you tried the latest release of the plugin? I posted it last Thursday.

Thanks for your feedback!

--Seth

ghost commented 11 years ago

Hey Seth,

Thanks for your quick reply. I have to say that I did not see the new version before now. I have taken it into review on our preprod servers and so far it’s looking good. The first to runs were a bit cpu heavy but the next ones looks good. No more high spikes.

Next is to test this on a server with far more load than this. That was what went wrong the last time.

Really cool!

Mads

From: Seth Schwartzman [mailto:notifications@github.com] Sent: 9. september 2013 15:11 To: nickfloyd/newrelic-perfmon-plugin Cc: Mads Eneqvist Subject: Re: [newrelic-perfmon-plugin] 100% cpu and ruby processes spawning up (#9)

Hi Mads,

It is being discussed as a native feature, but not yet on any roadmap for release. If I hear otherwise, I'll keep you posted.

About the CPU utilization, I noticed the same thing and so I cut down significantly on the number of simultaneous threads the plugin will spin up. It is currently capped at 5, whereas it used to hover around 20. Have you tried the latest release of the plugin? I posted it last Thursday.

Thanks for your feedback!

--Seth

— Reply to this email directly or view it on GitHubhttps://github.com/nickfloyd/newrelic-perfmon-plugin/issues/9#issuecomment-24072182.

sschwartzman commented 11 years ago

Good deal! I do realize that a native .NET plugin would make more sense here, to be honest this was a learning experience for me w/ Ruby. But it's seen lots of interest so, I've done my best to support it. I really appreciate the feedback. I look forward to hearing your results from running it in the heavily loaded servers.

So what do things look like now in terms of # of processes running (Ruby & Typeperf) and their CPU consumption? You shouldn't see more than 6 Ruby threads (parent + 5 children) and as many Typeperf instances at any given time.

Also, if it works out well for you, please give it a good rating in Plugin Central! :)

Thanks again, Seth

ghost commented 11 years ago

Hey,

We hope that this and the MSSQL plug will be a native feature. But so far we can work with this solution or we have to. It has been enabled on one production server now and looks like the first two spikes are still there with some disk i/o but looks better after 1-2 mins.

I can confirm that the number of spawned processes is less now, 5 typeperf.exe and one ruby. We know all about the Ruby learning curve as we have implemented Chef the last few months.

I have already rated it. 4* so far ☺

Do you have any new features to this plug in or other in the pipeline?

Mads

From: Seth Schwartzman [mailto:notifications@github.com] Sent: 9. september 2013 15:11 To: nickfloyd/newrelic-perfmon-plugin Cc: Mads Eneqvist Subject: Re: [newrelic-perfmon-plugin] 100% cpu and ruby processes spawning up (#9)

Hi Mads,

It is being discussed as a native feature, but not yet on any roadmap for release. If I hear otherwise, I'll keep you posted.

About the CPU utilization, I noticed the same thing and so I cut down significantly on the number of simultaneous threads the plugin will spin up. It is currently capped at 5, whereas it used to hover around 20. Have you tried the latest release of the plugin? I posted it last Thursday.

Thanks for your feedback!

--Seth

— Reply to this email directly or view it on GitHubhttps://github.com/nickfloyd/newrelic-perfmon-plugin/issues/9#issuecomment-24072182.

sschwartzman commented 11 years ago

Right now I'm just working on getting it stable, and re-learning .NET to properly port it. Can you think of other features that I should work in? You are welcome to fork it and add any cool code of your own!

Thanks, Seth

ghost commented 11 years ago

Yeah, I understand. Makes good sense. Don’t know about any other features. Maybe some cool integration with LogStash or other Log Management software. I have actually seen an issue with the perfmon plugin. When I met this morning all nodes which were running the plugin was greyed out. After a restart of the perfmon service it started to send data again. I have attached the logs.

Mads

From: Seth Schwartzman [mailto:notifications@github.com] Sent: 10. september 2013 02:27 To: nickfloyd/newrelic-perfmon-plugin Cc: Mads Eneqvist Subject: Re: [newrelic-perfmon-plugin] 100% cpu and ruby processes spawning up (#9)

Right now I'm just working on getting it stable, and re-learning .NET to properly port it. Can you think of other features that I should work in? You are welcome to fork it and add any cool code of your own!

Thanks, Seth

— Reply to this email directly or view it on GitHubhttps://github.com/nickfloyd/newrelic-perfmon-plugin/issues/9#issuecomment-24125629.

sschwartzman commented 11 years ago

Hi Mads,

The logs don't seem to be attached. Can you email them to me?

My email is seth at newrelic dot com.

Thanks, Seth

ghost commented 11 years ago

Hi Seth,

I let the perfmon plugin run for some days and now we see the issue again where it stops communicating. This is the last few lines of the error log

[2013-09-12 08:30:27 UTC] HTTP Connection Error: #Faraday::Error::ConnectionFailed SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed [2013-09-12 08:30:27 UTC] _ERROR: FAILED: No response [2013-09-12 08:30:27 UTC] Gathered 508 statistics [2013-09-12 08:31:27 UTC] HTTP Connection Error: #Faraday::Error::ConnectionFailed SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed [2013-09-12 08:31:27 UTC] _ERROR: FAILED: No response [2013-09-12 08:31:27 UTC] Gathered 508 statistics [2013-09-12 08:32:27 UTC] HTTP Connection Error: #Faraday::Error::ConnectionFailed SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed [2013-09-12 08:32:27 UTC] ****ERROR: FAILED: No response [2013-09-12 08:32:27 UTC] Gathered 508 statistics [2013-09-12 08:33:27 UTC] HTTP Connection Error: #Faraday::Error::ConnectionFailed SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

After service restart it works again.

Mads

From: Seth Schwartzman [mailto:notifications@github.com] Sent: 10. september 2013 14:37 To: nickfloyd/newrelic-perfmon-plugin Cc: Mads Eneqvist Subject: Re: [newrelic-perfmon-plugin] 100% cpu and ruby processes spawning up (#9)

Hi Mads,

The logs don't seem to be attached. Can you email them to me?

My email is seth at newrelic dot com.

Thanks, Seth

— Reply to this email directly or view it on GitHubhttps://github.com/nickfloyd/newrelic-perfmon-plugin/issues/9#issuecomment-24156046.

sschwartzman commented 11 years ago

I'll have to look into this when I'm back in my office, but we shouldn't see Faraday errors, as it (Faraday) was removed from the Plugin SDK. I'll re-open this ticket and thus keep it on my radar.

Thanks for your patience, Seth

sschwartzman commented 11 years ago

Hi Mads,

Thanks for your patience. I rebuilt the plugin with the latest version of the SDK and ensured that Faraday was removed. I also fixed the SSL Cert issue programmatically.

Please download the latest version here and try it out: https://github.com/nickfloyd/newrelic-perfmon-plugin/blob/master/standalone/perfmon_plugin_standalone.zip

Thanks, Seth

ghost commented 11 years ago

Hi Seth,

Sounds cool. Will update it as soon as I am in the office tomorrow I have seen servers running Server 2012 being more stable than 2008

Mads

From: Seth Schwartzman [notifications@github.com] Sent: Tuesday, September 24, 2013 6:18 PM To: nickfloyd/newrelic-perfmon-plugin Cc: Mads Eneqvist Subject: Re: [newrelic-perfmon-plugin] 100% cpu and ruby processes spawning up (#9)

Hi Mads,

Thanks for your patience. I rebuilt the plugin with the latest version of the SDK and ensured that Faraday was removed. I also fixed the SSL Cert issue programmatically.

Please download the latest version here and try it out: https://github.com/nickfloyd/newrelic-perfmon-plugin/blob/master/standalone/perfmon_plugin_standalone.zip

Thanks, Seth

— Reply to this email directly or view it on GitHubhttps://github.com/nickfloyd/newrelic-perfmon-plugin/issues/9#issuecomment-25019405.

sschwartzman commented 11 years ago

Hi Mads,

Have you had the opportunity to try the newest plugin rev?

Thanks, Seth

sschwartzman commented 11 years ago

Hi Mads,

Just committed a version that should fix the lingering ruby & exe processes at service stop. Please try that out when you have the opportunity. Be sure to run "uninstall_perfmon_plugin_service.bat" and "install_perfrmon_plugin_service.bat" when you install the new version. I'm not positive this is required but better to be safe.

Thanks, Seth

ghost commented 11 years ago

Hi Seth,

Good to hear from you. I have installed the new plugin on two test servers. I will monitor the servers for a couple of days and then I will get back to you.

Mads

From: Seth Schwartzman [mailto:notifications@github.com] Sent: 14. oktober 2013 22:11 To: nickfloyd/newrelic-perfmon-plugin Cc: Mads Eneqvist Subject: Re: [newrelic-perfmon-plugin] 100% cpu and ruby processes spawning up (#9)

Hi Mads,

Just committed a version that should fix the lingering ruby & exe processes at service stop. Please try that out when you have the opportunity. Be sure to run "uninstall_perfmon_plugin_service.bat" and "install_perfrmon_plugin_service.bat" when you install the new version. I'm not positive this is required but better to be safe.

Thanks, Seth

— Reply to this email directly or view it on GitHubhttps://github.com/nickfloyd/newrelic-perfmon-plugin/issues/9#issuecomment-26284862.