search

newrelic / nri-perfmon

Windows Perfmon / WMI On-Host Integration for New Relic Infrastructure
Other
13 stars 19 forks source link

Perfmon Counters ASP.NET Apps v4.0.30319 #10

Closed mvaanand closed 4 years ago

mvaanand commented 4 years ago

I updated config.json with the following and I'm unable to see ASP.NET Apps v4.0.30319 in the Insights Events:

{ "provider": "PerfCounter", "category": "ASP.NET Apps v4.0.30319", "counters": [ { "counter": "Requests Executing" }, { "counter": "Requests Failed" }, { "counter": "Requests In Application Queue" }, { "counter": "Requests Timed Out" }, { "counter": "Requests Total" }, { "counter": "Request Execution Time" }, { "counter": "Request Wait Time" } ]

These are the Perfmon Counters I'm trying to set up:

\ASP.NET Apps v4.0.30319(Total)\Requests Executing \ASP.NET Apps v4.0.30319(Total)\Requests Failed \ASP.NET Apps v4.0.30319(Total)\Requests In Application Queue \ASP.NET Apps v4.0.30319(Total)\Requests Timed Out \ASP.NET Apps v4.0.30319(Total)\Requests Total \ASP.NET Apps v4.0.30319(Total)\Request Execution Time \ASP.NET Apps v4.0.30319(Total)\Request Wait Time

Any help is appreciated.

mvaanand commented 4 years ago

I'm seeing the following error:

2019-10-15T17:03:04-04:00" level=error msg="Integration command failed" error="exit status 3762504530" instance=nri-perfmon-metrics integration=com.newrelic.perfmon prefix=integration/nri-perfmon stderr="\nUnhandled Exception: System.InvalidOperationException: Could not locate Performance Counter with specified category name 'ASP.NET Apps v4.0.30319', counter name 'Requests Executing, Requests Failed, Requests In Application Queue, Requests Timed Out, Requests Total, Request Execution Time, Request Wait Time'.\r\n at System.Diagnostics.PerformanceCounter.InitializeImpl()\r\n at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, String instanceName, Boolean readOnly)\r\n at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, String instanceName)\r\n at NewRelic.PerfmonQuery..ctor(String pname, String caname, String coname, String iname) in Z:\eclipse\nri-perfmon\nri-perfmon\Plugin.cs:line 91\r\n at NewRelic.PerfmonPlugin.AddCounter(Counterlist aCounter, Int32 whichCounter) in Z:\eclipse\nri-perfmon\nri-perfmon\Plugin.cs:line 222\r\n at NewRelic.PerfmonPlugin..ctor(Options options, List`1 counters) in Z:\eclipse\nri-perfmon\nri-perfmon\Plugin.cs:line 148\r\n at NewRelic.Program.Main(String[] args) in Z:\eclipse\nri-perfmon\nri-perfmon\Program.cs:line 120\n" working-dir="C:\Program Files\New Relic\newrelic-infra\custom-integrations"

sschwartzman commented 4 years ago

Hi @mvaanand - the counter names are actually different when gotten programmatically from Perfmon. If you run this command at command line, you can see all of the counters for ASPNet: 

get-wmiobject -computername localhost -Namespace root\CIMV2 -Query "Select * from Win32_PerfFormattedData_ASPNET_ASPNETApplications"

If you take a look at the default config file, it has good examples for ASPNet. I extrapolated what's in there to convert what you sent to what I think should be a valid form. Please try this out:

{
  "provider": "ASPNET",
  "category": "ASPNETApplications",
  "counters": [{
      "counter": "ErrorsTotal"
    },
    {
      "counter": "RequestsExecuting"
    },
    {
      "counter": "RequestsFailed"
    },
    {
      "counter": "RequestsInApplicationQueue"
    },
    {
      "counter": "RequestsTimedOut"
    },
    {
      "counter": "RequestsTotal"
    },
    {
      "counter": "RequestExecutionTime"
    },
    {
      "counter": "RequestWaitTime"
    }
  ]
}
mvaanand commented 4 years ago

get-wmiobject -computername localhost -Namespace root\CIMV2 -Query "Select * from Win32_PerfFormattedData_ASPNET_ASPNETApplications"

The above command returns a zero for all counters.

I had tried that couple days back - when I see the events in the Insights dashboard the data is all zero's.

so I tried the below steps from the readme instructions:

If you specify the provider as PerfCounter, the plugin will retrieve the Windows Performance Counter instead of running a WMI query. This can be useful if WMI is returning "all 0's" in a query or the appropriate Performance Counter is easier to find. Example of usage: { "provider": "PerfCounter", "category": "ASP.NET v4.0.30319", "counters": [{ "counter": "Requests Current" }] }

This is not working either.

sschwartzman commented 4 years ago

Did you try the replacement for your counters that I provided you above?

mvaanand commented 4 years ago

I did and all i see are 0's. If i manually run the perfmon counter collection using perfmon I see the data. WMI query returns all 0's.

sschwartzman commented 4 years ago

It may be that perfCounter can only take one counter. When you try just this one, it fails?

{
"provider": "PerfCounter",
"category": "ASP.NET Apps v4.0.30319",
"counters": [{
"counter": "Requests Current"
}]
}

This is slightly different from the example, I added "Apps" here.

mvaanand commented 4 years ago

I get the following error:

"2019-10-17T13:03:40-04:00" level=info msg="Integration health check finished with some errors" instance=nri-perfmon-metrics integration=com.newrelic.perfmon prefix=integration/nri-perfmon working-dir="C:\Program Files\New Relic\newrelic-infra\custom-integrations" time="2019-10-17T13:03:43-04:00" level=error msg="Integration command failed" error="exit status 3762504530" instance=nri-perfmon-metrics integration=com.newrelic.perfmon prefix=integration/nri-perfmon stderr="\nUnhandled Exception: System.InvalidOperationException: Could not locate Performance Counter with specified category name 'ASP.NET Apps v4.0.30319', counter name 'Requests Current'.\r\n at System.Diagnostics.PerformanceCounter.InitializeImpl()\r\n at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, String instanceName, Boolean readOnly)\r\n at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, Boolean readOnly)\r\n at NewRelic.PerfmonQuery..ctor(String pname, String caname, String coname, String iname) in Z:\eclipse\nri-perfmon\nri-perfmon\Plugin.cs:line 86\r\n at NewRelic.PerfmonPlugin.AddCounter(Counterlist aCounter, Int32 whichCounter) in Z:\eclipse\nri-perfmon\nri-perfmon\Plugin.cs:line 222\r\n at NewRelic.PerfmonPlugin..ctor(Options options, List1 counters) in Z:\\eclipse\\nri-perfmon\\nri-perfmon\\Plugin.cs:line 148\r\n at NewRelic.Program.Main(String[] args) in Z:\\eclipse\\nri-perfmon\\nri-perfmon\\Program.cs:line 120\n" working-dir="C:\\Program Files\\New Relic\\newrelic-infra\\custom-integrations" time="2019-10-17T13:04:01-04:00" level=error msg="Integration command failed" error="exit status 3762504530" instance=nri-perfmon-metrics integration=com.newrelic.perfmon prefix=integration/nri-perfmon stderr="\nUnhandled Exception: System.InvalidOperationException: Could not locate Performance Counter with specified category name 'ASP.NET Apps v4.0.30319', counter name 'Requests Current'.\r\n at System.Diagnostics.PerformanceCounter.InitializeImpl()\r\n at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, String instanceName, Boolean readOnly)\r\n at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, Boolean readOnly)\r\n at NewRelic.PerfmonQuery..ctor(String pname, String caname, String coname, String iname) in Z:\\eclipse\\nri-perfmon\\nri-perfmon\\Plugin.cs:line 86\r\n at NewRelic.PerfmonPlugin.AddCounter(Counterlist aCounter, Int32 whichCounter) in Z:\\eclipse\\nri-perfmon\\nri-perfmon\\Plugin.cs:line 222\r\n at NewRelic.PerfmonPlugin..ctor(Options options, List1 counters) in Z:\eclipse\nri-perfmon\nri-perfmon\Plugin.cs:line 148\r\n at NewRelic.Program.Main(String[] args) in Z:\eclipse\nri-perfmon\nri-perfmon\Program.cs:line 120\n" working-dir="C:\Program Files\New Relic\newrelic-infra\custom-integrations" time="2019-10-17T13:04:18-04:00" level=error msg="Integration command failed" error="exit status 3762504530" instance=nri-perfmon-metrics integration=com.newrelic.perfmon prefix=integration/nri-perfmon stderr="\nUnhandled Exception: System.InvalidOperationException: Could not locate Performance Counter with specified category name 'ASP.NET Apps v4.0.30319', counter name 'Requests Current'.\r\n at System.Diagnostics.PerformanceCounter.InitializeImpl()\r\n at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, String instanceName, Boolean readOnly)\r\n at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, Boolean readOnly)\r\n at NewRelic.PerfmonQuery..ctor(String pname, String caname, String coname, String iname) in Z:\eclipse\nri-perfmon\nri-perfmon\Plugin.cs:line 86\r\n at NewRelic.PerfmonPlugin.AddCounter(Counterlist aCounter, Int32 whichCounter) in Z:\eclipse\nri-perfmon\nri-perfmon\Plugin.cs:line 222\r\n at NewRelic.PerfmonPlugin..ctor(Options options, List`1 counters) in Z:\eclipse\nri-perfmon\nri-perfmon\Plugin.cs:line 148\r\n at NewRelic.Program.Main(String[] args) in Z:\eclipse\nri-perfmon\nri-perfmon\Program.cs:line 120\n" working-dir="C:\Program Files\New Relic\newrelic-infra\custom-integrations"

sschwartzman commented 4 years ago

I'm fixing up the PerformanceCounter code some to make it more flexible with counters. Please give me a little time (hopefully today or tomorrow but possibly into next week) and I'll get back to you with a binary to test.

mvaanand commented 4 years ago

Awesome. Thx Seth.

sschwartzman commented 4 years ago

Can you unzip and drop in this exe to see if it works with your original config file (as in, what we started this issue with)? nri-perfmon.exe.zip

Now when it encounters an invalid counter, it will say as much in the log, but not throw an unhandled exception, and other counters that are valid will still run. Also added handling of instances not being listed (it gets all instances when that happens) and use of the '*' for counter gets all counters in that category. Whew!

mvaanand commented 4 years ago

This time there are no unhandled exceptions.

ASPNET, ASPNETApplication still return 0's for all counters.

ASP.NET Apps v4.0.30319 doesn't show up in the events. This uses the PerfCounter Provider.

Attaching config.json and log for your review: newrelic-infra.zip

config.zip

sschwartzman commented 4 years ago

This shouldn't change the ASPNET and ASPNETApplication counters, if they're getting you 0's that's why we're playing with PerfCounter.

As for the PerfCounter provider, can you run "SHOW EVENT TYPES" in NRQL and paste what it comes back with?

Also, can you just run the following at command line and paste the output here?

cd \Program Files\New Relic\newrelic-infra\custom-integrations
nri-perfmon\nri-perfmon.exe

Running at command-line is an easier way to test it than via Infra.

mvaanand commented 4 years ago

{ "results": [ { "eventTypes": [ "ASPNET", "ASPNETApplications", "ASPNETStateService", "AWSPrice", "AdvisorSample", "AjaxRequest", "AlertsRecentEvents", "ApacheSample", "Apigee_Http_CheckSample", "Apigee_Proc_CheckSample", "AppBuildEvent", "AutoScalingGroupSample", "AutoScalingInstanceSample", "AutoScalingLaunchConfigurationSample", "AutoScalingPolicySample", "AutoScalingRegionLimitSample", "BlockDeviceSample", "BrowserInteraction", "CassandraColumnFamilySample", "CassandraSample", "CloudwatchSample", "ComputeSample", "DPCPU", "DPConnectionsAccepted", "DPHTTPConnections", "DPHTTPMeanTransactionTime", "DPHTTPTransactions", "DPLoadBalancerStatus", "DPObjectStatus", "DPReceiveKbpsThroughput", "DPServicePerformance", "DPTransmitKbpsThroughput", "DPWSOperationMetrics", "DatastoreSample", "DnsSample", "DuplicatePodIPs", "HealthCheckSample", "InfrastructureEvent", "IntegrationDataFreshnessReport", "IntegrationError", "IntegrationProviderReport", "InternalK8sCompositeSample", "JVMApigeeSample", "JVMSample", "JavaScriptError", "JenkinsEvent", "K8sClusterSample", "K8sContainerSample", "K8sDeploymentSample", "K8sNamespaceSample", "K8sNodeSample", "K8sPodSample", "K8sReplicasetSample", "K8sVolumeSample", "KubectlGetNodes", "KubectlPluginLogin", "LiveChatAck", "LoadBalancerSample", "MIQ", "MSMQQueue", "MSMQService", "MVBScreenTimes", "Metric", "Mobile", "MobileCrash", "MobileRequest", "MobileRequestError", "MobileSession", "NETCLRExceptions", "NETCLRInterop", "NETCLRJit", "NETCLRLoading", "NETCLRLocksAndThreads", "NETCLRMemory", "NetworkPortSample", "NetworkSample", "NrAuditEvent", "NrIntegrationError", "NtlsEvents", "OracleDatabaseSample", "OracleTablespaceSample", "PageAction", "PageView", "PcfFirehoseEvent", "PostgresqlDatabaseSample", "PostgresqlIndexSample", "PostgresqlInstanceSample", "PostgresqlTableSample", "PrivateNetworkSample", "ProcessSample", "QueueSample", "Relationship", "ServerlessSample", "SesReceiptSample", "SesRegionSample", "Span", "StorageSample", "SyntheticCheck", "SyntheticRequest", "SyntheticsPrivateLocationStatus", "SystemSample", "SystemctlStatus", "Transaction", "TransactionError", "TrustedAdvisorSample", "VvovOperationsSample", "WeaveNetError", "WebService", "ZookeeperSample", "aemjmx", "apigee_nonprod_httpSample", "apigee_nonprod_processSample", "apigee_prod_httpSample", "apigee_prod_processSample", "cmdb", "curlSample", "flexStatusSample", "otdb", "otdb_dev", "procSample", "unixMonitor:Disk", "unixMonitor:DiskIO", "unixMonitor:NetworkIO", "unixMonitor:Process", "unixMonitor:Stats", "unixMonitor:Vmstat" ] } ], "metadata": { "guid": "84801a87-afc3-15af-f328-12ca738ba3ba", "routerGuid": "", "messages": [], "contents": [ { "function": "eventTypes" } ] } }

output.zip

sschwartzman commented 4 years ago

Glad I asked for the output! If you look at it, it had one HUGE event for all of the instances - the PerfCounter code wasn't built to split them up into separate events. So, I fixed it - now each instance will have a separate event of the same eventType, but with an attribute called "name" that contains the instance name.

Please try this out (attached zip of exe): nri-perfmon.exe.zip. I'm pretty confident we'll be good to go after this go-round.

mvaanand commented 4 years ago

After I moved in the latest ni-perfmon.exe all of the counters have stopped reporting in the Insights dashboards. ASPNET & ASPNETApplications custom events show up but I don't see any data. I don't see any errors either. ASP.NET Apps v4.0.30319 still doesn't show up in the insights events.

I'm attaching config.json for your review. Also attached output from manual invocation of nri-perfmon\nri-perfmon.exe.

{ "results": [ { "eventTypes": [ "ASPNET", "ASPNETApplications", "ASPNETStateService", "AWSPrice", "AdvisorSample", "AjaxRequest", "AlertsRecentEvents", "ApacheSample", "Apigee_Http_CheckSample", "Apigee_Proc_CheckSample", "AppBuildEvent", "AutoScalingGroupSample", "AutoScalingInstanceSample", "AutoScalingLaunchConfigurationSample", "AutoScalingPolicySample", "AutoScalingRegionLimitSample", "BlockDeviceSample", "BrowserInteraction", "CassandraColumnFamilySample", "CassandraSample", "CloudwatchSample", "ComputeSample", "DPCPU", "DPConnectionsAccepted", "DPHTTPConnections", "DPHTTPMeanTransactionTime", "DPHTTPTransactions", "DPLoadBalancerStatus", "DPObjectStatus", "DPReceiveKbpsThroughput", "DPTransmitKbpsThroughput", "DPWSOperationMetrics", "DatastoreSample", "DnsSample", "DuplicatePodIPs", "HealthCheckSample", "InfrastructureEvent", "IntegrationDataFreshnessReport", "IntegrationError", "IntegrationProviderReport", "InternalK8sCompositeSample", "JVMApigeeSample", "JVMSample", "JavaScriptError", "JenkinsEvent", "K8sClusterSample", "K8sContainerSample", "K8sDeploymentSample", "K8sNamespaceSample", "K8sNodeSample", "K8sPodSample", "K8sReplicasetSample", "K8sVolumeSample", "KubectlGetNodes", "KubectlPluginLogin", "LiveChatAck", "LoadBalancerSample", "MIQ", "MSMQQueue", "MSMQService", "MVBScreenTimes", "Metric", "Mobile", "MobileCrash", "MobileHandledException", "MobileRequest", "MobileRequestError", "MobileSession", "NETCLRExceptions", "NETCLRInterop", "NETCLRJit", "NETCLRLoading", "NETCLRLocksAndThreads", "NETCLRMemory", "NetworkPortSample", "NetworkSample", "NrAuditEvent", "NrDailyUsage", "NrIntegrationError", "NtlsEvents", "OracleDatabaseSample", "OracleTablespaceSample", "PageAction", "PageView", "PcfFirehoseEvent", "PostgresqlDatabaseSample", "PostgresqlIndexSample", "PostgresqlInstanceSample", "PostgresqlTableSample", "PrivateNetworkSample", "ProcessSample", "QueueSample", "Relationship", "ServerlessSample", "SesReceiptSample", "SesRegionSample", "Span", "StorageSample", "SyntheticAudit", "SyntheticCheck", "SyntheticRequest", "SyntheticsPrivateLocationStatus", "SystemSample", "SystemctlStatus", "Transaction", "TransactionError", "TrustedAdvisorSample", "VvovOperationsSample", "WeaveNetError", "WebService", "ZookeeperSample", "aemjmx", "apigee_nonprod_httpSample", "apigee_nonprod_processSample", "apigee_prod_httpSample", "apigee_prod_processSample", "cmdb", "curlSample", "flexStatusSample", "iaasalert", "otdb", "otdb_dev", "procSample", "unixMonitor:Disk", "unixMonitor:DiskIO", "unixMonitor:NetworkIO", "unixMonitor:Process", "unixMonitor:Stats", "unixMonitor:Vmstat" ] } ], "metadata": { "guid": "3148cc88-56f5-ea75-752b-e536e5a5506f", "routerGuid": "", "messages": [], "contents": [ { "function": "eventTypes" } ] } }

Data.zip config.zip

sschwartzman commented 4 years ago

Ahh, this is a new issue, not specific to PerfCounter. If you look at the data you sent, it has the events properly split up. The problem is now, there are so many separate instances to report, its hitting the maximums of Insights. So I need to make the OHI split up the events into chunks below those limits. I will let you know what I come up with.

sschwartzman commented 4 years ago

You can disregard my last comment :) The fix turned out to be far simpler than that. I had changed "event_type" to "eventType" in the OHI, and that cause all of the events to stop showing up. Once I switched it back, I got my events to re-appear.

Here is a fixed version, please give it a spin. Hopefully this is it!

nri-perfmon.exe.zip

mvaanand commented 4 years ago

ASPNET & ASPNETApplications custom events show up but the data is all 0's. ASP.NET Apps v4.0.30319 still doesn't show up in the insights events.

sschwartzman commented 4 years ago

Can you send me the data again then? And this time, can you send me any infra logs?

Also, we're not dealing with the "all 0's" thing with ASPNET/ASPNETApplications here, that's not something we can fix if the number is coming back as 0 from WMI. So you can remove those counters and don't need to test them further if they are not helping.

mvaanand commented 4 years ago

newrelic-infra.zip

I do see errors

sschwartzman commented 4 years ago

Ok, I figured out why that event type wasn't showing up this time, and my apologies for all of this back-and-forth. Event types cant have '.' in them, and when I was testing, I was using PerfCounters that don't have '.' in the name. So I've updated the code to convert '.' to '_', which is valid for Event type. So THIS time around it should work.

nri-perfmon.zip

Please try this out one more time. Ii it doesn't work, let's set up a working session next week to troubleshoot. Again, I'm sorry for having gone through multiple rounds with this, the PerfCounter code hasn't been used much so it was a learning experience for me as well.

Attached also is a screen shot of that event type working on my Win10 VM: Insights__Explorer

mvaanand commented 4 years ago

IT WORKS!!!!! TY so much Seth. I'm going to try out other perfmon counters that actually provide data.

There is a minor issue though. When I have to update the exe I have to pretty much stop Infra Agent and then Kill nri-perfmon.exe. Is there a way to include stopping nri-perfmon.exe with the infra agent stop? In addition after installing the perfmon plugin, I get a error pop up every time I stop the Infra Agent. The agent stops though but would be nice to not see the error pop up.

sschwartzman commented 4 years ago

I'm SO glad to hear it! I'm going to go ahead and close out this ticket.

I am not having that issue with my Infra agent... is there a chance you're running an older version? I remember that being an issue before (having to kill nri-perfmon.exe) but it's closing out gracefully for me when I stop the Infra service.

sschwartzman commented 4 years ago

@mvaanand I know this issue is resolved but just an update on the "minor issue" you saw. It turns out there was a regression in the Infra agent on Windows, such that when started/stopped as a service, it didn't close out long-running integrations like nri-perfmon. As of v1.7.6 of the Infastructure agent, that is fixed! I tested it on my Windows 10 VM just now, in fact.

https://docs.newrelic.com/docs/release-notes/infrastructure-release-notes/infrastructure-agent-release-notes/new-relic-infrastructure-agent-176

Happy monitoring!

--Seth

mvaanand commented 4 years ago

Thanks for the update. I installed v1.7.6 and I no longer see the issue with the recycle.

I have a specific label in the newrelic-infra.yml

custom_attributes: environment: Frontend

I'm able to lookup by the label name in the infra dashboard.

Specifically as part of the Perfmon data, there is a label in the Insights dashboard that has "Production" as the only value. I'm not sure where that comes from but is there a way to push custom labels from newrelic-infra.yml into insights?

sschwartzman commented 4 years ago

I can't speak on behalf of where that "Label.Envrionment" attribute is being set, but I believe you're looking for this?

https://docs.newrelic.com/docs/infrastructure/install-configure-manage-infrastructure/configuration/infrastructure-configuration-settings#custom-attributes

When I added custom attributes like below to newrelic-infra.yml, they showed up in Infrastructure, as well as in all of the events from the Perfmon OHI:

license_key: (redacted)
verbose: 0
log_file: c:\temp\nria.log
custom_attributes:
  testing: seth-test

Here's what it looks like in Infrastructure:

Infrastructure_by_New_Relic

Here's what it looks like in Insights. This is the "ASPNET" event created by the Perfmon OHI:

Insights__Explorer
mvaanand commented 4 years ago

I'm now able to see the labels. Was a mistake in the newrelic-infra.yml.

Appreciate your help as always Seth.

Thanks,

Vijay Anand Manjamuthu.

*VCM IT - DEVOPS - Retail Application Support.*

RAS ALL (ras.all@verizon.com) – 24x7

On Fri, Dec 6, 2019 at 8:23 AM Seth Schwartzman notifications@github.com wrote:

I can't speak on behalf of where that "Label.Envrionment" attribute is being set, but I believe you're looking for this?

https://docs.newrelic.com/docs/infrastructure/install-configure-manage-infrastructure/configuration/infrastructure-configuration-settings#custom-attributes https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.newrelic.com_docs_infrastructure_install-2Dconfigure-2Dmanage-2Dinfrastructure_configuration_infrastructure-2Dconfiguration-2Dsettings-23custom-2Dattributes&d=DwMCaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=xvgx3zfvWFAs4eWKSAJYcmc9v9Rd8zAx36Ggg-dJa48&m=ZcIg3RbytPpZZ7M3HQnZtz3558Z6igAcVsOgSW3PnmY&s=njrcwNCeg4bXNiP4LAMXUhQ4_n1tB1Lf0y6Dg-o1WjM&e=

When I added custom attributes like below to newrelic-infra.yml, they showed up in Infrastructure, as well as in all of the events from the Perfmon OHI:

license_key: (redacted) verbose: 0 log_file: c:\temp\nria.log custom_attributes: testing: seth-test

Here's what it looks like in Infrastructure:

[image: Infrastructure_by_New_Relic] https://urldefense.proofpoint.com/v2/url?u=https-3A__user-2Dimages.githubusercontent.com_3536684_70329573-2Dba719680-2D1809-2D11ea-2D8ead-2De7a094d18b2d.png&d=DwMCaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=xvgx3zfvWFAs4eWKSAJYcmc9v9Rd8zAx36Ggg-dJa48&m=ZcIg3RbytPpZZ7M3HQnZtz3558Z6igAcVsOgSW3PnmY&s=h6v3kIkTfBuZmyy5VYjvx2V1JNC0-8JeLEscR7uE1pk&e=

Here's what it looks like in Insights. This is the "ASPNET" event created by the Perfmon OHI:

[image: InsightsExplorer] <https://urldefense.proofpoint.com/v2/url?u=https-3Auser-2Dimages.githubusercontent.com_3536684_70329656-2De68d1780-2D1809-2D11ea-2D8aaf-2Dcd1899c709f6.png&d=DwMCaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=xvgx3zfvWFAs4eWKSAJYcmc9v9Rd8zAx36Ggg-dJa48&m=ZcIg3RbytPpZZ7M3HQnZtz3558Z6igAcVsOgSW3PnmY&s=axiZpyMTe89agZvyu_jX8bMTyDNPa42Fv0i0hd9ks2s&e=>

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_newrelic_nri-2Dperfmon_issues_10-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DANP2TN4BCNGZNJ5SSEZV3V3QXJN4RA5CNFSM4JA63C2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGEHE5A-23issuecomment-2D562590324&d=DwMCaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=xvgx3zfvWFAs4eWKSAJYcmc9v9Rd8zAx36Ggg-dJa48&m=ZcIg3RbytPpZZ7M3HQnZtz3558Z6igAcVsOgSW3PnmY&s=MoS8Y8ZzkkC3tvsxz8BEE89GhpSuzqXUTE8RPFnhPMg&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ANP2TN2MCQ5WTBPYGADMIOLQXJN4RANCNFSM4JA63C2A&d=DwMCaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=xvgx3zfvWFAs4eWKSAJYcmc9v9Rd8zAx36Ggg-dJa48&m=ZcIg3RbytPpZZ7M3HQnZtz3558Z6igAcVsOgSW3PnmY&s=L9Y4CebgrgdsnDHYcj4_blFoRJohmlj3QY0ItTanlsM&e= .

mvaanand commented 4 years ago

What does the config.json look like for these perfmon counters:

.NET CLR Memory(w3wp)# Gen 0 Collections .NET CLR Memory(w3wp)# Gen 1 Collections .NET CLR Memory(w3wp)# Gen 2 Collections .NET CLR Memory(w3wp)\Gen 0 heap size .NET CLR Memory(w3wp)\Gen 1 heap size .NET CLR Memory(w3wp)\Gen 2 heap size .NET CLR Memory(w3wp)\Large Object Heap size .NET CLR Memory(w3wp#1)\% Time in GC .NET CLR LocksAndThreads(w3wp#1)\Total # of Contentions

sschwartzman commented 4 years ago

All of those are all in the default config.json, which I just reorganized to be easier to review and update:

Memory: https://github.com/newrelic/nri-perfmon/blob/master/nri-perfmon/config/config.json#L167 Locks and Threads: https://github.com/newrelic/nri-perfmon/blob/master/nri-perfmon/config/config.json#L153

mvaanand commented 4 years ago

Are these Perfmon Counters or WMI ?

sschwartzman commented 4 years ago

WMI - you can tell that because if they were Perfmon Counters, it'd say "PerfCounter" in the Provider for each section. These are tried-and-true counters that work well coming from WMI. If you want to re-write as PerfCounters, that's up to you.

mvaanand commented 4 years ago

I tried and i'm not sure what is wrong. Need your help. Attaching config.json. Please let me know what i'm doing wrong with the config.

config.zip

sschwartzman commented 4 years ago

Can you leave these as WMI counters? They should work fine and will save you time of trying to replicate.

mvaanand commented 4 years ago

Will work with the WMI counters. Thanks Seth.

sschwartzman commented 4 years ago

For what it's worth, I dumped your config file exactly as written into my test Win10 server and it worked perfectly. Attached is the output I got.

perfcounter-test.txt

I'd still recommend using the original (WMI) version of config.json, since it's the default used by all other users of the Perfmon OHI and works very well. If it's not reporting back the counters you expect, then you can play with the PerfCounters.

mvaanand commented 4 years ago

The problem I saw was when I ran the NRQL in NR. It was returning an error. Seemed like it couldnt recognize the _NET_CLR_Memory or _NET_CLR_LocksAndThreads even though these appear under custom events.

Thanks,

Vijay Anand Manjamuthu.

*VCM IT - DEVOPS - Retail Application Support.*

RAS ALL (ras.all@verizon.com) – 24x7

On Tue, Dec 10, 2019 at 1:32 PM Seth Schwartzman notifications@github.com wrote:

For what it's worth, I dumped your config file exactly as written into my test Win10 server and it worked perfectly. Attached is the output I got.

perfcounter-test.txt https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_newrelic_nri-2Dperfmon_files_3946783_perfcounter-2Dtest.txt&d=DwMCaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=xvgx3zfvWFAs4eWKSAJYcmc9v9Rd8zAx36Ggg-dJa48&m=Dncylmmf-yrQSHIZKFKT_wqQgSSqzdRHKzuuLxybA9A&s=oLG0fk_YB48Uutm3Rlr18GpMylssXTEFOOllFmv0dDo&e= I'd still recommend using the original (WMI) version of config.json, since it's the default used by all other users of the Perfmon OHI and works very well. If it's not reporting back the counters you expect, then you can play with the PerfCounters.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_newrelic_nri-2Dperfmon_issues_10-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DANP2TN2XWATHLQHHJSKW4I3QX7VDJA5CNFSM4JA63C2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGQR73I-23issuecomment-2D564207597&d=DwMCaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=xvgx3zfvWFAs4eWKSAJYcmc9v9Rd8zAx36Ggg-dJa48&m=Dncylmmf-yrQSHIZKFKT_wqQgSSqzdRHKzuuLxybA9A&s=ORAicWM4d5CkSHEO3YbMjIGPbpCT_XBJn8GkzfPJVTE&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ANP2TN5BZ65RMRJAIQTK6TLQX7VDJANCNFSM4JA63C2A&d=DwMCaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=xvgx3zfvWFAs4eWKSAJYcmc9v9Rd8zAx36Ggg-dJa48&m=Dncylmmf-yrQSHIZKFKT_wqQgSSqzdRHKzuuLxybA9A&s=AU8-bk65S-xe9XMoLCApHLZo6X9d1EKZDLXbnTy4POA&e= .

sschwartzman commented 4 years ago

Ah, you didn't mention that the first time! So the problem had nothing to do with the Perfmon OHI. Next time, please provide that sort of detail so I know what kind of problem I'm trying to track down.

Here's the solution by the way: Try putting the event type in backticks (`) like this:

SELECT count(*) FROM `_NET_CLR_LocksAndThreads` SINCE 30 MINUTES AGO
mvaanand commented 4 years ago

It worked. Thanks Much Seth. Pleasure working with you.

Thanks,

Vijay Anand Manjamuthu.

*VCM IT - DEVOPS - Retail Application Support.*

RAS ALL (ras.all@verizon.com) – 24x7

On Tue, Dec 10, 2019 at 1:46 PM Seth Schwartzman notifications@github.com wrote:

Try putting them in backticks (`) like this:

SELECT count(*) FROM _NET_CLR_LocksAndThreads SINCE 30 MINUTES AGO

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_newrelic_nri-2Dperfmon_issues_10-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DANP2TN4OLLJ5CZZ7PEFLXSLQX7W2HA5CNFSM4JA63C2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGQTSTY-23issuecomment-2D564214095&d=DwMCaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=xvgx3zfvWFAs4eWKSAJYcmc9v9Rd8zAx36Ggg-dJa48&m=tuQpf78UkMwnRFBgtZk3s5HYApvG_wFl113vbl7-AIg&s=Wqujb4bkdiMQ_BKuW0r0Yddx_P5_mqneuDJyAR8-0DQ&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ANP2TN3O3FMRR6L6CIAAHUTQX7W2HANCNFSM4JA63C2A&d=DwMCaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=xvgx3zfvWFAs4eWKSAJYcmc9v9Rd8zAx36Ggg-dJa48&m=tuQpf78UkMwnRFBgtZk3s5HYApvG_wFl113vbl7-AIg&s=RwP6PEKp1uCno_MuEwarRXtOoVZ7pJ_PyhgvASSyg4E&e= .