microsoftarchive / data-pipeline

Exploring the problem of high-scale data ingestion on Azure
MIT License
6 stars 4 forks source link

Performance counter errors in Car.Dispatcher.Worker role deployment #30

Closed rajinders closed 9 years ago

rajinders commented 9 years ago

I deployed Cars.Dispatcher.Worker role successfully. I was viewing the diagnostics data in visual studio and I am seeing 18 errors. All of them are related to performance counters: PdhExpandWildCardPath(.NET CLR NETWORKING 4.0.0.0\bytes received} failed

There are 5 errors for .NET CLR Networking 4 errors for EventHub 9 errors related to ServiceBus client.

Let me know if there is a way to resolve these errors or if they can be ignored.

bennage commented 9 years ago

Hi @rajinders,

We'll schedule some time to look into this. Aside from the errors, are events being successfully routed?

rajinders commented 9 years ago

I ran Simulator with default settings. I verified that messages were sent to the event hub successfully. I look at the performance counters with Azure management studio and I cannot find any data for counters like "EventHub dispatcher(WaWorkerHost)\Total Messages sent etc.

Even the Cars.Simulator.Worker role has similar errors. No perfmon data for that counters with errors. When I RDP in to the simulator worker role instance I saw these errors in D:\logs\errorlog file time,machinename,role,instance,level,logger,message,exception 2015-02-02 21:46:40.9536,RD0003FF461742,Cars.Simulator.WorkerRole,WorkerRole_IN_0,Warn,Simulator,Initializing performance counters for WaWorkerHost failed. Instrumentation disabled,"EXCEPTION OCCURRED:System.InvalidOperationException The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly. Void InitializeImpl() at System.Diagnostics.PerformanceCounter.InitializeImpl() at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, String instanceName, Boolean readOnly) at Microsoft.Practices.DataPipeline.Instrumentation.PerformanceCounterDefinition.CreatePerformanceCounter(String instanceName) in c:\Git\data-pipeline-master\src\Implementation\Shared\Core\Instrumentation\PerformanceCounterDefinition.cs:line 22 at Microsoft.Practices.DataPipeline.Cars.Dispatcher.Simulator.Instrumentation.SenderInstrumentationPublisher..ctor(String instanceName, SenderInstrumentationManager instrumentationManager) in c:\Git\data-pipeline-master\src\Implementation\Simulator\Cars.Simulator\Instrumentation\SenderInstrumentationPublisher.cs:line 22 at Microsoft.Practices.DataPipeline.Cars.Dispatcher.Simulator.Instrumentation.SenderInstrumentationManager.CreatePublisher(String instanceName) in c:\Git\data-pipeline-master\src\Implementation\Simulator\Cars.Simulator\Instrumentation\SenderInstrumentationManager.cs:line 71 System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly. at System.Diagnostics.PerformanceCounter.InitializeImpl() at System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, String instanceName, Boolean readOnly) at Microsoft.Practices.DataPipeline.Instrumentation.PerformanceCounterDefinition.CreatePerformanceCounter(String instanceName) in c:\Git\data-pipeline-master\src\Implementation\Shared\Core\Instrumentation\PerformanceCounterDefinition.cs:line 22 at Microsoft.Practices.DataPipeline.Cars.Dispatcher.Simulator.Instrumentation.SenderInstrumentationPublisher..ctor(String instanceName, SenderInstrumentationManager instrumentationManager) in c:\Git\data-pipeline-master\src\Implementation\Simulator\Cars.Simulator\Instrumentation\SenderInstrumentationPublisher.cs:line 22 at Microsoft.Practices.DataPipeline.Cars.Dispatcher.Simulator.Instrumentation.SenderInstrumentationManager.CreatePublisher(String instanceName) in c:\Git\data-pipeline-master\src\Implementation\Simulator\Cars.Simulator\Instrumentation\SenderInstrumentationManager.cs:line 71"

bennage commented 9 years ago

Thanks. We'll investigate.

lbrader commented 9 years ago

Ok, I have the latest and doing an analysis.

bennage commented 9 years ago

Thanks @lbrader. I was able to reproduce the same errors mentioned in the initial comment.

lbrader commented 9 years ago

I can reproduce it locally on my box. The issue occurs on the local box due to a privilege issue.

If you run the Consoles for each of the 3 roles - the counters are created with no issue. Then you can run the worker roles locally without issues. If they don't exist locally - then the bug occurs.

Why the worker role doesn't have the local privilege is the question now. And are we seeing this on an Azure deployment.

First, I'll validate the Azure deployment. Since we didn't see this issue originally on an Azure deployment. If it is occurring, bigger issue. If not, then we have local configuration issue.

Should have an answer later today.

lbrader commented 9 years ago

Erg. The issue is occurring on all Azure based deployments.

Issue is that the scripts installing the counter are no longer running as admin. Why - good question. Will work to close this item.

lbrader commented 9 years ago

Figured out the issue. A refactoring moved the installer piece to a different DLL. Will work with Christopher to fix this item.

bennage commented 9 years ago

Fixed in #34