search

microsoft / AzureFunctionforSplunkVS

An Azure Function to make Azure Monitor telemetry available to a Splunk monitoring system.
MIT License
50 stars 75 forks source link

Bunch of failures when functions are processing messages from event hub to splunk #28

Closed ramkvlss closed 3 years ago

ramkvlss commented 4 years ago

Hello Greg,

We have deployed this code to send all the events from Eventhub to Splunk. It is working great and sending most of the data to splunk, but we see bunch of error count on the function runs to process different category of events/messages. Here is the sample log and the current success/error count to get an idea of what % is failing. Wondering if anything changed that is causing these errors and if we need to make any changes to the function code to make this 100% success ?

Also, is it going to auto try to send messages from fault processor queue or no and if yes, is there going to be certain no of retires and give up ? Can you please check on these sample failure messages for each category and give some direction on how to fix ?

EhActivityLogsExt : Success-57715 ErrorCount-30601

Error emitting messages to output binding: The specified key 'operationName' does not exist in the ExpandoObject.. The messages were held in the fault processor queue for handling once the error is resolved. The specified key 'operationName' does not exist in the ExpandoObject. Executed 'EhActivityLogsExt' (Failed, Id=xxxxxxxxxxxxxxxx, Duration=84ms

EhDiagnosticLogsExt : Success-18697 ErrorCount-6542

Error emitting messages to output binding: Unable to extract resourceid or resourceId from the message.. The messages were held in the fault processor queue for handling once the error is resolved. Unable to extract resourceid or resourceId from the message. Executed 'EhDiagnosticLogsExt' (Failed, Id=xxxxxxxxxxxxxxxx, Duration=92ms)

EhMetricsExt : Success-2131 ErrorCount-1157

Error emitting messages to output binding: Unable to extract resourceid or resourceId from the message.. The messages were held in the fault processor queue for handling once the error is resolved. Unable to extract resourceid or resourceId from the message. Executed 'EhMetricsExt' (Failed, Id=xxxxxxxxxxxxxxxxxxxxxxx, Duration=127ms)

FaultProcessor : Success-7 ErrorCount-186658

Executing 'FaultProcessor' (Reason='New queue message detected on 'transmission-faults'.', Id=xxxxxxxxxxxxxxx) Trigger Details: MessageId: xxxxxxxxxxxxxxxxxxxxxxxx, DequeueCount: 5, InsertionTime: 2020-10-21T14:48:17.000+00:00 Json: null Fault Data: xxxxxxxxxxxxxxxxxxxxxxxxxxxx Blob Reader: transmission-faults Blob Attribute: Microsoft.WindowsAzure.Storage.Blob.BlobProperties Object reference not set to an instance of an object. FaultProcessor failed to transmit: xxxxxxxxxxxxxxxxxxxxxxxxxxxx FaultProcessor failed to transmit Executed 'FaultProcessor' (Failed, Id=xxxxxxxxxxxxxxxxxxxxxxxxx, Duration=86ms) FaultProcessor failed to transmit

ramkvlss commented 4 years ago

@sebastus @msftgits @microsoftopensource @brettroquemore

Can you please look into this when you get time ?