search

microsoft / AzureMonitorAddonForSplunk

A Splunk add-on (aka modular input) that brings Metrics and Diagnostic Logs from various Azure ARM resources and the subscription-wide Activity Log (aka Audit Log) to Splunk Enterprise.
Other
63 stars 45 forks source link

Multiple Errors #110

Closed GISplunk closed 3 years ago

GISplunk commented 5 years ago

Greetings,

We continuously have the following errors occur regarding our TA-Azure_Monitor app, any assistance on how to resolve these would be greatly appreciated.

message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" ^ message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Connection.sendFrame (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:329:10) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Object.frames.writeFrame (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64:9) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at ReceiverLink.Link.attach (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:152:27) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Timeout._onTimeout (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:270:12) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Timer.listOnTimeout (timers.js:214:5) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at ontimeout (timers.js:386:11) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at tryOnTimeout (timers.js:250:5) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" stream.write(buffer, callback); message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" /opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64 message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" TypeError: Cannot read property 'write' of null

sebastus commented 5 years ago

How frequently does this occur? Assuming your TA runs once per minute, is it happening each minute?

On Wed, Jan 9, 2019 at 3:30 PM GISplunk notifications@github.com wrote:

Greetings,

We continuously have the following errors occur regarding our TA-Azure_Monitor app, any assistance on how to resolve these would be greatly appreciated.

message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" ^ message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Connection.sendFrame (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:329:10) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Object.frames.writeFrame (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64:9) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at ReceiverLink.Link.attach (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:152:27) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Timeout._onTimeout (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:270:12) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Timer.listOnTimeout (timers.js:214:5) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at ontimeout (timers.js:386:11) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at tryOnTimeout (timers.js:250:5) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" stream.write(buffer, callback); message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" /opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64 message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" TypeError: Cannot read property 'write' of null

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Microsoft/AzureMonitorAddonForSplunk/issues/110, or mute the thread https://github.com/notifications/unsubscribe-auth/ABOrrKlC5qeudTzYFiAUmTs7dABaDedNks5vBgsEgaJpZM4Z3t1z .

GISplunk commented 5 years ago

Correct, once per minute.

sebastus commented 5 years ago

Sorry - I'm unclear what you mean. I need to know - is the error occurring each minute?

On Wed, Jan 9, 2019 at 5:22 PM GISplunk notifications@github.com wrote:

Correct, once per minute.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Microsoft/AzureMonitorAddonForSplunk/issues/110#issuecomment-452772417, or mute the thread https://github.com/notifications/unsubscribe-auth/ABOrrEVoaYQNqKjTtiwoPKPCjwgDb6L2ks5vBiU5gaJpZM4Z3t1z .

GISplunk commented 5 years ago

Apologies, yes, the error is occurring every minute.

sebastus commented 5 years ago

That usually means that no records are making it into Splunk at all. Is this what you're observing?

In past, I have seen this as more intermittent. I have yet to come across one where the error is so persistent. I have no solution at this point. The error comes from an underlying module that I don't have any control over.

Have you taken a look at the Azure Function solution for getting Azure Monitor messages into Splunk? There's a link to it in the readme of the TA. It reads the same event hubs and sends messages to the Splunk HEC port. It does not use the same problem module.

On Thu, Jan 10, 2019 at 1:47 PM GISplunk notifications@github.com wrote:

Apologies, yes, the error is occurring every minute.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Microsoft/AzureMonitorAddonForSplunk/issues/110#issuecomment-453101285, or mute the thread https://github.com/notifications/unsubscribe-auth/ABOrrBTK_M1eQlUITW73GxjG_-ZuVM7eks5vB0SGgaJpZM4Z3t1z .

sebastus commented 5 years ago

I discovered that if the TA tries to access a hub that does not exist, the stream.write error occurs 100% of the time. This is a new condition - it only did it intermittently in times passed. Delete all hubs from hubs.json that do not exist in the hub namespace. This should fix the problem.

mdmosarafmd commented 4 years ago

This still not working for me. i kept only specific hubs in my hubs.json file. i configured 5 different inputs for diagnostic logs. I'm able to see logs from on event hub namespace but not for other 4 event hub namespace. Getting same erro.

error

message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" ^ message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Connection.sendFrame (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:329:10) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Object.frames.writeFrame (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64:9) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at ReceiverLink.Link.attach (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:152:27) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Timeout._onTimeout (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:270:12) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Timer.listOnTimeout (timers.js:214:5) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at ontimeout (timers.js:386:11) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at tryOnTimeout (timers.js:250:5) message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" stream.write(buffer, callback); message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" /opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64 message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" TypeError: Cannot read property 'write' of null