search

microsoft / AzureMonitorAddonForSplunk

A Splunk add-on (aka modular input) that brings Metrics and Diagnostic Logs from various Azure ARM resources and the subscription-wide Activity Log (aka Audit Log) to Splunk Enterprise.
Other
62 stars 45 forks source link

TypeError: Cannot read property 'write' of null #125

Closed Paul1896 closed 5 years ago

Paul1896 commented 5 years ago

Hello,

I hope you can help me with my issue. I already read the threads with the same issue but I didn't solve the problem yet.

02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh"     at Timer.listOnTimeout (timers.js:214:5)
--
  | 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh"     at tryOnTimeout (timers.js:250:5)
  | 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh"     at ontimeout (timers.js:386:11)
  | 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh"     at Timeout._onTimeout (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:270:12)
  | 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh"     at ReceiverLink.Link.attach (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:152:27)
  | 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh"     at Connection.sendFrame (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:329:10)
  | 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh"     at Object.frames.writeFrame (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64:9)
  | 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" TypeError: Cannot read property 'write' of null
  | 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh"         ^
  | 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh"   stream.write(buffer, callback);
  | 02-25-2019 15:55:41.317 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" /opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64

CLI:

/opt/splunk/etc/apps/TA-Azure_Monitor/bin>./azure_activity_log.sh
ERROR Modular input Error: Receiving input definitions prior to streaming timed out.

Port 5671 and Port 5672 are also open for outbound connections.

hubs.json contains only one hub which is created with the same name on azure side.

{
 "heartbeat": "resourceId"
}

Thank you!

sebastus commented 5 years ago

how many partitions does the heartbeat hub have? It has to be 4.

On Mon, Feb 25, 2019 at 4:01 PM Paul1896 notifications@github.com wrote:

Hello,

I hope you can help me with my issue. I already read the threads with the same issue but I didn't solve the problem yet.

02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Timer.listOnTimeout (timers.js:214:5)

--

| 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at tryOnTimeout (timers.js:250:5)

| 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ontimeout (timers.js:386:11)

| 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Timeout._onTimeout (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:270:12)

| 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ReceiverLink.Link.attach (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:152:27)

| 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Connection.sendFrame (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:329:10)

| 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Object.frames.writeFrame (/opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64:9)

| 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" TypeError: Cannot read property 'write' of null

| 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" ^

| 02-25-2019 15:55:41.318 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" stream.write(buffer, callback);

| 02-25-2019 15:55:41.317 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" /opt/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64

CLI:

/opt/splunk/etc/apps/TA-Azure_Monitor/bin>./azure_activity_log.sh

ERROR Modular input Error: Receiving input definitions prior to streaming timed out.

Port 5671 and Port 5672 are also open for outbound connections.

hubs.json contains only ony hub which is created with the same name on azure side.

{

"heartbeat": "resourceId"

}

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Microsoft/AzureMonitorAddonForSplunk/issues/125, or mute the thread https://github.com/notifications/unsubscribe-auth/ABOrrIdqRk9zXo55UDCrsK2kaDP-qM9lks5vRAjhgaJpZM4bQNBf .

Paul1896 commented 5 years ago

@sebastus Partition count is set to 4.

sebastus commented 5 years ago

and the heartbeat hub actually exists in the hub namespace?

On Tue, Feb 26, 2019 at 8:29 AM Paul1896 notifications@github.com wrote:

@sebastus https://github.com/sebastus Partition count is set to 4.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Microsoft/AzureMonitorAddonForSplunk/issues/125#issuecomment-467344621, or mute the thread https://github.com/notifications/unsubscribe-auth/ABOrrBLjeg1pJnODWCcUXEKSfMkzPyPXks5vRPB9gaJpZM4bQNBf .

Paul1896 commented 5 years ago

yes only the heartbeat hub exists in the hub namespace and has a partition count of 4

sebastus commented 5 years ago

I'll need to take a closer look at this. Please email me: golive@microsoft.com.

On Tue, Feb 26, 2019 at 9:40 AM Paul1896 notifications@github.com wrote:

yes only the heartbeat hub exists in the hub namespace and has a partition count of 4

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Microsoft/AzureMonitorAddonForSplunk/issues/125#issuecomment-467368523, or mute the thread https://github.com/notifications/unsubscribe-auth/ABOrrCLHeUYkg-uQs5NxSGzLInhUMXtUks5vRQElgaJpZM4bQNBf .

sebastus commented 5 years ago

The solution to this is as follows:

The hub name "insights-operational-logs" was listed in hubs.json. There was no log profile to export Activity Log to event hub. The diagnostic logs data input was configured.

There are 3 things wrong with this:

  1. "insights-operational-logs" hub is for Activity Logs only. It is created by the platform when the first log is exported. It should never be in hubs.json.
  2. because there was no log profile to export Activity logs to event hub, this hub (in #1) did not exist.
  3. Because the hub did not exist and the diagnostic logs data input was configured, the data input was trying to read from a nonexistent hub. This is the basis of the error message.

If the log profile existed and therefore the hub existed, the data input would not have complained. But the messages would not be categorized correctly and some errors would be thrown due to differences in the structure of the messages. If the Activity Log data input were also configured, both data inputs would be reading from the same hub - creating a race condition. No errors would occur, but some messages would be indexed strangely due to being processed by the diagnostic logs TA.