Can I integrate alerts to all search heads in a search head cluster?

[ThreatHunterDiary]

Hi,

I have a search head cluster with 3 members. I want to integrate alert manager app in the search head cluster in such a way that on all the search heads I should be able to get all the alerts OR all the alerts should come on any one search head.

Because right now what is happening is that alert manager app has been installed on all the search heads through deployer and "alerts" index is also created on all the search heads.

And whenever the scheduled searches run different alerts are coming on different different search heads. Typically the search head which initiates the search gets the alert triggered in alert manager.

How do I integrate Alert Manager so that the alert gets triggered in either all the search head's or any one??

[simcen]

Hi

On a search head cluster, it’s best practice to forward all events to the indexers. In that case, every search head will be able to find the events in the alert index and that will solve you rproblem.

Thank you

On 13 Oct 2017, 12:29 +0100, Jeet Ashutosh Pandya notifications@github.com, wrote:

Hi, I have a search head cluster with 3 members. I want to integrate alert manager app in the search head cluster in such a way that on all the search heads I should be able to get all the alerts OR all the alerts should come on any one search head. Because right now what is happening is that alert manager app has been installed on all the search heads through deployer and "alerts" index is also created on all the search heads. And whenever the scheduled searches run different alerts are coming on different different search heads. Typically the search head which initiates the search gets the alert triggered in alert manager. How do I integrate Alert Manager so that the alert gets triggered in either all the search head's or any one?? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[ThreatHunterDiary]

Hi @simcen ,

I followed your advice and now I have what I wanted. I am getting same alerts across all the Search Heads. But now the issue is I am not able to change the state of the alert or I am not able to add comments to specific alerts.

The save button after comments is not leading anywhere.

[simcen]

Ok, I’ll check over the weekend. Maybe there’s a bug related to the use of the app on SHC even if it was verified. I’ll let you know.

On 13 Oct 2017, 14:22 +0100, Jeet Ashutosh Pandya notifications@github.com, wrote:

Hi, I followed your advice and now I have what I wanted. I am getting same alerts across all the Search Heads. But now the issue is I am not able to change the state of the alert or I am not able to add comments to specific alerts. The save button after comments is not leading anywhere. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[ThreatHunterDiary]

Hi @simcen,

Here are some errors I am getting in splunkd.log file which migh be helpfull in figuring out the root cause.

10-13-2017 18:50:05.195 +0530 INFO  sendmodalert - Invoking modular alert action=alert_manager for search="Search Name X" sid="scheduler__admin__search__RMD55db1f8da8d5ff393_at_1507900500_25_82469BEB-462F-41A6-9883-DBAE6CD97079" in app="search" owner="admin" type="saved"
10-13-2017 18:50:05.625 +0530 ERROR sendmodalert - action=alert_manager STDERR -  Traceback (most recent call last):
10-13-2017 18:50:05.625 +0530 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 427, in <module>
10-13-2017 18:50:05.626 +0530 ERROR sendmodalert - action=alert_manager STDERR -      createIncidentChangeEvent(event, metadata['job_id'], settings.get('index'))
10-13-2017 18:50:05.626 +0530 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 157, in createIncidentChangeEvent
10-13-2017 18:50:05.626 +0530 ERROR sendmodalert - action=alert_manager STDERR -      input.submit(event, hostname = socket.gethostname(), sourcetype = 'incident_change', source = 'alert_handler.py', index=index)
10-13-2017 18:50:05.626 +0530 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/lib/python2.7/site-packages/splunk/input.py", line 180, in submit
10-13-2017 18:50:05.626 +0530 ERROR sendmodalert - action=alert_manager STDERR -      raise splunk.RESTException, (serverResponse.status, msg_text)
10-13-2017 18:50:05.626 +0530 ERROR sendmodalert - action=alert_manager STDERR -  splunk.RESTException: [HTTP 400] ["message type=WARN code=None text=supplied index 'alerts' missing;"]
10-13-2017 18:50:05.642 +0530 INFO  sendmodalert - action=alert_manager - Alert action script completed in duration=445 ms with exit code=1
10-13-2017 18:50:05.642 +0530 WARN  sendmodalert - action=alert_manager - Alert action script returned error code=1
10-13-2017 18:50:05.642 +0530 ERROR sendmodalert - Error in 'sendalert' command: Alert script returned error code 1.
10-13-2017 18:50:05.642 +0530 ERROR SearchScheduler - Error in 'sendalert' command: Alert script returned error code 1., search='sendalert alert_manager results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__admin__search__RMD55db1f8da8d5ff393_at_1507900500_25_82469BEB-462F-41A6-9883-DBAE6CD97079/per_result_alert/tmp_0.csv.gz" results_link="https://SearchHead1:10443/app/search/search?q=%7Cloadjob%20scheduler__admin__search__RMD55db1f8da8d5ff393_at_1507900500_25_82469BEB-462F-41A6-9883-DBAE6CD97079%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now"'

Let me know if you find anything.

Thanks.

[simcen]

I know this issue. Try to create the index on the search heads. I found out that the definition must be there even if the events are getting forwarded. Will try to fix but the workaround should work

On 13 Oct 2017, 14:49 +0100, Jeet Ashutosh Pandya notifications@github.com, wrote:

Hi @simcen, Here are some errors I am getting in splunkd.log file which migh be helpfull in figuring out the root cause. 10-13-2017 18:50:05.195 +0530 INFO sendmodalert - Invoking modular alert action=alert_manager for search="Search Name X" sid="scheduleradminsearchRMD55db1f8da8d5ff393_at_1507900500_25_82469BEB-462F-41A6-9883-DBAE6CD97079" in app="search" owner="admin" type="saved" 10-13-2017 18:50:05.625 +0530 ERROR sendmodalert - action=alert_manager STDERR - Traceback (most recent call last): 10-13-2017 18:50:05.625 +0530 ERROR sendmodalert - action=alert_manager STDERR - File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 427, in 10-13-2017 18:50:05.626 +0530 ERROR sendmodalert - action=alert_manager STDERR - createIncidentChangeEvent(event, metadata['job_id'], settings.get('index')) 10-13-2017 18:50:05.626 +0530 ERROR sendmodalert - action=alert_manager STDERR - File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 157, in createIncidentChangeEvent 10-13-2017 18:50:05.626 +0530 ERROR sendmodalert - action=alert_manager STDERR - input.submit(event, hostname = socket.gethostname(), sourcetype = 'incident_change', source = 'alert_handler.py', index=index) 10-13-2017 18:50:05.626 +0530 ERROR sendmodalert - action=alert_manager STDERR - File "/opt/splunk/lib/python2.7/site-packages/splunk/input.py", line 180, in submit 10-13-2017 18:50:05.626 +0530 ERROR sendmodalert - action=alert_manager STDERR - raise splunk.RESTException, (serverResponse.status, msg_text) 10-13-2017 18:50:05.626 +0530 ERROR sendmodalert - action=alert_manager STDERR - splunk.RESTException: [HTTP 400] ["message type=WARN code=None text=supplied index 'alerts' missing;"] 10-13-2017 18:50:05.642 +0530 INFO sendmodalert - action=alert_manager - Alert action script completed in duration=445 ms with exit code=1 10-13-2017 18:50:05.642 +0530 WARN sendmodalert - action=alert_manager - Alert action script returned error code=1 10-13-2017 18:50:05.642 +0530 ERROR sendmodalert - Error in 'sendalert' command: Alert script returned error code 1. 10-13-2017 18:50:05.642 +0530 ERROR SearchScheduler - Error in 'sendalert' command: Alert script returned error code 1., search='sendalert alert_manager results_file="/opt/splunk/var/run/splunk/dispatch/scheduleradminsearchRMD55db1f8da8d5ff393_at_1507900500_25_82469BEB-462F-41A6-9883-DBAE6CD97079/per_result_alert/tmp_0.csv.gz" results_link="https://SearchHead1:10443/app/search/search?q=%7Cloadjob%20scheduler__admin__search__RMD55db1f8da8d5ff393_at_1507900500_25_82469BEB-462F-41A6-9883-DBAE6CD97079%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now"' Let me know if you find anything. Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[simcen]

This will be resolved in v2.2.0. Just commited a change today. You can update the app already from git, just make sure to rename the app folder to "alert_manager"