search

alertmanager / alert_manager

Splunk Alert Manager with advanced reporting on alerts, workflows (modify assignee, status, severity) and auto-resolve features
Other
81 stars 44 forks source link

some alerts were not displayed in alert manager Incident Posture interface #101

Open nick3619 opened 8 years ago

nick3619 commented 8 years ago

I installed Alert manager with version 1.1,and now i found some alerts were not displayed. Actually we have received some alerts which sent by alert manager,but we can see only few alert displayed in alert manager Incident Posture interface,it was abnormal ,could you have any advice to resolve this problem,thanks!

my2ndhead commented 8 years ago

Can you compare the splunk scheduler.log with alert_manager.log (under $SPLUNK_HOME/var/log/splunk) to see if alerts that have fired (scheduler.log) are really opened in Alert Manager (alert_manager.log).

Thanks Mika>

nick3619 commented 8 years ago

Hi Mika,

We just used transaction to group the two logs by “job_id / sid”.

Each transaction has 8 events, as below:

  1.  DEBUG Parsed arguments
  2.  INFO Creating incident
  3.  DEBUG Create event will be
  4.  DEBUG Auto assign (owner change)
  5.  DEBUG Auto assign (status change)
  6.  INFO Incident initial stage added to collection
  7.  INFO Alert results …. written to collection incident_results
  8.  INFO SavedSplunker

The alert is triggered as scheduled. But not all the alerts are displayed in alert manager, even though it is recorded in alert_manager.log as”written to collection incident_results”.

I also ran “|inputlookup incidents” and I found all the matching incidents from above (alert manager). So looks like they are just not displayed in the GUI.

Thanks in advance for your help.

Rgds,

Nick

发件人: my2ndhead [mailto:notifications@github.com] 发送时间: 2016年1月4日 16:39 收件人: simcen/alert_manager 抄送: nick3619 主题: Re: [alert_manager] some alerts were not displayed in alert manager Incident Posture interface (#101)

Can you compare the splunk scheduler.log with alert_manager.log (under $SPLUNK_HOME/var/log/splunk) to see if alerts that have fired (scheduler.log) are really opened in Alert Manager (alert_manager.log).

Thanks Mika>

— Reply to this email directly or view it on GitHub https://github.com/simcen/alert_manager/issues/101#issuecomment-168611721 .说明: 说明: https://github.com/notifications/beacon/APXV-SuTePjgSC6n-lyX1Al9LO1IQ3ZMks5pWicKgaJpZM4Gs1-H.gif

simcen commented 8 years ago

Hi Did you install the TA on both Search Head and Indexers? Also, can you check with "|inputlookup incidents" if there are any empty columns?

Thanks Simon

simcen commented 8 years ago

Do you have any feedback on my latest comment?

rfronteau commented 8 years ago

I have the same problem and when I check "inputlookup incidents", I have result.

But I don't Install TA on Search Head ? I must do this ?

rfronteau commented 8 years ago

I found the problem. TA must install in Search Head and Indexers but your Deployment Matrix say TA must install on Indexers only.

simcen commented 8 years ago

Can you tell me which deployment matrix you're refering to? Both http://docs.alertmanager.info/Documentation/AlertManager/latest/AlertManager/Beforeyoudeploy and the README.md mention to install the TA on the Search Heads and on Indexers. Do the incidents now show up in the dashboard?

rfronteau commented 8 years ago

No in this page. There is no "x" in TA search head box

Le jeu. 12 mai 2016 12:04, Simon notifications@github.com a écrit :

Can you tell me which deployment matrix you're refering to? Both http://docs.alertmanager.info/Documentation/AlertManager/latest/AlertManager/Beforeyoudeploy and the README.md mention to install the TA on the Search Heads and on Indexers. Do the incidents now show up in the dashboard?

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/simcen/alert_manager/issues/101#issuecomment-218713937

rfronteau commented 8 years ago

Yes now incidents appear on incidents posture view

Le jeu. 12 mai 2016 12:34, Romuald Fronteau rfronteau@cfsl-asso.org a écrit :

No in this page. There is no "x" in TA search head box

Le jeu. 12 mai 2016 12:04, Simon notifications@github.com a écrit :

Can you tell me which deployment matrix you're refering to? Both http://docs.alertmanager.info/Documentation/AlertManager/latest/AlertManager/Beforeyoudeploy and the README.md mention to install the TA on the Search Heads and on Indexers. Do the incidents now show up in the dashboard?

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/simcen/alert_manager/issues/101#issuecomment-218713937

simcen commented 8 years ago

You're right, thanks for the hint!

thunderking66 commented 8 years ago

Good Afternoon

We are currently using the Alert Manager and cannot see events in the Incident Posture section of the App, We have our Splunk set up where we use a deployment server to push updates throughout. The Alert manager and TA is installed on the search head. And the TA is installed on the indexers as well.