Slack Alarm type event not notifying on recovery

[mdiorio]

First - thank you for this script. It is absolutely what I was looking for.

Based on your description: Recovery message from Zabbix will update and then delete already posted message in Slack (--slack_mode=alarm) Recovery message from Zabbix will be posted as new message (--slack_mode=event)

It appears that we'd like to use the alarm mode for slack so we only see active issues in the channel. Event mode works perfectly. I can send problems and resolutions and they both show up in the channel.

If I set --slack_mode=alarm I can get the problem event, but the OK event never is received in Slack.

From the command line it works the same. I can get the Problem event, but the OK event doesn't arrive the first time it's sent. If I send it again, then I get it in the channel, but it doesn't update or delete the already posted message.

Any ideas? Thanks!

-bash-4.2$ ./zbx-notify --slack --api_token=token --slack_mode=alarm '#zabbix-alerts' 'PROBLEM:Test Test failure' 'Host: testHostname Trigger: PROBLEM: testHostname Temp test failure Timestamp: 20170217 10:27:00 EST eventid: 100555'

In slack: [10:28] PROBLEM:Test Test failure Host: testHostname Trigger: PROBLEM: testHostname Temp test failure Timestamp: 20170217 10:27:00 EST eventid: 100555

-bash-4.2$ ./zbx-notify --slack --api_token=token --slack_mode=alarm '#zabbix-alerts' 'OK:Test Test failure' 'Host: testHostname Trigger: OK: testHostname Temp test failure Timestamp: 20170217 10:29:00 EST eventid: 100555'

In slack: Nothing arrives

-bash-4.2$ ./zbx-notify --slack --api_token=token --slack_mode=alarm '#zabbix-alerts' 'OK:Test Test failure' 'Host: testHostname Trigger: OK: testHostname Temp test failure Timestamp: 20170217 10:29:00 EST eventid: 100555'

In slack: [10:31] OK:Test Test failure Host: testHostname Trigger: OK: testHostname Temp test failure Timestamp: 20170217 10:29:00 EST eventid: 100555

[mdiorio]

A little additional detail from looking into the code a tiny bit. In the storage file, I see the problem pushed to the top of the file.

On the first OK event sent, I see that the problem is popped from the top, but no event is sent to Slack.

On the 2nd OK event, there is nothing matching in the temp storage file to pop, but the message is successfully sent.

[v-zhuravlev]

Hi, i'll check it. In the meantime, try to add --nofork --debug. I think i'm able to reproduce that

[mdiorio]

So here's the same set of commands: Send alert: -bash-4.2$ ./zbx-notify --slack --api_token=xoxb-142571249477-pMKPwiG7qo55DaN11f8pP34M --slack_mode=alarm --debug --nofork '#zabbix-alerts' 'PROBLEM:Test Test failure' 'Host: testHostname Trigger: PROBLEM: testHostname Temp test failure Timestamp: 20170217 11:26:00 EST eventid: 100556' $VAR1 = bless( { 'web_api_url' => 'https://slack.com/api/', 'last_err' => '', 'api_token' => 'xoxb-142571249477-pMKPwiG7qo55DaN11f8pP34M', 'debug' => 1, 'channel' => '#zabbix-alerts', 'mock_url' => undef }, 'SlackBot' ); $VAR1 = { 'hipchat' => { 'message_format' => 'text', 'from' => undef, 'notify' => 'true' }, 'status' => 'PROBLEM', 'subject' => 'PROBLEM:Test Test failure', 'pagerduty' => { 'client' => undef, 'client_url' => undef }, 'message' => 'Host: testHostname Trigger: PROBLEM: testHostname Temp test failure Timestamp: 20170217 11:26:00 EST eventid: 100556', 'slack' => { 'mode' => 'alarm' }, 'eventid' => '100556', 'severity' => 'Not classified' }; Host: testHostname Trigger: PROBLEM: testHostname Temp test failure Timestamp: 20170217 11:26:00 EST eventid: 100556 message is not JSON, going to proceed as with regular text Alarm message or plain event message! ** GET https://slack.com/api/chat.postMessage?token=xoxb-142571249477-pMKPwiG7qo55DaN11f8pP34M&channel=%23zabbix-alerts&attachments=%5B%7B%22color%22%3A%22%23DBDBDB%22%2C%22text%22%3A%22Host%3A+testHostname+Trigger%3A+PROBLEM%3A+testHostname+Temp+test+failure+Timestamp%3A+20170217+11%3A26%3A00+EST+eventid%3A+100556%22%2C%22title%22%3A%22PROBLEM%3ATest+Test+failure%22%2C%22fallback%22%3A%22PROBLEM%3ATest+Test+failure%22%7D%5D&as_user=true ==> 200 OK Slack response OK.

Send OK - fails -bash-4.2$ ./zbx-notify --slack --api_token=xoxb-142571249477-pMKPwiG7qo55DaN11f8pP34M --slack_mode=alarm --debug --nofork '#zabbix-alerts' 'OK:Test Test failure' 'Host: testHostname Trigger: OK: testHostname Temp test failure Timestamp: 20170217 11:27:00 EST eventid: 100556' $VAR1 = bless( { 'web_api_url' => 'https://slack.com/api/', 'last_err' => '', 'api_token' => 'xoxb-142571249477-pMKPwiG7qo55DaN11f8pP34M', 'debug' => 1, 'channel' => '#zabbix-alerts', 'mock_url' => undef }, 'SlackBot' ); $VAR1 = { 'hipchat' => { 'message_format' => 'text', 'from' => undef, 'notify' => 'true' }, 'slack' => { 'mode' => 'alarm' }, 'status' => 'OK', 'subject' => 'OK:Test Test failure', 'eventid' => '100556', 'pagerduty' => { 'client' => undef, 'client_url' => undef }, 'message' => 'Host: testHostname Trigger: OK: testHostname Temp test failure Timestamp: 20170217 11:27:00 EST eventid: 100556' }; Host: testHostname Trigger: OK: testHostname Temp test failure Timestamp: 20170217 11:27:00 EST eventid: 100556 message is not JSON, going to proceed as with regular text Alarm recovery message! $VAR1 = { 'ts' => '1487348798.000127', 'channel' => 'C3PSC1TAR' }; Failed to updateMessage: no text is provided

Send OK a second time: -bash-4.2$ ./zbx-notify --slack --api_token=xoxb-142571249477-pMKPwiG7qo55DaN11f8pP34M --slack_mode=alarm --debug --nofork '#zabbix-alerts' 'OK:Test Test failure' 'Host: testHostname Trigger: OK: testHostname Temp test failure Timestamp: 20170217 11:27:00 EST eventid: 100556' $VAR1 = bless( { 'web_api_url' => 'https://slack.com/api/', 'last_err' => '', 'api_token' => 'xoxb-142571249477-pMKPwiG7qo55DaN11f8pP34M', 'debug' => 1, 'channel' => '#zabbix-alerts', 'mock_url' => undef }, 'SlackBot' ); $VAR1 = { 'hipchat' => { 'message_format' => 'text', 'from' => undef, 'notify' => 'true' }, 'slack' => { 'mode' => 'alarm' }, 'status' => 'OK', 'subject' => 'OK:Test Test failure', 'eventid' => '100556', 'pagerduty' => { 'client' => undef, 'client_url' => undef }, 'message' => 'Host: testHostname Trigger: OK: testHostname Temp test failure Timestamp: 20170217 11:27:00 EST eventid: 100556' }; Host: testHostname Trigger: OK: testHostname Temp test failure Timestamp: 20170217 11:27:00 EST eventid: 100556 message is not JSON, going to proceed as with regular text Alarm recovery message! No messages found to be deleted ** GET https://slack.com/api/chat.postMessage?token=xoxb-142571249477-pMKPwiG7qo55DaN11f8pP34M&channel=%23zabbix-alerts&attachments=%5B%7B%22color%22%3A%22%23CCFFCC%22%2C%22text%22%3A%22Host%3A+testHostname+Trigger%3A+OK%3A+testHostname+Temp+test+failure+Timestamp%3A+20170217+11%3A27%3A00+EST+eventid%3A+100556%22%2C%22title%22%3A%22OK%3ATest+Test+failure%22%2C%22fallback%22%3A%22OK%3ATest+Test+failure%22%7D%5D&as_user=true ==> 200 OK Slack response OK.

[v-zhuravlev]

I recklessly broke it few commits ago. Thanks for reporting this. Fixed in https://github.com/v-zhuravlev/zabbix-notify/commit/2c60fd2c95565dd9b755e9bcd90215d1b47a22b2 please give it a try