Event filtering in sensu-plugin is deprecated causing the handler not to always work

[pietervogelaar]

I experience that the slack handler does not always seem to work. In the sensu-server.log I see:

{
  "timestamp": "2017-09-27T11:47:34.157563+0200",
  "level": "info",
  "message": "handler output",
  "handler": {
    "command": "handler-slack.rb",
    "type": "pipe",
    "filters": [
      "production_environment",
      "state_change_with_occurrences_critical"
    ],
    "severities": [
      "ok",
      "critical"
    ],
    "handle_flapping": false,
    "handle_silenced": false,
    "name": "slack"
  },
  "event": {
    "id": "e47824be-0e42-4586-8944-77c946ab84c3"
  },
  "output": [
    "warning: event filtering in sensu-plugin is deprecated, see http://bit.ly/sensu-plugin\nconnection refused attempting to query the sensu api for a stash\nconnection refused attempting to query the sensu api for a stash\nconnection refused attempting to query the sensu api for a stash\nwarning: occurrence filtering in sensu-plugin is deprecated, see http://bit.ly/sensu-plugin\nonly handling every 60 occurrences: myexamplehost.org/check_http_example_api\n"
  ]
}

In cases that the handler DOES work I see also the client and check hash in the hash above. How can this be fixed?

[majormoses]

We need to remove the filtering from the slack handler per: https://blog.sensuapp.org/deprecating-event-filtering-in-sensu-plugin-b60c7c500be3

[vegardx]

Any update on this? Seems related to the issues I'm seeing when using payload_template. As soon as I remove the payload_template the messages are sent to Slack. Nothing in the logs, whatsoever, except for the part about deprecated event filtering.

[pietervogelaar]

As a workaround I created my own Slack handler in python which works great:

#!/usr/bin/env python

import argparse
import httplib
try:
    import simplejson as json
except ImportError:
    import json
import os
import sys
import traceback

try:
    import requests
    requests.packages.urllib3.disable_warnings()
except ImportError:
    raise ImportError('Missing dependency "requests". \
        Do ``pip install requests``.')

try:
    import yaml
except ImportError:
    raise ImportError('Missing dependency "pyyaml". \
        Do ``pip install pyyaml``.')

OK_CODES = [httplib.OK, httplib.CREATED, httplib.ACCEPTED, httplib.CONFLICT]
UNREACHABLE_CODES = [httplib.NOT_FOUND]
WEBHOOK_TOKEN = None
DASHBOARD_URL = None

def _post_webhook(body, verbose=False):
    url = 'https://hooks.slack.com/services/{}'.format(WEBHOOK_TOKEN)

    headers = {
        'Content-Type': 'application/json; charset=utf-8',
    }

    try:
        if verbose:
            print('Webhook POST: url: %s, headers: %s, body: %s\n' % (url, headers, body))
        r = requests.post(url, data=json.dumps(body), headers=headers, verify=True)
    except:
        raise Exception('Cannot connect to Slack endpoint %s.' % url)
    else:
        status = r.status_code

        if status in UNREACHABLE_CODES:
            msg = 'Webhook URL %s does not exist. Check Slack documentation!' % (url)
            raise Exception(msg)

        if status not in OK_CODES:
            sys.stderr.write('Failed posting Sensu event to Slack. HTTP_CODE: \
                %d\n' % status)
        else:
            sys.stdout.write('Sent Sensu event to Slack. HTTP_CODE: \
                %d\n' % status)

def _post_event_to_slack(payload, verbose=False):

    if payload['check']['status'] == 0:
        status = 'OK'
        color = '#36A64F'
    elif payload['check']['status'] == 1:
        status = 'WARNING'
        color = '#FFCC00'
    elif payload['check']['status'] == 2:
        status = 'CRITICAL'
        color = '#FF0000'
    else:
        status = 'UNKNOWN'
        color = '#6600CC'

    title = '<{0}/{1}?check={2}|{1}/{2}> - {3}'.format(DASHBOARD_URL,
                                                       payload['client']['name'],
                                                       payload['check']['name'],
                                                       status)

    text = payload['check']['output']
    client = '{0} ({1})'.format(payload['client']['name'], payload['client']['address'])

    if 'environment' in payload['client']:
        environment = payload['client']['environment']
    else:
        environment = None

    if 'class' in payload['client']:
        classification = payload['client']['class']
    else:
        classification = None

    body = {
        'icon_url': 'https://avatars2.githubusercontent.com/u/10713628?v=4&s=400',
        'attachments': [
            {
                'title': title,
                'text': text,
                'color': color,
                'fields': [
                    {
                        'title': 'client',
                        'value': client,
                        'short': False
                    },
                    {
                        'title': 'environment',
                        'value': environment,
                        'short': True
                    },
                    {
                        'title': 'class',
                        'value': classification,
                        'short': True
                    }
                ]
            }
        ],
        'username': 'sensu'
    }

    try:
        _post_webhook(body=body, verbose=verbose)
        return True
    except:
        traceback.print_exc(limit=20)
        print('Cannot send event to Slack')
        sys.exit(4)

    return False

def _set_config_opts(config_file, verbose=False):
    global WEBHOOK_TOKEN
    global DASHBOARD_URL

    if not os.path.exists(config_file):
        print('Configuration file %s not found. Exiting!!!' % config_file)
        sys.exit(1)

    with open(config_file) as f:
        config = yaml.safe_load(f)

        if verbose:
            print('Contents of config file: %s' % config)

        WEBHOOK_TOKEN = config['webhook_token']
        DASHBOARD_URL = config['dashboard_url']

def main(config_file, payload, verbose=False):
    _set_config_opts(config_file=config_file, verbose=verbose)
    _post_event_to_slack(payload, verbose=verbose)

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='StackStorm sensu event handler.')
    parser.add_argument('config_path',
                        help='Exchange to listen on')
    parser.add_argument('--verbose', '-v', required=False, action='store_true',
                        help='Verbose mode.')
    args = parser.parse_args()
    payload = sys.stdin.read().strip()

    try:
        payload = json.loads(payload)
    except:
        print('Invalid JSON payload %s.' % payload)
        sys.exit(3)

    try:
        client = payload['client']['name']
        check = payload['check']['name']
    except KeyError:
        print('Invalid payload spec %s.' % payload)

    main(config_file=args.config_path, payload=payload, verbose=args.verbose)

[majormoses]

That is indeed strange, I have not had the time to look more into this as I don't use slack at our work. I can't think of why the filtering would change if you specify a template. I will try to take a quick peak and see if see if anything jumps out.

[majormoses]

so re-reading through the original message I think this might be able to be resolved by bumping the dependency of sensu-plugin to 2.x and it might solve our problems. I will put together a PR an link it here. If someone can test it out and it works we can merge and release.

[vegardx]

Thanks, I'll build it and try it today!

[vegardx]

Unless I'm doing something wrong it seems like events are still being filtered. Is there anything in particular you'd like to see as for logs? I'm doing this on a quite busy server, so logging can be a little hard to dissect or follow, so I might be overlooking something.

[majormoses]

Hmm, the only way I can explain that would be if you have defined in your handler some filter such as occurrences.

[majormoses]

Is it the same message as before?

[majormoses]

Re-opening as @vegardx says the change did not address their problem, if you could try with the released gem and let us know. In addition if I could see the handler config for slack and if there are any filters applied please include them.

[windowsrefund]

Using 3.0.0 now but handler-slack-multichannel.rb is not working.

{"timestamp":"2018-01-16T14:48:56.939399-0500","level":"error","message":"handler output","handler":{"command":"handler-slack-multichannel.rb","type":"pipe","name":"slack"},"event":{"id":"4ed44eb7-e8dd-4176-80bb-72e0afbf68bb"},"output":["/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-slack-3.0.0/bin/handler-slack-multichannel.rb:171:incompile_channel_list': undefined method join' for \"ops-alerts\":String (NoMethodError)\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-slack-3.0.0/bin/handler-slack-multichannel.rb:185:inslack_channels'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-slack-3.0.0/bin/handler-slack-multichannel.rb:205:in handle'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugin-2.3.0/lib/sensu-handler.rb:77:inblock in '\n"]}`

[majormoses]

Did you update a working setup or is this a new one? The reason I ask is that this error indicates a configuration error: https://github.com/sensu-plugins/sensu-plugins-slack/blob/3.0.0/bin/handler-slack-multichannel.rb#L169-L172. Specifically the line it is breaking on is here: https://github.com/sensu-plugins/sensu-plugins-slack/blob/3.0.0/bin/handler-slack-multichannel.rb#L171 which only triggers if you do not have any client or check configured channels and is a string rather than an array.

$ irb
irb(main):001:0> a = 'ops-alerts'
=> "ops-alerts"
irb(main):002:0> a.join(',')
NoMethodError: undefined method `join' for "ops-alerts":String
    from (irb):2
    from /var/lib/jenkins/.rbenv/versions/2.3.5/bin/irb:11:in `<main>'
irb(main):003:0> b = ['ops-alerts']
=> ["ops-alerts"]
irb(main):004:0> b.join(',')
=> "ops-alerts"

We could probably put in some extra validation in this function to help more easily surface the issue: https://github.com/sensu-plugins/sensu-plugins-slack/blob/3.0.0/bin/handler-slack-multichannel.rb#L160

[windowsrefund]

The setup worked with 2.0.0. Here is my /etc/sensu/conf.d/slack.json

{
  "slack": {
    "channels": {
      "default": "ops-alerts"
    },
    "webhook_urls": {
      "foo": "https://....",
      "bar": "https://...",
      "baz": "https://...",
      "ops-alerts": "https://..."
    }
  }
}

As you suggested, changing the default to a list rather than a string did fix the issue. Thank you.

[majormoses]

OK it looks like someone assumed that it should always be an array at some point we can probably make it so it supports either. I will not have time to work on this until maybe the weekend but maybe I can get something for you to test by next week. In the meantime I guess you should downgrade to 2.0 I am still not sure why you did not hit this with 2.0 but maybe something in the upstream sensu-plugin library caused those to be swallowed.