sensu / sensu

Monitoring for today's infrastructure.
https://sensu.io
MIT License
2.89k stars 386 forks source link

Check dependencies are only filtered by the check script and not the server #931

Closed warmfusion closed 7 years ago

warmfusion commented 9 years ago

As a user when i configure a service check with a dependency on one or more local or remote checks I would expect the server to handle checking the state of dependents

However, filtering appears to be exclusively handled by the sensu-plugin which may only optionally used by the check command being implemented.

This means any check I create that does not also determine the state of its dependants or use the sensu-plugin parent class will not be filtered properly.

portertech commented 9 years ago

Sensu handlers using sensu-plugin make use of this custom functionality. Sensu checks do not check their dependencies themselves. Check dependencies is not part of core.

warmfusion commented 9 years ago

Is that why dependencies have mysteriously (as I can't find a commit that removed them) disappeared from the documentation?

portertech commented 9 years ago

Check dependencies is not a Sensu Core attribute, it's custom, used only by sensu-plugin. This attribute, along with check occurrences and refresh need to be documented under sensu-plugin, in the docs. Thinking sensu-plugin gets its own page.

portertech commented 9 years ago

Or we can add a heading to the checks reference documentation with the sensu-plugin attributes.

portertech commented 9 years ago

https://sensuapp.org/docs/latest/checks#sensu-plugin-attributes

hanynowsky commented 8 years ago

Hello! Please give me a check and a handler that make use of the sensu-plugin and thus make the dependencies functionality available for the given check.

I have a serious problem here and I am risking my job because I bet everything on SENSU! I am unable of making use of the dependencies functionality.

# cat check_procs_apache2.json 
{
    "checks": {
        "check_apache_proc": {
            "command": "check-process.rb -p apache2",
            "interval": 60,
            "occurrences": 2,
            "subscribers": ["apachetwo"],
            "handlers": ["flapjack", "logstash", "irc", "riemann"]
        }   
    }   
}
root@sensu-server (sensu) in /etc/sensu/conf.d/check 
# cat check_procs_crond.json 
{
    "checks": {
        "check_crond_proc": {
            "command": "check-process.rb -p cron -C 1",
            "dependencies": ["check_apache_proc"],
            "interval": 60,
            "occurrences": 2,
            "subscribers": ["production"],
            "handlers": ["flapjack", "logstash", "irc"]
        }
    }
}

When apache is stopped on a client and then have CRITICAL event from the apache proc check, and if I stop CRON daemon on the same client, I would still get a critical event from the cron proc check! Which means the dependencies functionality is not called at all and this is driving me crazy! I am using sensu 0.22.0

Please help

warmfusion commented 8 years ago

@hanynowsky - This isn't StackOverflow - Your issue relates to specific configuration and isn't necessarily related to the issue I reported.

In your case, Sensu is operating as designed - when both services are stopped it is correct for sensu to report criticals on both those services. The problem I think you may be having is that the sensu-handler that is processing the events isn't treating the dependencies correctly; This ticket reports my concern that sensu-server doesn't filter events, but demands that handlers do that work (Duplicating commonly required effort on each handler).

You need to check the handlers you're using to fire notifications/logs/alerts to ensure they are using the sensu-plugin (or check the dependencies itself).

calebhailey commented 8 years ago

Related: https://github.com/sensu/sensu/issues/861

portertech commented 7 years ago

Check dependency filtering is now available via a Sensu extension: https://github.com/sensu-extensions/sensu-extensions-check-dependencies

This extension will be included in Sensu 0.29