Add integration testing for all the plugins repos

majormoses commented 7 years ago

See example PRs linked below!

[ ] sensu-plugins-ansible
[ ] sensu-plugins-apache
[ ] sensu-plugins-aws
[x] sensu-plugins-beanstalk
[ ] sensu-plugins-bigpanda
[ ] sensu-plugins-bluepill
[ ] sensu-plugins-campfire
[ ] sensu-plugins-cassandra
[ ] sensu-plugins-ceph
[ ] sensu-plugins-cgroups
[ ] sensu-plugins-chatwork
[ ] sensu-plugins-chef
[ ] sensu-plugins-clockworksms
[ ] sensu-plugins-conntrack
[ ] sensu-plugins-consul
[ ] sensu-plugins-couchbase
[ ] sensu-plugins-couchdb
[ ] sensu-plugins-cpu-checks
[ ] sensu-plugins-cucumber
[ ] sensu-plugins-dashing
[ ] sensu-plugins-datadog
[x] sensu-plugins-dcos
[ ] sensu-plugins-dhcp
[ ] sensu-plugins-disk-checks
[x] sensu-plugins-dns
[ ] sensu-plugins-docker
[ ] sensu-plugins-dovecot
[ ] sensu-plugins-eep
[x] sensu-plugins-elasticsearch
[ ] sensu-plugins-entropy-checks
[ ] sensu-plugins-environmental-checks
[ ] sensu-plugins-erlang
[ ] sensu-plugins-etcd
[ ] sensu-plugins-execute
[ ] sensu-plugins-eye
[ ] sensu-plugins-feature-requests
[ ] sensu-plugins-filesystem-checks
[ ] sensu-plugins-flowdock
[ ] sensu-plugins-fluentd
[ ] sensu-plugins-freeradius
[ ] sensu-plugins-ftp
[ ] sensu-plugins-gearman
[ ] sensu-plugins-geckoboard
[ ] sensu-plugins-gelf
[ ] sensu-plugins-github
[ ] sensu-plugins-gluster
[ ] sensu-plugins-golang
[ ] sensu-plugins-google-spreadsheet
[ ] sensu-plugins-gpg
[ ] sensu-plugins-graphite
[ ] sensu-plugins-graylog
[ ] sensu-plugins-growthforecast
[ ] sensu-plugins-haproxy
[ ] sensu-plugins-hardware
[ ] sensu-plugins-hbase
[ ] sensu-plugins-hipchat
[ ] sensu-plugins-http
[ ] sensu-plugins-hubot
[ ] sensu-plugins-icecast
[ ] sensu-plugins-iis
[ ] sensu-plugins-imap
[ ] sensu-plugins-imkayac
[ ] sensu-plugins-influxdb
[ ] sensu-plugins-io-checks
[ ] sensu-plugins-ipmi
[ ] sensu-plugins-ipvs
[ ] sensu-plugins-irc
[ ] sensu-plugins-java
[ ] sensu-plugins-jenkins
[ ] sensu-plugins-jolokia
[ ] sensu-plugins-kannel
[ ] sensu-plugins-kegbot
[ ] sensu-plugins-kubernetes
[ ] sensu-plugins-ldap
[ ] sensu-plugins-librato
[x] sensu-plugins-load-checks
[ ] sensu-plugins-logs
[ ] sensu-plugins-logstash
[ ] sensu-plugins-lvm
[ ] sensu-plugins-lxc
[ ] sensu-plugins-mackerel
[ ] sensu-plugins-mailer
[ ] sensu-plugins-mailgun
[ ] sensu-plugins-memcached
[ ] sensu-plugins-memory-checks
[ ] sensu-plugins-mesos
[ ] sensu-plugins-messagemedia
[ ] sensu-plugins-mongodb
[ ] sensu-plugins-monit
[ ] sensu-plugins-mysql
[ ] sensu-plugins-nbzget
[ ] sensu-plugins-netscaler
[ ] sensu-plugins-network-checks
[ ] sensu-plugins-newrelic
[ ] sensu-plugins-nginx
[ ] sensu-plugins-nrpe
[ ] sensu-plugins-ntp
[ ] sensu-plugins-nvidia
[ ] sensu-plugins-officehours
[ ] sensu-plugins-openldap
[ ] sensu-plugins-openstack
[ ] sensu-plugins-opentsdb
[ ] sensu-plugins-openvpn
[x] sensu-plugins-opsgenie
[ ] sensu-plugins-pacemaker
[ ] sensu-plugins-pagerduty
[ ] sensu-plugins-percona
[ ] sensu-plugins-php-fpm
[ ] sensu-plugins-pingdom
[ ] sensu-plugins-ponymailer
[ ] sensu-plugins-postfix
[x] sensu-plugins-postgres
[ ] sensu-plugins-process-checks
[ ] sensu-plugins-puma
[ ] sensu-plugins-puppet
[ ] sensu-plugins-pushover
[ ] sensu-plugins-qmail
[ ] sensu-plugins-rabbitmq
[ ] sensu-plugins-raid-checks
[x] sensu-plugins-redis
[ ] sensu-plugins-request-tracker
[ ] sensu-plugins-resque
[ ] sensu-plugins-rethinkdb
[ ] sensu-plugins-riak
[ ] sensu-plugins-riemann
[ ] sensu-plugins-rspec
[ ] sensu-plugins-selinux
[ ] sensu-plugins-sensu
[ ] sensu-plugins-sentry
[ ] sensu-plugins-serverspec
[ ] sensu-plugins-sftp
[ ] sensu-plugins-sidekiq
[ ] sensu-plugins-signifai
[ ] sensu-plugins-sip
[ ] sensu-plugins-skel
[ ] sensu-plugins-skyline
[ ] sensu-plugins-slack
[ ] sensu-plugins-sms
[ ] sensu-plugins-snmp
[ ] sensu-plugins-solr
[ ] sensu-plugins-splunk
[ ] sensu-plugins-springboot
[ ] sensu-plugins-ssl
[ ] sensu-plugins-statuspage
[ ] sensu-plugins-strongswan
[ ] sensu-plugins-supervisor
[ ] sensu-plugins-switchvox
[ ] sensu-plugins-syslog-ng
[ ] sensu-plugins-telapi
[ ] sensu-plugins-telegram
[ ] sensu-plugins-tempodb
[ ] sensu-plugins-tomcat
[ ] sensu-plugins-trafficserver
[ ] sensu-plugins-tripwire
[ ] sensu-plugins-twemproxy
[ ] sensu-plugins-twilio
[ ] sensu-plugins-twitter
[ ] sensu-plugins-ubiquiti
[ ] sensu-plugins-uchiwa
[ ] sensu-plugins-unicorn
[ ] sensu-plugins-ups
[ ] sensu-plugins-uptime-checks
[ ] sensu-plugins-varnish
[ ] sensu-plugins-victorops
[ ] sensu-plugins-vmstats
[ ] sensu-plugins-vsphere
[ ] sensu-plugins-windows
[ ] sensu-plugins-wordpress
[ ] sensu-plugins-xen
[ ] sensu-plugins-xmpp
[ ] sensu-plugins-zendesk
[ ] sensu-plugins-zookeeper

luisdavim commented 7 years ago

I believe sensu-plugins-dcos is covered.

barryorourke commented 6 years ago

sensu-plugins-load-checks can be ticked of now.

absolutejam commented 6 years ago

This list amazes me. Didn't realise there were so many.

majormoses commented 6 years ago

This is actually a subset as it has probably not been updated for any new plugins added since it's creation but yes there are 180+ plugins. It is a very large effort to push this kind of testing across all the plugins. I will have some blog posts coming out talking about this.

absolutejam commented 6 years ago

@majormoses is there a good place to provide some example tests or things to test?

I know one example that would have helped me when I was trying to get to grips with both Ruby and rspec for a PR I submitted is check-dns-shared. The tests for CLI args are really clear and concise (And I'm determined rewrite mine to match, haha), but they act as a great sanity check that anyone can write.

majormoses commented 6 years ago

I need to create something a bit more formal in the form of a best practices doc but between the following links I think is a good start while I work on writing this up better:

Also I have added the example you mentioned to the skel along with the skeleton of all the framework config. I think this is a very good start for now.

As far as things to test I think that really depends on the particular plugin so we might want to make individual issues for more specific things But I think there are a few universal things to check:

is each check executable and has a working ruby environment? See a shared example and it's invocation
do passing conflicting args/config through an unknown|3?
Each check/metric, handler, mutator should probably have a min of 1 positive (returns ok|0) check and another to demonstrate that say a critical|2 or |warning|1 threshold is met. Certainly no hard rule and ideally there will be more checks for more things but I find that just doing something makes it easier for people to grok what you want when you ask them to add tests if there are already tests in the repos.

If I had to start in a new repo I find the easiest ones to work with are the high level checks like ping, alive, healthy, etc. It usually requires the least amount of effort to bootstrap/mock and test. This sets the stage without figuring out how to do multi master postgres replication. While eventually we will want this eventually start with the simplest of use cases. The more complicated stuff will come when people start seeing the value of the tests failing when they break stuff.

I have linked a few examples in this issue as well. I'd like an actual doc in addition to the blog posts, skel enhancements, and the few I have done so far is what I have time for right now. Maybe next week I will start creating this more formal document with more examples and talking about what types of things to test but there is some of that in the blog posts already.

One more thing to add is to start looking at your actual check definitions and try to write a test for each one with those options (not necessarily the values but exercising the options). If even 10% of people using plugins did this we would have crazy good coverage. Ideally we would cover every option but I suggest testing either what you are doing (being selfish is ok on this matter) or to distill the most common use cases. Ones that are super generic like check-http.rb might be a little hard to start thinking about as there are so many things to check but I am available to shed some light on a per case basis until we have done enough analysis across all the plugins to come up with a better list of things to check. The best tests are basically just using them like you would in production. If you have to tell your test to do funky things then it is probably not the best test as you are already changing what you are doing from production. One of the reasons I wrote those tests like that is because that how you would actually test it manually. You would create some DNS records and then check for the ones you want the same as cli or through a sensu check definition.

sensu-plugins / community

Add integration testing for all the plugins repos #41