search

rthallisey / clapper

Synchronize setup and deployment for the Director.
Apache License 2.0
12 stars 17 forks source link

Additional validations #98

Open dyasny opened 7 years ago

dyasny commented 7 years ago

As per the set of tests I have created for internal testing, here's a list of infrastructure components I'd like to be able to see:

  1. If the UC/OC were deployed with SSL, verify all endpoints are actually listening on SSL enabled ports
  2. check SELinux for errors all on nodes (I usually grep for AVC denials)
  3. check HAProxy (I usually curl the stat page and parse the output for errors)
  4. check Galera on all the nodes (mysql -e "SHOW STATUS LIKE 'wsrep%'"; and parse the output for problematic messages - wsrep_local_state_comment - must be in sync, wsrep_cluster_status, wsrep_cluster_size - must equal to the number of controllers etc etc)
  5. check pacemaker on all the relevant nodes (I look for failure messages in pcs status)
  6. check RabbitMQ (rabbitmqctl status and look for lines that start with "Error")
  7. check MongoDB
  8. Check Redis
  9. services status on all nodes. Will be even more relevant with composable roles, since the service list will match the service to node mapping in the deployment yaml
  10. check ceph (ceph health and ceph status are the commands I use)
  11. check keepalived (for the versions where it's relevant) - this one cna be tricky since there is no status command, I had to parse the config file and then verify the IPs and services were actually there using nc/telnet/curl

I can share the code for my tests internally if that will help

tomassedovic commented 7 years ago

Good stuff, thanks!

FYI, this repo is deprecated and everything has moved here:

http://git.openstack.org/cgit/openstack/tripleo-validations/

And we track issues & suggestions here:

https://bugs.launchpad.net/tripleo/+bugs with the validations tag.

I'll have to pop off for the year, but I can add the issues there in a few weeks if you don't beat me to it.