Integration Tests

[arm4b]

Before publishing on Ansible Galaxy, add some Integration/Smoke tests, increase infrastructure coverage.

Infrastructure Testing Frameworks

• [ ] Evaluate ServerSpec / InSpec - ruby-based. Famous, big community, nice looking and best in his class. But the team prefers Python (st2 main lang).
• [ ] Evaluate BATS - Bash Automated Testing System, which is a bit easier.
• [ ] Evaluate TestInfra - Python-based infra testing framework. Still pretty young, very small community. Intro.

  Automation

• [x] ~Molecule project - automation for testing Ansible roles~
• [x] TestKitchen - automation to spin up Vagrant or Docker or even AWS instance and test Ansible roles with Serverspec/BATS against just spinned up machines.

More info: http://stackoverflow.com/a/37006819/4533625 Continue work started in https://github.com/StackStorm/ansible-st2/issues/7

[jvrplmlmn]

I'd personally go for InSpec. Most of the code from https://github.com/StackStorm/chef-stackstorm/issues/29 can be re-used here.

[arm4b]

https://github.com/StackStorm/ansible-st2/tree/master/test/integration somewhat serves the purpose. Nightly CI Travis jobs that test both stable and development st2 versions even spot several regressions in st2 core before other stackstorm e2e tests could catch it.