Automated Testing

[philipnorton42]

Ok, so now that we have a password free version of Vlad we can start some automated testing. I would prefer some way of testing every commit and getting a report if anything went wrong. Future plans would be to supply different settings files but for now I just want to test a default install.

Travis CI I've tried (for a few hours) to get Travis CI to do set up a vagrant environment and run Vlad. This doesn't work. There is nothing specifically saying 'Virtualbox/Vagrant is not supported' but after trying all sorts of things to get it working there is simply no way this can work. What is annoying is that it ~almost~ works. Travis CI is a great service though and I was impressed by what it did do.

Jenkins This is definitely possible, but might take a little bit of setup, and will most likely be self hosted. The difficulty here is that we can't use a VM to run the tests and as such the hosting costs are quite high. We could set it up on a local machine somewhere, but I don't currently have one spare and it might even be hosted on my broadband connection. Frankly, I don't really want to have to support a public facing machine like this so I would prefer some form of hosted solution.

Anything else So, are there any hosted solutions that would allow us to test a vagrant build? I went for Travis CI originally, but it doesn't support what we need so if people can post potential solutions then I'm happy to setup the integrations.

[philipnorton42]

So it turns out that it is possible to test on Travis CI, but only through running the ansible tasks locally. This isn't perfect, but it's a good start. I've added syntax checking and am moving through the automated build of the system but you can see progress here: https://travis-ci.org/hashbangcode/vlad

[dixhuit]

Brilliant. We need this so bad!

I'd love to help with this but have never touched Travis before. If you fancy bringing me up to speed at some point then I'd gladly muck in.

[philipnorton42]

Essentially, everything is controlled via the .travis.yml file. We set some stuff up, run some install commands and then execute the ansible-playbook command a few times to syntax check the playbook and then actually run it locally (ie. not through Vagrant). I've managed to get through most of the playbook execution but right now it's failing on not finding a php.ini file and I'm trying to figure out why...

[philipnorton42]

The cool thing about Travis is that it tests every time a push is made or when a pull request is made. There just needs to be a .travis.yml file in the repo. Github then reports on the status of the build and will warn you when merging pull requests that fail. Nice :)

[philipnorton42]

Ok, after much messing about (and actually breaking Vlad in a couple of places) I have reverted the automated running of the ansible tasks and just left the syntax checking bit in. I'd started to go down the route of altering Vlad to allow the tests to run, which isn't really ideal :)

Currently, travis will check to make sure that there are no syntax errors in the code, which is great. But we still need to figure out some way of running an end to end test on a system in a automated (or semi-automated) way.

[dixhuit]

Top work Phil. Shame we can't test end to end with Travis but automated syntax checking will still save us from plenty of potential problems :)

I suggest that we return to this after we tag the next release.

[mbarcia]

Philip, I too think this is the way to go. But I too think it will require a good amount of work (sigh). :-)

But it does have an enormous potential ie. Travis revealed a bug in one of my recent PRs, which wasn't happening in my environment, and I was able to fix it right away.

BTW, you might want to take a look at rolespec. Its summary "A shell based test library for Ansible that works both locally and over Travis-CI." looks very promising.

Cheers

[philipnorton42]

Interesting, I'll take a closer look at rolespec :) Thanks!

[dixhuit]

ansible-lint is another project we may want to evaluate as part of our tests. I'm not sure how much this would overlap with Travis' syntax checking though.

[philipnorton42]

I quite like ansible-lint so I've added that (it was quite easy) and tweaked a few things to allow tests to pass.

One thing I did exclude was the resources check, I did this by using the following command:

ansible-lint -x resources site.yml

This was done for a few reasons, but the main thing it checks for is when using a 'command' task when the module exists. But this is needed in a few situations in various playbooks. Here are a couple of examples:

In the PHP Comper install playbook:

- name: composer | install Composer (shared)
  shell: curl -sS https://getcomposer.org/installer | php && mv composer.phar /usr/local/bin/composer creates=/usr/local/bin/composer
  tags: composer
  sudo: true

Which produces the following warning:

[ANSIBLE0006] curl used in place of get_url module
/vagrant/vlad/playbooks/roles/php/tasks/composer.yml:0
Task/Handler: composer | install Composer (shared)

This is possible to change, but I'm not sure about altering it away from the established Composer install method just to allow for a lint check.

In the PimpMyLog playbook:

- name: ensure that the package isn't effected by mode changes
  command: git config core.filemode false chdir=/var/www/pimpmylog
  sudo: true
  changed_when: false

Which produces the following warning

[ANSIBLE0006] git used in place of git module
/vagrant/vlad/playbooks/roles/pimpmylog/tasks/pimpmylog.yml:0
Task/Handler: ensure that the package isn't effected by mode changes

This is interesting as there is no way around this. I suppose we could just remove this task, but it causes problems when running provisioning on the box a second time (the git repo complains that there are changes when we've just changed the mode).

[dixhuit]

Agreed that this is the right approach here. Linting for linting's sake is crazy :)