Add proper infra tests

[jnv]

Probably using molecule with Testinfra. It can be run on Travis.

Lighter solution with multi-distro testing: https://www.jeffgeerling.com/blog/2018/how-i-test-ansible-configuration-on-7-different-oses-docker

[jnv]

https://github.com/nickjj/rolespec

[conorsch]

I strongly encourage you to adopt Molecule v2, which is a dream for testing Ansible roles. Integration with testinfra is provided out of the box.

[jnv]

@conorsch Molecule seems a bit too robust for needs of this role, although it is interesting. Do you have any example of role tested by Molecule on Travis so I can learn/copy from it?

[letharion]

@jnv I'm curious about what sort of testing you consider necessary here. I struggled a little bit with what exactly would be appropriate for this, and here's a suggestion for how to go about it:

1) Bundle an old .deb file in the repo for some innocuous application, say bsdgames (https://packages.debian.org/wheezy/bsdgames) 2) Install the old .deb, and run this role against the same environments. 3) Call run-parts on the daily cron, which should now upgrade the old package. 4) Check the package version.

It's not entirely fail safe, as the included package might not be in all distros at all times, and run parts won't run with exactly the same env as the real cron, but it's the best I can come up with ATM. Thoughts?

[jnv]

@letharion Thanks for the suggestion! I like your ideas but to be honest, I wouldn't like to take this route.

The issue with these kind of roles is that they're walking the fine line between simple configuration management and working around shortcomings of the software which is being integrated. I'm trying to avoid the latter one, so I don't think the tests should actually check whether unattended-upgrades software itself is working. Rather my idea was to check that:

• The role installs all needed dependencies
• The role properly writes configuration options
• These options are visible to apt and unattended-upgrades

Other stuff, like setting up cron and actually upgrading packages, is concern of the software itself; if it's broken, it needs to be fixed upstream, not patched by this role.

[letharion]

is concern of the software itself; if it's broken, it needs to be fixed upstream, not patched by this role.

That is entirely true, and I'm not sure why I didn't think about it this way to begin with.

I guess this is where I confess I'm completely new to ansible role testing, so you'll have to read this comment in that context. Looking around for ways to help with this particular issue, I ended up on http://docs.ansible.com/ansible/latest/test_strategies.html, and the way I read this recommendation:

Ansible resources are models of desired-state. As such, it should not be necessary to test that services are started, packages are installed, or other such things. Ansible is the system that will ensure these things are declaratively true. Instead, assert these things in your playbooks.

is that tests such as "installs all needed dependencies" should be avoided, if the tests end up effectively re-stating what the role is already saying. Perhaps there's something in between though that should be tested?

[MikeTheCanuck]

I wonder as well about how to test that the code itself actually corrects the issue it’s there to address (and not test all the infra dependencies around it).

It seems that unless you have some test that first verified that the target is not in the end state before this package is deployed, then the post-deploy positive rest result would be meaningless (since the target could have been in the correct state before the code deployed).

I’m not reading as closely as I should, so I may be missing something obvious.

[jnv]

My goal with tests for this role is to verify that it works across various distributions and versions. I agree that testing just fact that something was installed is not very helpful. The tests should at least check that the system is in consistent state and that user's preferences are correctly propagated.

I have tried a few approaches and settled on using Ansible for tests; it seems adequate for such a simple role. You can take a look at test playbook. Basically it provisions a bunch of containers for each distro/version combination with provision_docker role and:

1. Executes the role,
2. runs the role again for idempotency check,
3. performs assertions against apt-conf output,
4. performs dry run of unattended-upgrades

There are a few things to improve, namely testing scenarios (e.g. there is no other check for setting variables) and parallel runs on CI. Also, since tests use official distribution images, Python needs to be installed every time.

Previously I have also explored lower-level approach with separate testing environments, as described on Travis blog and by Jeff Geerling. I especially like that each distro/version combination has a separate environment and the failure is clearly visible in the build matrix (for example here). However Ansible runs inside the containers, which requires extra dependencies, so it is not really a clean slate system for testing. Also it is a bit slower since only five builds can run in parallel. You can see this approach in tests-envs branch.