Role Testing POC - for Partner content

[chouseknecht]

Create a POC that includes the following:

• A role that installs and configures a thing in a VM
• Within the role, generate the molecule directory structure
• Add a custom Ansible playbook that tests the role. It should include multiple sets of tasks and asserts that exercise and test the role.
• A job in Software Factory that runs the test playbook for the same platforms enumerated in the molecule secnarios
• Connect the job to the role repository so that the job executes when a PR is submitted

From the POC we want to learn the following:

• How to use Molecule
• How we might ask Galaxy contributors to structure Molecule tests
• How to create and execute jobs in Software Factory
• How gating works in Software Factory
• What data is available from a job run, and how might it inform Galaxy?

[paulfantom]

Molecule also uses yamllint internally (as well as ansible-lint), so this isn't exactly a replacement.

What do you want to use molecule for? Full role testing or just some linting?

[chouseknecht]

@paulfantom see revised initial comment. Hadn't really thought through the whole thing until now.

[paulfantom]

I am doing exactly that in @cloudalchemy roles however not in VM but in docker containers! I think I can share my knowledge if you want.

[webknjaz]

@paulfantom please do

[paulfantom]

What form do you want? A video conference call?

[kharkevich]

Hey, I'm use molecule for all my roles too. I'm ready to share my knowledge with anyone. Also, I use docker (for public) and another drivers (for private) in molecule testing roles

[chouseknecht]

With help from @rcarrillocruz we've learned that Molecule tests, assuming the Docker provisioner, can execute on Zuul with no changes. (Keep in mind, the use fo Zuul as a CI platform is only for partner content.) We also know that Travis can execute Molecule tests as well, assuming the Docker provisioner, with no changes.

Based on these preliminary findings, it seems safe to say that we can move forward with Molecule as our testing framework, and craft some working examples of how both Community and Partner contributors can implement Molecule.

For partners, we may have to be prescriptive about how the tox.ini file is configured. For example, we may only support testing of recent versions of Ansible, and not go all the way back to 2.0.

On the Galaxy import side we will want to do the following (I think):

• Verify that tests passed on a public CI platform. Initial support will be implemented for Travis, but we'll want to expand this.
• Maybe inspect the existence of a Molecule directory in the repo
• Maybe devise a way to inspect the level of testing that's being done - sort of like code coverage

[chouseknecht]

Still to be determined... How to use molecule to test a multi-content repository.

[paulfantom]

I have couple of thoughts about this topic, so I'll just dump them here. Maybe it will be useful for someone.

1) When using docker provisioner you still need to get images from somewhere and usually official docker images of linux distros don't come with init system. This means that using any service management module (ex. systemd) won't work in such containers. I solved it by delivering my custom images, but I wonder how this will be solved here?

2) What would be run in default molecule scenario? Role with default variables?

3) "devise a way to inspect the level of testing that's being done - sort of like code coverage" I would love that! However from my experience there is no good automated way of doing that. Molecule supports concept of scenarios and I ended up using two of them: one for role with only default values and second one where everything is changed. But there might be a possibility when there is a need for more scenarios for full code coverage and that usually breaks automated tooling.

4) "inspect the existence of a Molecule directory in the repo" That could be a good start for checking role quality and later allow sorting roles by such metric or just have a filter. It should be fairly simple to implement and have quite large impact on UX.

[chouseknecht]

It'll be up to the content author to provide the tests and test scenarios. If you want to use custom images, that'll work, as long as they're publicly accessible.

FWIW, we're using a copy of your grafana role for our POC.

[webknjaz]

To simplify the lookup of (molecule) tests presence, it might be a good idea to provide a way to specify a path to them and maybe a command for running those in the metadata.

[paulfantom]

Molecule tests are quite easy to localize as molecule has quite static directory layout:

molecule/
  <scenario_directory>/           <-- usually named "default"
    molecule.yml                  <-- scenario config
    {create,destroy,prepare}.yml  <-- playbooks to manage test env
    playbook.yml                  <-- actual role execution playbook
    tests/                        <-- directory with verifier (ex. testinfra) tests

In simplest and most naive way you just need to check if there is a molecule directory.

[webknjaz]

Fair enough. If we can force everyone to follow this :)

[paulfantom]

Add to molecule directory to ansible-galaxy init and hope for the best? :smile: