Deduplicate role inclusion for molecule testing?

[acozine]

We're running dependencies like the common role multiple times per test run for some projects. For example, when the tests run for the lib_jobs role, molecule runs:

• include lib_jobs
• common
• ruby
• common (second time)
• deploy_user
• common (third time)
• deploy user (second time)
• mariadb
• redis
• common (fourth time)
• common again (fifth time)
• bind9
• common (sixth time)
• deploy_user (third time)
• common (seventh time)
• ruby (second time)
• passenger
• postgresql
• nodejs
• rails_app
• sidekiq_worker
• common (eighth time)
• common again (ninth time)
• bind9 (second time)
• common (tenth time)
• deploy_user (fourth time)
• common (eleventh time)
• ruby (third time)
• passenger (second time) and so on until I got bored and stopped scrolling, see https://github.com/pulibrary/princeton_ansible/runs/4957465231?check_suite_focus=true for the full run.

In production, of course, the duplication may be needed where different roles run on different servers, so the dependencies must run on each server, but it seems inefficient for testing. Can anything be done to streamline the test runs?

[hackartisan]

it's very curious. shouldn't ansible deduplicate the meta.yml entries and only run each once?

https://docs.ansible.com/ansible/latest/user_guide/playbooks_reuse_roles.html#running-role-dependencies-multiple-times-in-one-play

[acozine]

I don't think Ansible deduplicates down the nested tree, and we have a lot of nested dependencies. For example, the dependency tree for the lib_jobs role looks like this:

• ruby (which depends on common)
• deploy_user (which also depends on common)
• mariadb (which depends on deploy_user ((which in turn depends on common)) )
• redis (no dependencies!!!)
• sidekiq_worker (which depends on rails_app ((which in turn depends on common, bind9, deploy_user, passenger, postgresql, nodejs. . . most of which have dependencies of their own)) )
• rails_app (which depends on common, bind9, deploy_user, passenger, postgresql, nodejs . . . see above)
• hr_share (which depends on samba)

[hackartisan]

This makes me think we should move away from using meta for dependencies and toward just putting everything in playbooks.

[acozine]

That's a good idea, and would help with visibility. We can do this one role at a time.

Steps to implement:

• [ ] update playbooks in playbooks/ to run all role dependencies currently defined in roles/<role_name>/meta/main.yml files
• [ ] remove meta/main.yml files from roles
• [ ] configure molecule to use the production playbooks instead of duplicating playbooks in roles/<role_name>/molecule/default/ (to prevent drift in testing)

Anything else?

[acozine]

See also this SO post.

[hackartisan]

@carolyncole I'd interested if you have thoughts here, since you and @kayiwa worked so much on the molecule setup.

[acozine]

Okay, I tried a quick "proof of concept" (#2693), which showed me how complex this change would be. Definitely something to discuss broadly.

[acozine]

At the level of an individual role, refactoring the way we maintain role dependencies will require:

For each role dependency we remove from the meta/main.yml file of Role A:

• [ ] ensure the README for Role A lists the dependency
• [ ] add the dependency to the molecule/default/converge.yml file for Role A, so it runs in tests
• [ ] add the dependency to every playbook that runs Role A
• [ ] until all roles are converted, add the dependency to meta/main.yml for any other role that lists Role A as a dependency, and to any playbooks that run those roles

[hackartisan]

If we do decide to move forward on this we can close #1204 as wontfix.

[acozine]

Also related to #2892, which fixed a failure caused by dependency testing.