[WIP] Ability to auto-bootstrap module requirements on-demand

[ssbarnea]

Proposal: Ability to auto-bootstrap module requirements on-demand

Author: Sorin Sbarnea <@ssbarnea> IRC: zbr

Date: 2019-03-08

• Status: New
• Proposal type: core
• Targeted release: <future release #, ex. 2.2>
• Associated PR: <link to GH PR in ansible/proposals if PR was submitted>
• Estimated time to implement: <X days, weeks, etc.>

Motivation / Problems

Quite often ansible users discover that their (core) module is not working on the target system because they are missing a python package on this system.

Obviously that the presence of the required python modules must be cached as a fact as we do not want to re-check their presence on each execution.

Solution proposal 1

Fixing this requires extra bootstrapping tasks which require additional time to write/test/install and we could probably improve the user experience by gradually adding auto-bootrapping support to modules.

For example openstack modules do require shade python library so they should expose the required module in their metadata and ansible should attempt to verify their presense and run the bootstrapping code(role?) before running the module.

Solution proposal 2

Alternatively we could think about a different approach where the module itself could raise a special exception similar to ImportError when an import fails that triggers the bootstrapping logic in Ansible. If bootstrapping fails module fails, nothing is changed. Still, if bootstrapping suceeds the same task is called again and will likely succeed.

Testing (optional)

TBD

Documentation (optional)

Does / should this require documentation? If so, describe here. This section is optional but could be helpful.

Anything else?

If you'd like to add anything else beyond the above required and optional sections, you are welcome to do so.

[agaffney]

There are a few things to consider here:

• pip vs. system package manager for installing deps
• for installation using pip, system level or --user?
• what about python packages that require a compiler and/or C libraries/headers?
• do we automatically upgrade/downgrade python libs that already exist in the env?
• how do modules report to the user that they took these extra steps?
• how does this work for utility programs needed by a module?
• how does this work outside of python modules (such as powershell)?

[bcoca]

A few more things:

• installing via pip/npm/etc or via OS package manager is very dependent on 'policy' in the user's environment
• The user 'using the module' might not have rights to install anything and can get into trouble if they do 'install locally' .. many environments have strict policies against that.
• The module/plugin author would now have to supply all possible 'package names', they can be very different across platforms for the same library
• Version management is also different for the platforms, which is a problem for when a specific version or range of versions is required.
• What do we do when the requirement is the 'interpreter version'? i.e python >= 3.5, ruby <=2.3

[geerlingguy]

I originally posted over in https://github.com/ansible/proposals/issues/57 about this, but I think it fits more here—Execution Environments are in need of a standard way to define requirements for collection plugins and other content, and they've started to use this layout: https://github.com/ansible-collections/community.kubernetes/pull/211

[AlanCoding]

Yes, there is some stuff in-progress at https://github.com/ansible/ansible-builder, you may want to skim over the top-level README file, and then any feedback would be appreciated.

Quite often ansible users discover that their (core) module is not working on the target system because they are missing a python package on this system.

A big distinction between that effort and this discussion is that I strongly want the role of ansible-builder to be limited to the control machine, as opposed to the targeted hosts. For many plugin types like lookup plugins, inventory plugins, etc. you need the dependencies installed in the environment where ansible-playbook is called. You also really don't want that call to make unprompted changes to the system. That same machine may still run modules that interact with web APIs as local actions.

For modules running on some target host, it makes sense that you may want to use other modules in order to assure that dependencies are satisfied. That's a very different form of problem in my mind.

[bcoca]

Aside from asnible-builder and execution environments, with collections now you can also have 'boostrap plays' included in the plays directory. I would encourage a 'install_runtime_dependencies.yml' convention and tags based on plugin/module required.

[ssbarnea]

Why not extending requirements.yml with new properties in addition to roles and collections?

[bcoca]

because not all requirements are a simple in many environments you MUST use OS package, in others you don't have access to pypi.org .. and requirements are not all just python libs.

There is no universal format to handle all the transliterations of a library's/cli tool's package (pyyaml vs libpyyaml vs pythonX-pyyaml), why sometimes a playbook with many options will be only way to cover all.

[ssbarnea]

That makes sense, it as not clear that we were talking about a bootstrapping playbook.

Still, this means that it should be stored inside playbooks/ folder, just to follow the already documented layout. I do not fancy the idea of diverging from that layout.

[bcoca]

not adding new layout, just naming convention