Module and role search path for external packages

[pcahyna]

Proposal: Module and role search path for external packages

Author: Pavel Cahyna @pcahyna Date: 2017/03/28

• Status: New
• Proposal type: process
• Estimated time to implement: 1 day

Motivation

There may be third-party roles and modules which the providers will want to distribute as distribution-specific binary packages (RPMs, ...) instead of using Ansible Galaxy. There are already some RPMs in Fedora which provide Ansible roles and/or modules, examples include ceph-ansible, kubernetes-ansible, ansible-openstack-modules. There are valid reasons why the authors may want to keep such content outside Galaxy. Vendors of third-party software may want to provide and support Ansible roles and modules to manage their products in their usual way (package repositories). The same holds for OS vendors who will want to provide management tools for their OS in the same way as the OS itself (again, package repositories). As there is no policy on where to put roles and modules, such packages usually put their roles and modules in their own directories and the packages do not work out-of-the box (i.e. Ansible does not find the roles/packages without changing configuration files or environment variables). Distributions should have packaging policies for Ansible roles and modules to solve this, but they will need to be supported by some changes to the Ansible source, as Ansible does not currently provide a good way to set distribution-specific search path for roles and modules.

Concerning modules, according to the ansible(1) manual page, the default search path to load modules from is /usr/share/ansible, but this is not actually true.

Concerning roles, according to https://galaxy.ansible.com/intro#download "(...) Ansible downloads roles to ANSIBLE_ROLES_PATH, which defaults to /etc/ansible/roles". This directory could be leveraged for the intended purpose, but it seems suboptimal, as roles are supposed to be reusable and thus rarely to be modified, therefore do not belong to /etc, where host-specific system configuration belongs (according to the FHS).

In principle distributions could set library= and roles_path= in ansible.cfg to achieve their desired policies without having to modify Ansible sources, but this setting would be overriden by user-specified customizations using environment variables or configuration files, which does not seem correct. It would be preferable to let user customizations be prepended to such distribution-specific search paths instead of overriding them. As an analogy, PYTHONPATH is being prepended to the distribution-specific Python module search path (such as /usr/lib64/python2.7/site-packages) instead of overriding it.

Problems

• No established location for third-party roles and modules distributed outside Ansible and Galaxy
• Roles and modules provided by packages do not work out-of-the-box - they are not in the search paths since there are no established search paths (except the default settings)
  • there is no established search path for distribution-provided roles (except /etc/ansible/roles) and modules
  • there is no established search path for locally-installed roles and modules (by an administrator or an unprivileged user).
• /etc/ansible/roles is the default setting of the search path for roles, but /etc is a poor choice for mostly immutable data such as roles
• If distribution-specific search paths are set as defaults in the configuration file, they are overriden by user settings, which is not desirable (user-set search path shall augment the distribution-provided one, not replace it).

Solution proposal

• Provide a search path for modules, easily customizable by distributions (so probably kept in a separate source file). The paths provided in the ANSIBLE_LIBRARY environment variable and the library configuration parameter should be prepended to it, instead of overriding it. Suggested default is ~/.ansible/modules:/usr/local/share/ansible/modules:/usr/share/ansible/modules. The first component allows the user to install modules without requiring root privileges (~/.local/share/ansible/modules could be also used, but ~/.ansible exists already). The second component allows the administrator to install modules outside the control of the distribution's packaging system. The third component is for modules installed using the distribution's packaging system. The search path should also include direct subdirectories of those (this is easy, since PluginLoader._get_paths() in lib/ansible/plugins/__init__.py already supports including subdirectories since 55e26db290860df6692dd8ef7595deb0d8478d5c.)
• Provide a search path for roles, easily customizable by distributions (so probably kept in a separate source file). The paths provided in ANSIBLE_ROLES_PATH and the roles_path configuration parameter should be prepended to it, instead of overriding it. Suggested default is ~/.ansible/roles:/etc/ansible/roles:/usr/local/share/ansible/roles:/usr/share/ansible/roles. /etc/ansible/roles provides the compatibility with the current default and the possibility of changing the installed roles by copying them from /usr/share to /etc and modifying them (similar system exists e.g. for udev rules). The other components have the same rationale as in the case of modules. http://docs.ansible.com/ansible/galaxy.html#roles-path already suggests using /etc/ansible/roles:~/.ansible/roles for ANSIBLE_ROLES_PATH, so this proposal will only make this suggestion into a default. Ansible Galaxy will then download the roles to ~/.ansible/roles (if unprivileged and the directory exists) or /usr/local/share/ansible/roles (if privileged and the directory exists) or /etc/ansible/roles.

Documentation

• The documentation for the -M command line parameter, the ANSIBLE_LIBRARY environment variable, the /usr/share/ansible/ path in the manual page will need to be updated. In particular, ANSIBLE_LIBRARY will not override the default path anymore, but will prepend to it. (In practice, this should not be a problem, since there apparently is no default at the moment and the documentation is wrong.)
• Conventions, Best Practices, and Pitfalls (http://docs.ansible.com/ansible/dev_guide/developing_modules_best_practices.html) will need to be updated to instruct RPM packagers to use a subdirectory of /usr/share/ansible/modules (currently it suggests /usr/share/ansible).
• http://docs.ansible.com/ansible/galaxy.html#roles-path and https://galaxy.ansible.com/intro#download will need to be updated.

[jctanner]

+1

[jctanner]

https://github.com/ansible/ansible/pull/23038

[bcoca]

https://github.com/ansible/ansible/issues/12089 <= also removed hardcoded paths from man pages as they were 'wrong' depending on distro.

[sivel]

I would love to see us use setuptools entry_points for purposes of plugin loading. We've talked about it in the past. That allows modules to be pip installable, and automatically found.

[tima]

While on the topic of standardized paths for extending Ansible I've really wanted to see this implemented to simplify the management of extensions like plugins and, taking this proposal into account, roles: https://github.com/ansible/ansible/issues/12157

[tabowling]

Do we still need this? I am using ansible-2.3 on Fedora. Changing roles_path in /etc/ansible/ansible.cfg to this:

roles_path    = /etc/ansible/roles:/usr/share/ansible/roles

Seems to give me the behavior I desire.

I have linux-system-roles installed from galaxy in /etc/ansible/roles. I have rhel-system-roles installed from the test package installed to /usr/share/ansible/roles.

With these lines in my playbook:

  roles:
    - role: rhel-system-roles.network
    - role: linux-system-roles.network

It works, and seems to prefer the linux-system-roles. If I remove that line, it falls back to the rhel-system-roles perfectly.

[pcahyna]

@tabowling Of course one can accomplish the same by changing the configuration file, but the point of the proposal was to provide reasonable defaults that anybody (who packages Ansible roles) can rely on.

[bcoca]

@pcahyna the big issue is that 'reasonable' changes a lot depending on distro/OS, the following have all been proposed as locations, mostly to match where 'their distro/os' expects such things:

/usr/ansible/
/usr/lib/ansible/
/usr/share/ansible
/usr/local/ansible
/opt/ansible
/var/lib/ansible
/var/cache/ansible
/var/ansible

which makes me just want to ...... let package managers decide for their own targets.