How to declare/handle external python dependencies in ansible collections/plugins

mutech commented 1 year ago

Summary

I am developing an ansible module in a collection that requires an external python dependency.

In my development context, that dependency is present, but when I try to execute the module on a target, it is not found.

So far, I encountered dependencies in actions, info modules or filters, that is in ansible plugins executing on the controller. Sometimes they are installed automatically, if not I just installed them manually on the controller. That's fine.

I assumed that Python dependencies required on the target would be packed into the Ansiballs, but from what I can see, that does not happen or if it does, it doesn't work in my case. Other than on the controller, having to install dependencies on target systems would imho not be acceptable.

I was trying to find any mention of this problem in the documentation, some best practices, even talked to bard and chatgpt and found nothing.

Then I tried to find examples for how others handle this, and in the limited time I invested for this search, I did not find any modules that use external dependencies. To me it looks like the reason for this is probably because modules evade using dependencies to avoid my current problem.

I think the documentation for developing modules should mention this, either telling authors to not use dependencies or how to do it correctly.

I'm new to both Python and Ansible, so there is a good chance that this is the reason why I hit the wall hard here, but since I read somewhere that roles should not be used to implement logic, a rookie trying to implement something in Python using external libraries should not be a fringe use case.

Meta: I probably chose the wrong issue type, but I don't find a better one.

Issue Type

Documentation Report

Component Name

lib/ansible

Ansible Version

$ ansible --version
ansible [core 2.15.5]
  config file = /home/mu/share/projects/utech/ansible/ansible.cfg
  configured module search path = ['/home/mu/share/projects/utech/ansible/library']
  ansible python module location = /home/mu/.asdf/installs/python/3.11.6/lib/python3.11/site-packages/ansible
  ansible collection location = /home/mu/share/projects/utech/ansible:/home/mu/share/projects/utech/ansible/{{ ANSIBLE_HOME }}/collections:/usr/share/ansible/collections
  executable location = /home/mu/.asdf/installs/python/3.11.6/bin/ansible
  python version = 3.11.6 (main, Oct 17 2023, 10:07:08) [GCC 11.4.0] (/home/mu/.asdf/installs/python/3.11.6/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True

Configuration

# if using a version older than ansible-core 2.12 you should omit the '-t all'
$ ansible-config dump --only-changed -t all
COLLECTIONS_PATHS(/home/mu/share/projects/utech/ansible/ansible.cfg) = ['/home/mu/share/pr>
CONFIG_FILE() = /home/mu/share/projects/utech/ansible/ansible.cfg
DEFAULT_HOST_LIST(/home/mu/share/projects/utech/ansible/ansible.cfg) = ['/home/mu/share/pr>
DEFAULT_MODULE_PATH(/home/mu/share/projects/utech/ansible/ansible.cfg) = ['/home/mu/share/>
DEFAULT_ROLES_PATH(/home/mu/share/projects/utech/ansible/ansible.cfg) = ['/home/mu/share/p>
DEFAULT_STDOUT_CALLBACK(/home/mu/share/projects/utech/ansible/ansible.cfg) = yaml
DISPLAY_SKIPPED_HOSTS(/home/mu/share/projects/utech/ansible/ansible.cfg) = False
EDITOR(env: EDITOR) = vi
PERSISTENT_COMMAND_TIMEOUT(/home/mu/share/projects/utech/ansible/ansible.cfg) = 30

CALLBACK:
========

default:
_______
display_skipped_hosts(/home/mu/share/projects/utech/ansible/ansible.cfg) = False

OS / Environment

Ubuntu 22.04

Additional Information

Since the form insists on me adding information, this is what I think is related:

Ansible has extensive documentation, but it's incredibly hard and an overall rather frustrating experience to develop any kind of reusable components. This starts with the terminology (everything seems to be a plugin. Actions are plugins executed on the controller, modules are plugins executed on the host - confusing), continues with the fact that each artefact type has it's own and often completely difference interface with ansible (argument parsing/spec), there is a lot of redundancy, it's unclear which mechanism is best for some use case, and I have no idea what the implications of making this or that choice would be.

I often have use cases that seem to be rather natural and if I try to implement them I encounter road blocks all around me and when I search on google, I don't find anything, as if I'm the only person having this particular problem.

In this particular case, I have the choice to find out how to handle dependencies or to copy&paste the content of an external module (and its recursive dependencies if any) into a module_utils folder. I really don't want to do this (first because it's ugly, second because I don't want to take ownership over the code). Can I be the first user having this problem? I feel really stupid here.

Code of Conduct

[X] I agree to follow the Ansible Code of Conduct

ansibot commented 1 year ago

Files identified in the description:

lib/ansible

If these files are incorrect, please update the component name section of the description or use the component bot command.

bcoca commented 1 year ago

I can see how you missed it, it is only a short blurb about dependencies.

For plugins in general the dependencies only need to be installed on the controller, for modules, they need to be installed on the target, for many modules this still equates to the controller, specially network appliance/API ones.

The ansiballz only include the dependencies that use module_utils, going to any python/other language libraries is something that we did not implement as they can always include things like bindings into compiled code/libraries. For example, pylibacl depends on C library libacl.

One thing you can do is add a 'bootstraping' play to your collection that allows for installing dependencies for your modules and plugins (these can just target localhost), which can be in any language, not just Python. For the hosts: entry for modules there are several ways to adjust to the user context, use 'all' and/or a variable and have the user supply their inventory with either a limit or the variable containing the target groups/hosts.

mutech commented 1 year ago

Hi Brian, good to meet you again! Thanks for pointing me at that page, it's one that I actually didn't read yet. But even so, it does not clarify that dependencies are not handled by ansible and your answer raised some more questions on my part:

The first one is whether Ansible uses some virtual environment on the targets that shield it from whatever python installation is deployed there. I would not want to install software package indiscriminately on target systems just to be able to run some automation code. If that is not the case, I don't think modules should have any dependencies on python libraries, because that would just be too much of a burden, whether or not modules would document such dependencies and their implications.

One thing you can do is add a 'bootstraping' play to your collection that allows for installing dependencies for your modules

Provided that this play installing python dependencies on the target would not "pollute" the target's python installation, would such a bootstrap play be executed automatically? I have little hope, but if it would be run the first time some modules from that collection would run on a particular target, making the whole process transparent to the user of the module, this would be somewhere between great and good enough.

What I would expect or hope would be that there is a venv (or whatever python functionality would be best suited) in the target's remote-user .ansible directory where such dependencies would be installed.

I have a few rather quirky targets, such a QNAP NAS, where python installations are hard to find and keep changing with every new release, require admin permissions to update and have a lot of directories with 0777 perms and all kinds of scary stuff. I really wouldn't want touch any of that in manually executed bootstrap plays and much less ask users of my module to have to do it (this is not about this module I'm working on, but about the general principle of module dependencies on targets as a general approach).

I'm extending a bit on my original question. I'm working on adding to the community.dns collection to support OVH as DNS provider, later planning to add support for Freeipa and Bind9. I like their interface and the fact that they abstract over different API's which is what I want. I was wondering why they implemented the API calls as modules and not actions, but assumed there is a good reason for it. Considering that modules have this limitation (or what I consider to be a limitation) regarding dependencies, I doubt that this was a good choice, but I also don't want to deviate too much from their style in order to get my changes accepted. On the other hand, the OVH api does a lot of quirky stuff that I don't really want to maintain separately, considering that they offer a python wrapper for their API. I'm back to the question if I take the train and copy & paste or the machete and dig my way through the jungle.

It get's even worse for the Bind9 module, because dnspython already implements all the parsing and validation I would need but it's not something I would want to copy to module_utils and much less would want to maintain (and here I'm hoping that dnspython doesn't have a plethora of it's own dependencies, didn't check yet)

What would be the canonical clean approach an experienced Ansible module developer would take here? That's something I would love to see in a "best practices" document somewhere in the Ansible documentation.

It's good to find advice to encapsulate an import in try and present a readable error message. I didn't and got an ugly stacktrace instead, but that's something I can understand, though it's not pretty. I find it very difficult to choose the proper route to go and get a simple DNS api wrapper implemented, in this case even having the luxury to have a working prototype with community.dns and these guys seem to know what they're doing.

mutech commented 1 year ago

going to any python/other language libraries is something that we did not implement as they can always include things like bindings into compiled code/libraries.

I can see that being a good decision. But what would happen if an ambitions developer would put python code with bindings into the module_utils folder? I don't know if that's doable (lack of Python experiences), but I guess it would just fail when the Ansiballz is assembled or when it's run on the target. That's not very different from a failure to collect python modules that would not work for the same reason and just fail in a different stage of the process, or am I missing something here?

bcoca commented 1 year ago

First to address the 'ansible using a venv' issue, the python used is at the control of the inventory and playbook author, that CAN be a venv, you can even use Ansible to setup such venv. In most cases users just use the system python, in the case of some distributions, the 'user' python as they keep the 'system' python apart. I personally use PEP-370, but I'm a weird one.

Most modules that were accepted in core used to have very minimal dependencies for this reason, we even have our own url module_utils code to avoid using things like requests (specially since it has had such a complex history and every utility uses a diff version). Even now most collections use minimal dependencies, most that add one are restricted by the service/API they use (for example. boto for those that want to use aws).

The 'bootstrap' I suggested is far from being a convention, much less automated, but something you can/should add to the collection/role/module/plugin documentation (can even be part of the error message when reqs not found). We actually do 'boostrap' module dependencies in some cases, but these are very restricted and well known targets (install apt.py for the apt module), but it is not something we want or do in general as we don't want to pollute the target systems, this is really something of an exception we made for very 'base' functionality to work out of the box.

As for the community.dns choice, I don't know the specifics, but probably erred on the side of flexibility, modules can execute both on controller and target, actions are restricted to the controller. While most of their use might be using delegate_to: localhost, they kept the option open for cases in which the service is not directly accessible.

As for using a C-binded library in module_utils ... yes, you can also add a module that does rm -rf / on import ... that should never make it into production except in the most careless scenarios. That you "can" does not mean you "should" nor should we add guardrails for every crazystupid thing someone can do.

Another possible solution is mitogen library, it was designed to proxy python libs from the source machine to the target, there is even a mitogen patch for ansible, but it has issues keeping up with current versions. This is something we've always been looking at integrating more, but there are still issues, like differences in python versions between the machines, respawning diff interpreters, ... there are years of tickets/irc meetings/etc explaining these.

This discussion has become pretty detailed for an issue, which I'm only keeping open until we find a good way to improve the documentation on requirements. Any further discussion about features should go to either the proposals site or the projects forum.

mutech commented 1 year ago

Thanks again Brian! What I take home to my personal collection of best practices is this:

Do not introduce python dependencies in modules
- Pollution of target with these dependencies
- Overhead to provide dependencies such as manual mgmt. of venv to mitigate pollution and others
Do not use action plugins.
- For people facing connectivity issues with controllers - use modules instead
- Because of the interface (modules are easier to develop - use them)
If you need external python modules break one of the former rules
- if your functionality does not affect the target of the plugin, use an action plugin
- Pollution of the controller does not matter (as much)
- otherwise wait until somebody else commits the crime or copies the code.
- alternatively face unexpected issues with delegate_to
- wait for the day when code you copied to module_utils breaks because it's outdated
If you don't find or understand something in Ansible documentation, write an issue because Brian cares (or is on punishment detail). When you're told to post to proposals you used up your quota.

EDIT: That is to say if you don't see a need for the documentation to be updated, the ticket could be closed, should I?

samccann commented 1 year ago

@Andersson007 - can you evaluate this to see if https://github.com/ansible/ansible-documentation/issues/676#issuecomment-1771605945 matches your understanding and perhaps recommend where we can clarify this in the docs?

Andersson007 commented 1 year ago

@mutech hello, thanks for giving back to the project by opening such a great-formulated issue! Your feedback is especially appreciated as you have a fresh look at all the stuff.

@bcoca thanks for the detailed explanation - it should be reflected/elaborated in docs, and I agree this discussion perfectly fits the Forum.

@samccann thanks for pinging! Whether it matches my understanding or not, there are many other smarter folks whose understanding this might not match, so I think the larger community involvement is needed here:)

Let's start with defining things to clarify in the docs. So I can extract the following:

Actions vs Modules: when to develop what, pros & cons, etc.
Elaborate on the dependencies thing: more recommendations/clarifications mentioned throughout the discussion, venv thing, etc.

Does this ^ summary reflect issues?

The next steps can be:

Create separate discussion topics on the Forum, involve Core and Committee.
Create PRs based on the agreement.
I think the topics are broad and important, so maybe we should create a separate doc for each? Then we could embed them in proper places.

Thoughts?

samccann commented 11 months ago

@Andersson007 Thanks and sorry folks it took me so long to get back to this one. I like your approach. Can you open the two topics on the forum and add those links here for folks to follow along?

I'll keep this issue in 'triage' for now since we'd be waiting on that discussion before knowing the scope of the docs change here.

thanks everyone!

Andersson007 commented 7 months ago

folks, sorry, don't feel like i have time to create the topics on the forum. Whoever wants to proceed, feel free to do it

samccann commented 7 months ago

Thanks @Andersson007 - I opened the following topics on the forum. Folks- please review and comment there so we can get the docs in the right place for these two issues:

ansible / ansible-documentation