search

ansible / awx

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.
Other
14k stars 3.42k forks source link

Support multiple Ansible versions, per playbook, user or group #34

Closed dagwieers closed 6 years ago

dagwieers commented 7 years ago
SUMMARY

With another major version on the horizon it's going to be essential to be able to select the Ansible version to work with, so you can migrate playbooks, users or groups on your own schedule, rather than a big bang upgrade.

Since Ansible AWX is already using virtualenvs, it shouldn't be that hard to hook into this to support different versions.

ISSUE TYPE
wenottingham commented 7 years ago

This is best done via isolated nodes (or groups of isolated nodes) with the needed Ansible setup on them. See docs/clustering.md.

dagwieers commented 7 years ago

@wenottingham Are you providing me a quick workaround with the existing code-base, or rather telling me: "We are not going to pursue this in a future release as you can already achieve something similar using isolated nodes" ?

PS I am not looking for a quick fix :-)

wenottingham commented 7 years ago

If I'm looking at how someone would need a 'custom ansible' for a set of playbooks or organization, this could range form ansible version, to custom modules, to custom libraries used by those modules (and/or customized ssh, etc.).

Hence, selecting the full execution environment makes more sense to me than picking an Ansible version. That's doable right now with isolated node customization, and in the handwavy future would likely be done similarly with a hypothetical execution container in containerized AWX down the road.

dagwieers commented 7 years ago

@wenottingham I'd be picking the virtualenv really, but thanks for clarifying.

wenottingham commented 7 years ago

Yep, no problem - not against providing the idea, just think it's better off as an isolated execution node/container rather than multiple ansible venvs on random cluster nodes.... especially if it starts expanding into 'supporting multiple versions of Azure/shade/boto/etc at once'

ghjm commented 7 years ago

In Tower, the ansible venv only holds additional Python packages that will be used by Ansible. Ansible itself comes in from distro packaging. If you create another virtualenv and install a different Ansible version into it, you probably can't use --system-site-packages, meaning you would lose distro-level dependencies that you maybe want (like the selinux python bindings).

If this isn't important to your use case, then I definitely see the benefit of being able to select a virtualenv. (I think it would need to be a job template setting, not a user or group setting, but that's a detail.) I also see the supportability concern @wenottingham brings up. If someone creates their virtualenv with --system-site-packages and has mostly one Ansible version but partly another, it will create weird problems that will be hard to debug.

@dagwieers - do you have any ideas for how we could ensure the virtualenvs don't step on each other, or otherwise provide "guard rails" so this feature doesn't become a tarpit for users?

dagwieers commented 7 years ago

@ghjm Some of the concerns you mention are identical to docker images or isolated nodes. People can f*ck up and that can lead to problems. Main thing is, on the command-line we use different Ansible versions without even the need for virtualenvs, containers or isolated nodes.

Secondly, if you only require a different Ansible version (which IMO is the largest use-case) the virtualenvs could be managed by AWX itself. If people need a different environment (e.g. SSH version or other stuff that you cannot do with virtualenvs) using isolated nodes seems not too much to ask. But on the command-line I can easily switch Ansible version by setting just 2 environment variables. Compare this to the effort of docker (managing containers) and isolated nodes seems overly complicated.

Thirdly, yes, being able to specify different version per user and group is a necessity if you have a large enterprise setup (with multiple departments using Tower) where you may have one person testing the newer Ansible release before moving his whole team to the newer release. You really do not want to micro-manage this per playbook if you do not have to. At various customers (on the CLI) we manage this by environment variables and symlinks per user/group/default. Works like a charm, introducing Tower/AWX makes this hard to impossible.

I don't mind supporting more complex cases than what I propose, but keeping it simple if complexity is not required is important as well. And this is very easy to implement and to use, even without AWX.

hogarthj commented 7 years ago

Just to add a specific use case onto this ...

We're in the process of a 1.7 (yes, really) to 2.3 migration at present.

I'd like to make the process to get to 2.4 much simpler for the team here.

One of the issues we have is an isolated and secure environment with a strict policy on importing packages/data/source/etc

The approach I've taken to mitigate this is custom packages that install isolated ansible versions into /opt/ansible170 or /opt/ansible232 etc via virtualenv so that I can security sign off on a single RPM and not a dozen tarballs... with a complicated virtualenv install process (yum install gcc libyaml-devel libopenssl-devel ; do stuff ; yum remove ... etc etc )

There's wrapper scripts in /usr/bin/ (ansible170, ansible230, etc) which automatically point to the correct environment, and alternatives is used to set a "system default" ...

We'd love to get Ansible Tower in place (and indeed are just in early stages of discussions with Red Hat) but we'd really like a simple (ie minimal additional infrastructure and containers are a complete no-go for us at this time) way to select an ansible version for a given job template.

The simplest solution I can imagine is to just allow an option in the job template for "path to ansible, ansible-playbook, etc" and then allow the system admin determine how that translates to something on the system.

In my case our older ones we are migrating from I'd set to /usr/bin/ansible170-playbook and the ones we've finished porting on /usr/bin/ansible232-playbook

So long as the virtualenv uses the system python version (and not an SCL or IUS or EPEL parallel version) the questions about system libraries versus virtualenv libraries for things like llibselinux-python shouldn't be an issue ...

Although I don't think that would be a problem if the scope was limited to "what is the full path to ansible and ansible-playbook", leaving how that translates to something to the local admin.

As for bugs raised ... I'd say that within the scope of AWX the response to issues should be "can you duplicate with the supported system ansible" (EPEL6 or Fedora ... or the one in Extras on RHEL7) when a custom path to ansible or ansible-playbook is in effect.

wwitzel3 commented 7 years ago

Notes from planning session

awx

clustering

How do we install / register the venv on a new node that is added to the cluster? Re-run the role with the inventory and the management command will properly register the venv for the instance?

packaging

docs

show clear examples of using the roles to create the venv show clear examples of installing your deps in to the venv show examples of using the role to ensure the venv is present across the cluster

wenottingham commented 7 years ago

My inclination is that the clustering/packaging bits are an exercise for the admin ... I do not want to build a framework for this.

Long term, these are just container images that aren't managed as part of AWX registration.

ryanpetrello commented 7 years ago

@wenottingham we're thinking of providing a minimal role that can be used to bootstrap these new venvs - just so they have somewhere to start. You would run this role across your awx inventory, and it would:

  1. Copy /var/lib/awx/venvs/ansible to /var/lib/awx/venvs/your-custom-env
  2. Call awx-manage register-custom-venv instanceX /var/lib/awx/venvs/your-custom-env

At this point, you could continue running your own playbook tasks that, for example, installed Python dependencies into /var/lib/awx/venvs/your-custom-env (such as a custom Ansible version).

awx itself would display these registered venvs as selectable options for Organizations, Projects, and Job Templates, and would activate them for playbook runs.

For users who don't care about this functionality and never register additional venvs, we could just hide the venv selection and use the default ansible venv that comes w/ awx.

wenottingham commented 7 years ago

@ryanpetrello as a sample thing that lives in galaxy or in a library directory somewhere, sure.

I think building infrastructure to attempt to run/re-run manage this on node provisioning/deprovisioning (which is tangentially mentioned above) is way off in the weeds.

ryanpetrello commented 7 years ago

@wwitzel3 covered this really well; here are some explicit API changes we'll have to make to support this:

  1. /api/v2/ping/ would be enhanced to display registered venvs per instance, e.g.,
{
    ...
    "instances": [
        {
            "node": "awx",
            "heartbeat": "2017-10-18T16:11:16.836Z",
            "version": "1.0.1",
            "capacity": 425,
            "virtualenvs": ['your-custom-env'],
        },
    ],
    "ha": true,
    "version": "1.0.1-33-g3685cb5",
    ...
}

  1. /api/v2/instances/ would need to change in a similar manner to reflect custom virtualenv registration.

  2. A new virtualenv field would be added to JobTemplate, Project, and Organization to allow specifying a custom venv. Each of these would probably need their OPTIONS dynamically populated to include all possible registered choices.

  3. Job execution code would need to be updated to: a. Confirm the actual existance of the specified virtualenv at run time with meaningful error reporting. b. Activate the proper virtualenv when a custom one is specified. In lieu of a custom virtualenv, the default one we ship (/var/lib/awx/venvs/ansible) would be used.

ryanpetrello commented 7 years ago

@wenottingham noted, and I think we're in agreement. There will be a balance to strike here in terms of giving users the examples they need; I think something like a galaxy role and thorough documentation would give most power users (the ones likely to use this feature) what they need.

Even if we do none of the bootstrap hand-holding and only just add documented support for custom venv registration and selection, it sounds like it would solve the problem for many people asking for this.

wenottingham commented 7 years ago

The above implies venvs are per instance (in /api/vN/ping); assigning them implies they are global

matburt commented 7 years ago

Whatever is applied to ping needs to be available on /instances/ also, and primarily to be honest.

trahman73 commented 7 years ago

For the UI side:

Orgs image

Projects image

Templates image

ryanpetrello commented 7 years ago

@wenottingham

The above implies venvs are per instance (in /api/vN/ping); assigning them implies they are global

This is probably the biggest remaining question I have about this.

If we register venvs at the instance level, how do we know the correct options to provide for JT/Project/Organization? Do we just provide all possible options for all instances and assume the user doesn't configure something wrong (by choosing venv X that isn't installed on instance Y)?

It seems like that's how we handle manual projects - you register projects and there's an expectation of where they live. If the files aren't actually there at launch time, we display a meaningful error message.

wenottingham commented 7 years ago

It seems like that's how we handle manual projects - you register projects, manage the local file system yourself, and there's an expectation of where they live. If the files aren't actually there at run time, we display a meaningful error message.

That would be the model I would suggest.

ryanpetrello commented 6 years ago

Functionality is complete and merged on the API side:

https://github.com/ansible/awx/pull/940/files#diff-015b935452ac22e8799e290bcfa7cb07