ansible-core includes ansible collections

[yajo]

Describe the bug

When installing both ansible and ansible-core packages from nixpkgs, both get access to standard collections.

Only ansible should have that.

Steps To Reproduce

nix-shell -p python3Packages.ansible-core --command 'ansible-galaxy collection list'

Current behaviour

``` # /nix/store/dpabz9ih3pr5235vp624gs6igmm5wpym-python3.10-ansible-6.2.0/lib/python3.10/site-packages/ansible_collections Collection Version ----------------------------- ------- amazon.aws 3.3.1 ansible.netcommon 3.0.1 ansible.posix 1.4.0 ansible.utils 2.6.1 ansible.windows 1.10.0 arista.eos 5.0.1 awx.awx 21.4.0 azure.azcollection 1.13.0 check_point.mgmt 2.3.0 chocolatey.chocolatey 1.3.0 cisco.aci 2.2.0 cisco.asa 3.1.0 cisco.dnac 6.5.2 cisco.intersight 1.0.19 cisco.ios 3.2.0 cisco.iosxr 3.2.0 cisco.ise 2.5.0 cisco.meraki 2.10.1 cisco.mso 2.0.0 cisco.nso 1.0.3 cisco.nxos 3.1.0 cisco.ucs 1.8.0 cloud.common 2.1.2 cloudscale_ch.cloud 2.2.2 community.aws 3.4.0 community.azure 1.1.0 community.ciscosmb 1.0.5 community.crypto 2.4.0 community.digitalocean 1.21.0 community.dns 2.3.0 community.docker 2.7.0 community.fortios 1.0.0 community.general 5.4.0 community.google 1.0.0 community.grafana 1.5.1 community.hashi_vault 3.1.0 community.hrobot 1.5.0 community.libvirt 1.1.0 community.mongodb 1.4.2 community.mysql 3.3.0 community.network 4.0.1 community.okd 2.2.0 community.postgresql 2.2.0 community.proxysql 1.4.0 community.rabbitmq 1.2.2 community.routeros 2.2.0 community.sap 1.0.0 community.sap_libs 1.2.0 community.skydive 1.0.0 community.sops 1.2.3 community.vmware 2.7.0 community.windows 1.10.0 community.zabbix 1.7.0 containers.podman 1.9.4 cyberark.conjur 1.1.0 cyberark.pas 1.0.14 dellemc.enterprise_sonic 1.1.1 dellemc.openmanage 5.5.0 dellemc.os10 1.1.1 dellemc.os6 1.0.7 dellemc.os9 1.0.4 f5networks.f5_modules 1.18.0 fortinet.fortimanager 2.1.5 fortinet.fortios 2.1.6 frr.frr 2.0.0 gluster.gluster 1.0.2 google.cloud 1.0.2 hetzner.hcloud 1.8.1 hpe.nimble 1.1.4 ibm.qradar 2.0.0 ibm.spectrum_virtualize 1.9.0 infinidat.infinibox 1.3.3 infoblox.nios_modules 1.3.0 inspur.sm 2.0.0 junipernetworks.junos 3.1.0 kubernetes.core 2.3.2 mellanox.onyx 1.0.0 netapp.aws 21.7.0 netapp.azure 21.10.0 netapp.cloudmanager 21.18.0 netapp.elementsw 21.7.0 netapp.ontap 21.21.0 netapp.storagegrid 21.10.0 netapp.um_info 21.8.0 netapp_eseries.santricity 1.3.0 netbox.netbox 3.7.1 ngine_io.cloudstack 2.2.4 ngine_io.exoscale 1.0.0 ngine_io.vultr 1.1.2 openstack.cloud 1.8.0 openvswitch.openvswitch 2.1.0 ovirt.ovirt 2.2.0 purestorage.flasharray 1.13.0 purestorage.flashblade 1.9.0 purestorage.fusion 1.0.2 sensu.sensu_go 1.13.1 servicenow.servicenow 1.0.6 splunk.es 2.0.0 t_systems_mms.icinga_director 1.30.1 theforeman.foreman 3.4.0 vmware.vmware_rest 2.2.0 vyos.vyos 3.0.1 wti.remote 1.0.4 ```

Expected behavior

The list should be empty because I'm using the ansible-core package.

Additional context

According to the docs:

Ansible community package: Includes language, runtime, and selected Collections

ansible-core: Includes language, runtime, and builtin plugins

Thus, ansible-core is not expected to provide any collections besides ansible.builtin.

Notify maintainers

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.18.16-200.fc36.x86_64, Fedora Linux, 36.20220813.0 (Silverblue), nobuild`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.10.3`
 - channels(yajo): `"nixpkgs"`
 - nixpkgs: `/var/home/yajo/.nix-defexpr/channels/nixpkgs`

@moduon MT-1075

[ralismark]

ansible/core.nix has this:

  propagatedBuildInputs = [
    # depend on ansible instead of the other way around
    ansible

which I'm guessing makes the full ansible available when running ansible-galaxy, bringing in all the collections as well. Getting rid of that dependency causes ansible-galaxy to output this:

$ ansible-galaxy collection list 
usage: ansible-galaxy [-h] [--version] [-v] TYPE ...

Perform various Role and Collection related operations.

positional arguments:
  TYPE
    collection   Manage an Ansible Galaxy collection.
    role         Manage an Ansible Galaxy role.

options:
  --version      show program's version number, config file location, configured module
                 search path, module location, executable location and exit
  -h, --help     show this help message and exit
  -v, --verbose  Causes Ansible to print more debug messages. Adding multiple -v will
                 increase the verbosity, the builtin plugins currently evaluate up to
                 -vvvvvv. A reasonable level to start is -vvv, connection debugging might
                 require -vvvv.
ERROR! - None of the provided paths were usable. Please specify a valid path with --collections-path

Though, from a shell with ansible, ansible-core's ansible-galaxy doesn't find any collections.

PR https://github.com/NixOS/nixpkgs/pull/166447 seems relevant -- it got rid of the ansible-collections package, I'm not really sure why it did that, but ansible-core depended on ansible-collections before that as well.

[mweinelt]

Hi, I did those changes back when I maintained ansible in nixpkgs. I can quickly summarize the issues I've seen and why they exist that way.

1. Shadowing of package names

In 2.9 ansible contained the core and collections in one. In 2.10 that was separated into ansible-base for the core, and ansible for the collections. In 2.11 the core was renamed into ansible-core.

We still supported both 2.9, 2.10 and 2.11 at the same time, but we cannot have multiple versions of the ansible package, especially with the change of the package contents from core+collections to just collections.

The naming of the legacy packages has been dropped in #166447.

2. Dependency hack

What this issue actually complains about is that I changed how ansible and ansible-core depend on each other. Because in general you should be able to just install ansible-core and have a lean installation.

Upstream by default creates a dependency from ansible, the collections package, to ansible-core, the runtime package. The problem with that kind of dependency is, that when you install python3Packages.ansible you will not see the executables of its transitive dependencies in your PATH. That means you will not have any ansible executables. This is what you noticed here:

Though, from a shell with ansible, ansible-core's ansible-galaxy doesn't find any collections.

The only way to install ansible executables from nixpkgs is to directly install the ansible_2_15, ansible_2_14 etc. packages, all of which are an override of python3Packages.ansible-core for the relevant version. But that would leave you without collections.

Now how would you install collections? By overriding the ansible-core package to propagate the relevant collection packages. And that is ultimately what I decided to do, by reversing that dependency.

3. Collections

The thing about collections is, that they're huge (~350M). And nobody ever needs all collections. Instead, everyone usually manages them in a requirements.yml, which allows installing just selected parts of the collections package through ansible-collection collection install -r requirements.yml.

---

collections:
  - name: ansible.netcommon
  - name: ansible.posix
  - name: ansible.utils  # ipaddr filter
  - name: community.crypto  # openssh_keypair
  - name: community.docker
  - name: community.general
  - name: community.grafana
  - name: community.sops
  - name: netbox.netbox  # inventory and lookups

We currently lack a selection mechanism for these individual collection packages.

Also note, that if you separate those two packages again, the collections would need to made available be in the same python environment, that ansible-core already uses.

4. Maintenance

I've dropped maintenainership for the ansible packages over a year ago in 2d0f97be21750797ff946c644202b12cd317397d, and it would be good if one or multiple people could step up to fill that role. I would be available to recommend solutions or do reviews, to get this started.

One thing that needs to be done is move ansible out of python3Packages and fully into the top-level. There were always bits of ansible in pkgs/tools/admin/ansible and that's where all of this should live.

A module for ansible doesn't make much sense, given that the required collections are mostly project-specific. So it would most likely be function arguments, or a wrapper.