Virtualbox Dynamic Inventory Nested Group Assignments

lyrandy commented 2 weeks ago

Summary

My usage of Ansible takes advantage of Ansible's nested groups for variable management. I make heavy use of Vagrant + VirtualBox to speed up local development and testing of Ansible code.

The way the dynamic inventory assigns groups currently does not match the VirtualBox documentation. According to the documentation, you can assign a VM with multiple groups by delimiting it with a comma , character, and nest groups using the slash / character.

Ultimately, the objective is to be able to harmonize the way groups are set up with VirtualBox and with Ansible. It appears the concept of nested (or parent-child) groups match, so it should align.

See: https://www.virtualbox.org/manual/UserManual.html#gui-vmgroups and https://docs.ansible.com/ansible/latest/inventory_guide/intro_inventory.html#how-variables-are-merged

From VirtualBox Docs:

# Create multiple groups. For example:
VBoxManage modifyvm "vm01" --groups "/TestGroup,/TestGroup2"

#Create nested groups, having a group hierarchy. For example:
VBoxManage modifyvm "vm01" --groups "/TestGroup/TestGroup2"

The way the dynamic inventory parses the groups considers the / character as the delimiter for group assignment. As a consequence, we are unable to use nested groups in Ansible. If a user does not care about nested group, and assign multiple groups by simply using the / character (e.g. /TestGroup/TestGroup2), the VirtualBox UI will show a deeply nested list of groups before displaying the actual VM. See vm-1 in the screenshot for an example.

Screenshot_20240614_174813

The example I provide shows 2 issues resulting from how groups are parsed:

vm-1 is not set up with nested groups, but is set up to participate in 3 top-level groups. vm-2 groups have a trailing comma (except for the last one).

I did not spot mentions about how groups are automatically inferred in the official docs: https://docs.ansible.com/ansible/latest/collections/community/general/virtualbox_inventory.html

Issue Type

Bug Report

Component Name

virtualbox inventory

Ansible Version

ansible [core 2.17.0]
  config file = None
  configured module search path = ['/home/rly/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/rly/Documents/dev/sandbox/.venv/lib/python3.12/site-packages/ansible
  ansible collection location = /home/rly/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/rly/Documents/dev/sandbox/.venv/bin/ansible
  python version = 3.12.3 (main, Apr 23 2024, 09:16:07) [GCC 13.2.1 20240417] (/home/rly/Documents/dev/sandbox/.venv/bin/python)
  jinja version = 3.1.4
  libyaml = True

Community.general Version

# /home/rly/Documents/dev/sandbox/.venv/lib/python3.12/site-packages/ansible_collections
Collection        Version
----------------- -------
community.general 9.0.1

Configuration

CONFIG_FILE() = None

OS / Environment

My Host OS: 6.9.1-arch1-1 Vagrant: 2.4.1 VirtualBox: 7.0.18 r162988 VirtualBox VMs: bento/rockylinux-9 Vagrant images

Steps to Reproduce

I made a sample Vagrantfile to help illustrate. Notice that I'm purposefully trying to make the hierarchy of groups for vm-1 to be the reverse from lexicographical ordering.

Ensure your VirtualBox host has a host-only network named vboxnet0 with 192.168.56.1/24 set up. The Vagrantfile will also copy your SSH key into the VMs for convenience when using with Ansible.

Vagrant.configure("2") do |config|
  DEFAULT_VAGRANT_BOX = "bento/rockylinux-9"

  config.vm.define "vm-1" do |config|
    hostname = "vm-1"
    config.vm.box = DEFAULT_VAGRANT_BOX
    config.vm.hostname = hostname
    config.vm.network "private_network", ip: "192.168.56.10", name: "vboxnet0"

    config.vm.provider "virtualbox" do |vb|
      vb.name = hostname
      vb.customize ["modifyvm", :id, "--groups", "/zgroup1/ygroup2/xgroup3"]
    end

    config.vm.provision "shell", run: "always" do |s|
      ssh_pub_key = File.readlines("#{Dir.home}/.ssh/id_rsa.pub").first.strip
      s.inline = <<-SHELL
        echo "SSH Public key will be added to /home/vagrant/.ssh/authorized_keys"
        echo '#{ssh_pub_key}' >> /home/vagrant/.ssh/authorized_keys
      SHELL
    end
  end

  config.vm.define "vm-2" do |config|
    hostname = "vm-2"
    config.vm.box = DEFAULT_VAGRANT_BOX
    config.vm.hostname = hostname
    config.vm.network "private_network", ip: "192.168.56.11", name: "vboxnet0"
    config.vm.provider "virtualbox" do |vb|
      vb.name = hostname
      vb.customize ["modifyvm", :id, "--groups", "/othergroup1,/othergroup2,/othergroup3"]
    end

    config.vm.provision "shell", run: "always" do |s|
      ssh_pub_key = File.readlines("#{Dir.home}/.ssh/id_rsa.pub").first.strip
      s.inline = <<-SHELL
        echo "SSH Public key will be added to /home/vagrant/.ssh/authorized_keys"
        echo '#{ssh_pub_key}' >> /home/vagrant/.ssh/authorized_keys
      SHELL
    end
  end
end

And my simple inventory file

# virtualbox.yml
plugin: community.general.virtualbox

network_info_path: /VirtualBox/GuestInfo/Net/1/V4/IP 
running_only: true

compose:
  ansible_user: "'vagrant'"

And some sample group vars

grep -R my_custom_variable group_vars/ | sort

group_vars/all.yml:my_custom_variable: This is in "all" group
group_vars/xgroup3.yml:my_custom_variable: This is in "xgroup3" group
group_vars/ygroup2.yml:my_custom_variable: This is in "ygroup2" group
group_vars/zgroup1.yml:my_custom_variable: This is in "zgroup1" group

vagrant up
ansible -i virtualbox.yml -m debug -a "var=group" all
ansible -i virtualbox.yml -m debug -a "var=group_names" all
ansible -i virtualbox.yml -m debug -a "var=my_custom_variable" all

Expected Results

I expected to see vm-1 have its variable resolved properly, in consideration for nested groups when using /zgroup1/ygroup2/xgroup3, and for vm-2 to be part of separate groups when using /othergroup1,/othergroup2,/othergroup3

For vm-1, if we were to consider this in a static inventory, it might look like this:

all:
  vars:
    my_custom_variable: This is in "all" group
  children:
    zgroup1:
      vars:
        my_custom_variable: This is in "zgroup1" group
      children:
        ygroup2:
          vars:
            my_custom_variable: This is in "ygroup2" group
          children:
            xgroup3:
              vars:
                my_custom_variable: This is in "xgroup3" group
              hosts:
                vm-1:
                  ansible_host: 192.168.56.10
                  ansible_user: vagrant

and

ansible -i static.yml -m debug -a "var=my_custom_variable" all
vm-1 | SUCCESS => {
    "my_custom_variable": "This is in \"xgroup3\" group"
}

The same would be true with the dynamic inventory.

ansible -i virtualbox.yml -m debug -a "var=my_custom_variable" vm-1
vm-1 | SUCCESS => {
    "my_custom_variable": "This is in \"xgroup3\" group"
}

For vm-2, it should be part of the othergroup1, othergroup2, othergroup3 groups.

ansible -i virtualbox.yml -m debug -a "var=group_names" vm-2
vm-2 | SUCCESS => {
    "group_names": [
        "othergroup1",
        "othergroup2",
        "othergroup3"
    ]
}

Actual Results

ansible -i virtualbox.yml -m debug -a "var=group_names" all

vm-1 | SUCCESS => {
    "group_names": [
        "xgroup3",
        "ygroup2",
        "zgroup1"
    ]
}
vm-2 | SUCCESS => {
    "group_names": [
        "othergroup1,",
        "othergroup2,",
        "othergroup3"
    ]
}

In this case, vm-1 appears to be correctly assigned, but watch out!

ansible -i virtualbox.yml -m debug -a "var=my_custom_variable" all
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
vm-1 | SUCCESS => {
    "my_custom_variable": "This is in \"zgroup1\" group"
}
vm-2 | SUCCESS => {
    "my_custom_variable": "This is in \"all\" group"
}

The zgroup1 group vars wins because it's ordered last, lexicographically. This shows they're considered to be "on the same level".

Also, vm-2 groups have a trailing , character which is incorrect. This is due to the code simply splitting on the / character, so the , remains.

Code of Conduct

[X] I agree to follow the Ansible Code of Conduct

ansibullbot commented 2 weeks ago

Files identified in the description:

plugins/inventory/virtualbox.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

lyrandy commented 2 weeks ago

I believe this area can be improved to accommodate.

https://github.com/ansible-collections/community.general/blob/main/plugins/inventory/virtualbox.py#L180-L187

While doing some more digging, I also found https://github.com/ansible/ansible/issues/17100#issuecomment-504467928 which mentions that there aren't really a hierarchy of groups, since everything eventually renders down to hosts and vars. I think it's still helpful to conceptualize the group hierarchy to rationalize how variables will eventually "settle", so to speak.

felixfontein commented 2 weeks ago

As mentioned in the PR: this is not a bug, but a feature request.

lyrandy commented 2 weeks ago

Thank you for having a look at the PR and providing feedback. I'll address the CI and then make the recommended changes. I'm not too certain how to change the label from bug to feature, do I need to open a new issue?

For my own understanding, how are undocumented behaviours treated when it comes to making modifications to them? In my case, I did not notice any mention of how it integrates with VirtualBox groups, just the usual groups and keyed_groups keywords I've seen in basically all other dynamic inventory plugins. Is the approach to treat changes to the behaviour as a new feature until it is formally documented, after which it becomes more obvious to triage?

Also, breaking changes understandably must have sufficient communication and lead time for consumers to prepare for the change. How does it work to schedule the release of the breaking change? I only know that it will usually be included in major version updates, but I am curious about the other logistics before the release (like from when a PR is raised, how is it communicated, etc)

I am new to the scene so I appreciate your help and guidance regarding processes and workflows :)

ansible-collections / community.general