Open lod opened 1 year ago
I don't know if my issue is similar, but since Molecule 6.0.0, Molecule has been installing roles and collections into the roles and collections top level dirs in my roles and playbooks. It used to install it into ~/.cache/molecule/....
But then the other issue is that when Molecule actually runs the prepare stage, Ansible can't find my role because Ansible is running from the scenario's directory. I also find that adding ansible.cfg in the root of my project doesn't seem to help. This only seems to work when I set ANSIBLE_ROLES_PATH: ../../roles
under my scenario's provisioner parameter.
provisioner:
env:
ANSIBLE_ROLES_PATH: ../../roles
But this also forces me to:
So what's the proper solution here and am I doing this wrong?
Thanks for reporting this issue.
I'll take a look.
Guessing this is also related to https://github.com/ansible/molecule/issues/4017
Same issue here, can reproduce locally if I update everything to the latest releases on my Mac:
raise InvalidPrerequisiteError(msg)
ansible_compat.errors.InvalidPrerequisiteError: Collection 'community.docker' not found in '['/Users/jgeerling/.ansible/collections', '/usr/share/ansible/collections']'
I don't know if my issue is similar, but since Molecule 6.0.0, Molecule has been installing roles and collections into the roles and collections top level dirs in my roles and playbooks. It used to install it into ~/.cache/molecule/....
But then the other issue is that when Molecule actually runs the prepare stage, Ansible can't find my role because Ansible is running from the scenario's directory.
I experience exactly the same issue with Molecule 6.0.1. In the prepare stage, the roles cannot be found because they are installed in the top-level roles directory.
Note that Updating to latest ansible-compat 4.1.7 did not fix this issue
@lod and @geerlingguy
Could you help verify whether the issue is fixed in ansible-compat 4.1.7?
@isuftin and @hulr
I believe the issue you describe is different. We are still working on it.
What I noticed today (with ansible-compat 4.1.7) is that when the folder <role-name>/roles
exists, Molecule sets ANSIBLE_ROLES_PATH
:
$ molecule create
INFO default scenario test matrix: dependency, create, prepare
INFO Performing prerun with role_name_check=2...
INFO Set ANSIBLE_ROLES_PATH=roles:/home/user/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
INFO Using /home/user/.ansible/roles/<role name> symlink to current repository in order to enable Ansible to find the role using its expected full name.
I have to override it in the molecule.yml file, that roles installed from <role-name>/molecule/default/requirements.yml
will be found:
provisioner:
env:
ANSIBLE_ROLES_PATH: ../../roles
If the top-level <role-name>/roles
folder doesn't exist, Molecule doesn't set ANSIBLE_ROLES_PATH
and it works fine without overriding it in the molecule.yml file:
$ molecule create
INFO default scenario test matrix: dependency, create, prepare
INFO Performing prerun with role_name_check=2...
INFO Using /home/user/.ansible/roles/<role name> symlink to current repository in order to enable Ansible to find the role using its expected full name.
I seem to have the same issue as @isuftin:
INFO apple scenario test matrix: dependency, create, prepare, converge
INFO Performing prerun with role_name_check=0...
INFO Using /Users/gianni-cm/.config/ansible/roles/citymesh.openvpn symlink to current repository in order to enable Ansible to find the role using its expected full name.
INFO Running apple > dependency
WARNING Skipping, missing the requirements file.
WARNING Skipping, missing the requirements file.
INFO Running apple > create
INFO Running apple > prepare
WARNING Skipping, prepare playbook not configured.
INFO Running apple > converge
ERROR! the role 'citymesh.openvpn' was not found in /Users/gianni-cm/git/Ansible/openvpn/molecule/apple/roles:/Users/gianni-cm/.cache/molecule/openvpn/apple/roles:/Users/gianni-cm/git/Ansible:/Users/gianni-cm/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/Users/gianni-cm/git/Ansible/openvpn/molecule/apple
The error appears to be in '/Users/gianni-cm/git/Ansible/openvpn/molecule/apple/converge.yml': line 5, column 7, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
roles:
- citymesh.openvpn
^ here
This was working before v6.
The weird thing is the symlink exists under ~/.config/ansible/roles
but not under ~/.ansible/roles
Could you help verify whether the issue is fixed in ansible-compat 4.1.7?
Upgrading from ansible-compat 3.0.2
to 4.1.8
seems to have fixed the original issue around collections for me in github's action CI (I couldn't replicate locally).
I'm using molecule 6.0.1 and can provide links to the action if anyone needs.
Thanks @zhan9san working with ansible_compat-4.1.9
I think things are working as well on my end. Just updated to ansible-compat 4.1.0..
ansible==8.3.0
ansible-compat==4.1.10
ansible-core==2.15.3
molecule==6.0.2
molecule-plugins==23.5.0
Spoke too soon. It seems we're getting tripped up in the prepare stage.
We're running this on a GitLab runner...
Here's the interesting bits..
Running: molecule --verbose test --scenario-name aws-marketplace-rhel9-ec2
INFO Found config file /builds/ctek/automation/ansible/roles/docker/.config/molecule/config.yml
DEBUG Validating schema /builds/ctek/automation/ansible/roles/docker/molecule/aws-marketplace-rhel9-ec2/molecule.yml.
WARNING Driver ec2 does not provide a schema.
INFO aws-marketplace-rhel9-ec2 scenario test matrix: dependency, cleanup, destroy, syntax, create, prepare, converge, idempotence, side_effect, verify, cleanup, destroy
INFO Performing prerun with role_name_check=0...
INFO Running ansible-galaxy role install -vr roles/requirements.yml
INFO Running ansible-galaxy collection install -v -r collections/requirements.yml
INFO Set ANSIBLE_ROLES_PATH=roles:/root/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
INFO Using /root/.ansible/roles/wma.docker symlink to current repository in order to enable Ansible to find the role using its expected full name.
DEBUG Inventory /builds/ctek/automation/ansible/roles/docker/molecule/aws-marketplace-rhel9-ec2/../common/group_vars linked to /root/.cache/molecule/docker/aws-marketplace-rhel9-ec2/inventory/group_vars
DEBUG Inventory /builds/ctek/automation/ansible/roles/docker/molecule/aws-marketplace-rhel9-ec2/../common/ec2/host_vars linked to /root/.cache/molecule/docker/aws-marketplace-rhel9-ec2/inventory/host_vars
INFO Running aws-marketplace-rhel9-ec2 > dependency
Molecule aws-marketplace-rhel9-ec2 > dependency
00:03
Starting galaxy role install process
- extracting wma.ssl_filtering to /builds/ctek/automation/ansible/roles/docker/roles/wma.ssl_filtering
- wma.ssl_filtering (1.7.0) was installed successfully
- adding dependency: geerlingguy.java (2.3.1)
- downloading role 'java', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-java/archive/2.3.1.tar.gz
- extracting geerlingguy.java to /builds/ctek/automation/ansible/roles/docker/roles/geerlingguy.java
- geerlingguy.java (2.3.1) was installed successfully
INFO Dependency completed successfully.
Skipping install, no requirements found
INFO Dependency completed successfully.
...
INFO Running aws-marketplace-rhel9-ec2 > prepare
Molecule aws-marketplace-rhel9-ec2 > prepare
00:07
Using /root/.cache/molecule/docker/aws-marketplace-rhel9-ec2/ansible.cfg as config file
PLAY [Prepare RHEL 9 system] ***************************************************
TASK [Gathering Facts ] ********************************************************
Wednesday 06 September 2023 17:40:02 +0000 (0:00:00.013) 0:00:00.013 ***
ok: [aws-marketplace-rhel9-ec2--8]
TASK [Add root certificates to cert store name=wma.ssl_filtering] **************
Wednesday 06 September 2023 17:40:08 +0000 (0:00:05.134) 0:00:05.147 ***
ERROR! the role 'wma.ssl_filtering' was not found in /builds/ctek/automation/ansible/roles/docker/molecule/aws-marketplace-rhel9-ec2/roles:/root/.cache/molecule/docker/aws-marketplace-rhel9-ec2/roles:/builds/ctek/automation/ansible/roles:/root/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/builds/ctek/automation/ansible/roles/docker/molecule/aws-marketplace-rhel9-ec2
The error appears to be in '/builds/ctek/automation/ansible/roles/docker/molecule/aws-marketplace-rhel9-ec2/prepare.yml': line 15, column 15, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
include_role:
name: wma.ssl_filtering
^ here
So while it installed my wma.ssl_filtering role to /builds/ctek/automation/ansible/roles/docker/roles
(which also is not great because I don't want third party deps in my project), /builds/ctek/automation/ansible/roles/docker/roles
does not seem to be on the path that the prepare step looks at.
Also note my config.yml for molecule includes:
dependency:
name: galaxy
options:
role-file: molecule/common/role_requirements.yml
requirements-file: molecule/common/collections_requirements.yml
force: false
The role that can't be found is specified @ molecule/common/role_requirements.yml but NOT in meta/requirements.yml - Thats because it's a test-time only role. And it gets installed into my role's roles/ directory as a sibling to requirements.yml. However geerlingguy.java does not get installed there. That role is specified in meta/requirements.yml. And for scenarios without a prepare step, my role works fine because geerlingguy.java is found no problem.
Finally, the only other difference I can think of is that we install geerlingguy.java from ansible galaxy but we pull wma.ssl_filtering from our private gitlab instance.
Has there been any movement on the roles portion of this? This still seems to be an issue in the latest Molecule install. Still working as expected in molecule<6.0.0
Issue Type
I'm not entirely sure that this is a molecule bug, it may be a ansible-compat bug, I'm not familiar enough with the ecosystem to be certain.
However the failures started with molecule v6.0, so it should be documented here until it is fixed even if the fix is elsewhere.
Molecule and Ansible details
Test environment is an official Debian bookworm-slim container, the following steps were run to create the environment
Actual Behaviour
On running
molecule test
I get an exception, the tail of which isDetails
The collection files are all installed in /usr/local/lib/python3.11/dist-packages/ansible_collections
Ansible galaxy uses the following search paths in execute_list_collection, cli/galaxy.py:1619
The third element, AnsibleCollectionConfig.collection_paths, is set to
['/root/.ansible/collections', '/usr/local/lib/python3.11/dist-packages']
It isn't clear to me how collection_paths gets the dist-packages element, but it is there. https://docs.ansible.com/ansible/latest/reference_appendices/config.html#collections-paths says that the default doesn't include that path. There is no /etc/ansible/ or ~/.ansible.cfg
molecule uses ansible-compat, ansible-compat follows the documentation not the actual ansible behaviour and only searches
['/root/.ansible/collections', '/usr/share/ansible/collections']
. Which is why it can't find any of my collections.ansible-galaxy collection list
shows all the collectionsansible-galaxy collection install containers.podman community.docker ansible.posix
fails with " Nothing to do. All requested collections are already installed. If you want to reinstall them, consider using--force
."ansible -c local -i 127.0.0.1, -m containers.podman.podman_image_info all
finds and executes the module (which fails, but it finds and tries it)Workaround
ansible-galaxy collection install --collections-path /usr/share/ansible/collections containers.podman community.docker ansible.posix --force
This installs the specified collections in /usr/share/ansible/collections allowing molecule to run.
(I also tried symlinking but ansible-compat doesn't seem to follow the link)