vagrant on ansible 8.2.0 with error: ModuleNotFoundError: No module named 'ansible.module_utils.common.yaml'

[patrickstump]

With ansible 8.2.0 is installed and trying to run any molecule command [destroy, create, etc] that performs actions with the vagrant plugin, the following error occurs:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'ansible.module_utils.common.yaml'

Current dependency list:

ansible==8.2.0
ansible-compat==4.1.5
certifi==2023.5.7
docker==6.1.3
molecule==5.1.0
molecule-plugins==23.4.1
python-vagrant==1.0.0
requests==2.31.0
urllib3==1.26.16

This appears to have been introduced when ansible went to version >= 8.0.0. However, due to molecule >=5.0.0 dependency on ansible-compat I have been unable to revert to a lower ansible version before 8.0.0.

[patrickstump]

As I was investigating this further, I do not think the issue is in the vagrant module.

The cleaned up output from the failed module was

{"changed": false,
 "module_stderr": "Traceback (most recent call last):
 File \"~/.ansible/tmp/ansible-tmp-1690145770.228065-12524-71285942539959/AnsiballZ_vagrant.py\",line 107, in <module>
 _ansiballz_main()
 File \"~/.ansible/tmp/ansible-tmp-1690145770.228065-12524-71285942539959/AnsiballZ_vagrant.py\", line 99, in _ansiballz_main
 invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"~/.ansible/tmp/ansible-tmp-1690145770.228065-12524-71285942539959/AnsiballZ_vagrant.py\", line 47, in invoke_module
 runpy.run_module(mod_name='ansible.modules.vagrant', init_globals=dict(_module_fqn='ansible.modules.vagrant', _modlib_path=modlib_path),
 File \"/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py\", line 224, in run_module
 return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py\", line 96, in _run_module_code
 _run_code(code, mod_globals, init_globals,\n  File \"/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py\", line 86, in _run_code
 exec(code, run_globals)\n  File \"/var/folders/60/5_ppk3rd7jxfbd859bjclxxr0000gn/T/ansible_vagrant_payload_ahf17d5p/ansible_vagrant_payload.zip/ansible/modules/vagrant.py\", line 33, in <module>
 File \"~/project/.venv/lib/python3.10/site-packages/molecule/util.py\", line 40, in <module>
 from molecule.app import app
 File \"~/project/.venv/lib/python3.10/site-packages/molecule/app.py\", line 13, in <module>
 app = App()
 File \"~/project/.venv/lib/python3.10/site-packages/molecule/app.py\", line 10, in __init__
 self.runtime = Runtime(isolated=True)
 File \"~/project/.venv/lib/python3.10/site-packages/ansible_compat/runtime.py\", line 213, in __init__
 from ansible.utils.display import Display
 File \"~/project/.venv/lib/python3.10/site-packages/ansible/utils/display.py\", line 49, in <module>
 from ansible import constants as C
 File \"~/project/.venv/lib/python3.10/site-packages/ansible/constants.py\", line 12, in <module>
 from ansible.config.manager import ConfigManager
 File \"~/project/.venv/lib/python3.10/site-packages/ansible/config/manager.py\",
 line 21, in <module>
 from ansible.module_utils.common.yaml import yaml_load
 ModuleNotFoundError: No module named 'ansible.module_utils.common.yaml'\n",
 "module_stdout": "",
 "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

Nothing the last lines:

File \"~/project/.venv/lib/python3.10/site-packages/ansible/config/manager.py\",
 line 21, in <module>
 from ansible.module_utils.common.yaml import yaml_load
 ModuleNotFoundError: No module named 'ansible.module_utils.common.yaml'\n",
 "module_stdout": "",

showing the issue appears to be in ~/project/.venv/lib/python3.10/site-packages/ansible/config/manager.py This was only happening for me when I tried the vagrant module. The docker was working fine, thus I thought it was in vagrant.

[konstruktoid]

Hi, can you test https://github.com/ansible-community/molecule-plugins/issues/174#issuecomment-1633938474?

[TheDevMinerTV]

I'm still having the same error when using the fix from the linked issue comment.

[konstruktoid]

@TheDevMinerTV which Ansible version are you using?

[TheDevMinerTV]

# pip3 list | grep -E '(vagrant|molecule|ansible)'
ansible                   8.1.0
ansible-compat            4.1.2
ansible-core              2.15.2
ansible-lint              6.17.2
molecule                  5.1.0
molecule-plugins          23.4.2.dev21
python-vagrant            1.0.0

[TheDevMinerTV]

Never mind, it solved it for me. I forgot that we had to change how Python packages are used. Thanks Debian 12...

The GitLab runner apparently doesn't use the users .bashrc file as well... Oh well

[nejch]

@TheDevMinerTV can you elaborate a little bit on that? This happens to me as of Ansible 8 as well, using poetry to manage the python dependencies and I imagine there's some odd behavior in how python modules are loaded here.

Manually (re)installing packages with pip into another environment seems like the wrong way to solve this when tracking python dependencies explicitly so just want to see what others' solution was.

[TheDevMinerTV]

I'm installing all of my python modules in a global venv via Ansible that everything uses. My GitLab runner just wasn't using the new PATH variable for the venv.