'retrying' is present in the Mitogen importer blacklist

[kevinkirkup]

The full traceback is:
Traceback (most recent call last):
  File "master:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/runner.py", line 961, in _run
    self._run_code(code, mod)
  File "master:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/runner.py", line 938, in _run_code
    exec(code, vars(mod))
  File "master:/ansible/roles/kafka_topics/library/kafka_topic.py", line 66, in <module>
  File "<stdin>", line 1475, in load_module
  File "<stdin>", line 1404, in _refuse_imports
ModuleNotFoundError: 'retrying' is present in the Mitogen importer blacklist, therefore this context will not attempt to request it from the master, as the request will always be refused.

Feel free to write an issue in your preferred format, however if in doubt, use the following checklist as a guide for what to include.

• Which version of Ansible are you running?

  ansible-user@fd77116cca2b:/ansible$ ansible --version
  ansible 2.8.4
  config file = /ansible/ansible.cfg
  configured module search path = ['/ansible/modules']
  ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]

• Is your version of Ansible patched in any way? No

• Are you running with any custom modules, or module_utils loaded? No

• Have you tried the latest master version from Git? No, but have tried the latest release version 0.2.9.

• Do you have some idea of what the underlying problem may be? https://mitogen.networkgenomics.com/ansible_detailed.html#common-problems has instructions to help figure out the likely cause and how to gather relevant logs. From looking at other open issues, I thought it was initially related to using the wrong python, but retrying is installed with Python 3.6 which is being used by Mitogen.

  ansible-user@fd77116cca2b:/ansible$ pip3 freeze | grep retrying
  retrying==1.3.3

• Mention your host and target OS and versions Docker:

  FROM ubuntu:bionic

• Mention your host and target Python versions Hosts are also Ubuntu, Python3.6.

• If reporting any kind of problem with Ansible, please include the Ansible version along with output of "ansible-config dump --only-changed".

  ansible-user@a56e38bf64e3:/ansible$ ansible-config dump --only-changed
  ANSIBLE_PIPELINING(/ansible/ansible.cfg) = True
  ANSIBLE_SSH_ARGS(/ansible/ansible.cfg) = -C -o ControlMaster=auto -o ControlPersist=300s
  ANSIBLE_SSH_RETRIES(/ansible/ansible.cfg) = 4
  DEFAULT_CALLBACK_WHITELIST(/ansible/ansible.cfg) = ['profile_roles', 'profile_tasks']
  DEFAULT_FILTER_PLUGIN_PATH(/ansible/ansible.cfg) = ['/ansible/plugins/filter']
  DEFAULT_FORKS(/ansible/ansible.cfg) = 20
  DEFAULT_MODULE_PATH(/ansible/ansible.cfg) = ['/ansible/modules']
  DEFAULT_ROLES_PATH(/ansible/ansible.cfg) = ['/ansible/roles', '/home/ansible-user/ansible/roles']
  DEFAULT_STRATEGY(/ansible/ansible.cfg) = mitogen_linear
  DEFAULT_STRATEGY_PLUGIN_PATH(/ansible/ansible.cfg) = ['/usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/strategy']
  DEFAULT_TIMEOUT(/ansible/ansible.cfg) = 30
  DISPLAY_SKIPPED_HOSTS(/ansible/ansible.cfg) = False
  HOST_KEY_CHECKING(/ansible/ansible.cfg) = False

[kevinkirkup]

I've tried a number of different thing to figure this out without any luck. Any help would be appreciated.

[kevinkirkup]

Set: strategy: linear for this role and seeing a similar error from Ansible.

TASK [kafka_topics : Create Topics] ****************************************************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'retrying'
failed: [kato-virtual-pod-26-openstack1] (item={'key': 'containers_bgp_status', 'value': {'state': 'present', 'name': 'containers_bgp_status', 'partitions': 48, 'replication_factor': 1, 'config': {'retention.ms': 604800000}}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "containers_bgp_status", "value": {"config": {"retention.ms": 604800000}, "name": "containers_bgp_status", "partitions": 48, "replication_factor": 1, "state": "present"}}, "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 114, in <module>\n  File \"<stdin>\", line 106, in _ansiballz_main\n  File \"<stdin>\", line 49, in invoke_module\n  File \"/usr/lib/python3.6/imp.py\", line 235, in load_module\n    return load_source(name, filename, file)\n  File \"/usr/lib/python3.6/imp.py\", line 170, in load_source\n    module = _exec(spec, sys.modules[name])\n  File \"<frozen importlib._bootstrap>\", line 618, in _exec\n  File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module\n  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n  File \"/tmp/ansible_kafka_topic_payload_5xu4r5v6/__main__.py\", line 66, in <module>\nModuleNotFoundError: No module named 'retrying'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'retrying'

[kevinkirkup]

Import is working:

>>> import retrying
>>> os.path.abspath(retrying.__file__)
'/usr/local/lib/python3.6/dist-packages/retrying.py'

But isn't in sys.module:

def load_source(name, pathname, file=None):
    loader = _LoadSourceCompatibility(name, pathname, file)
    spec = util.spec_from_file_location(name, pathname, loader=loader)
    if name in sys.modules:
        module = _exec(spec, sys.modules[name])                                          <------------ Failing here

[kevinkirkup]

I was able to resolve this by manually installing the package on the target host. I'm was thinking that should be covered by mitogen or Ansible, but maybe not.

[s1113950]

You're correct, Ansible requires external python packages to be installed in order for it to use them. Glad you got your issue worked out though!

[kevinkirkup]

I think what confused me was the comment in #724. It made it sound like Mitogen would install it on the remote host.

[s1113950]

I just checked the comment you mentioned; you're correct in that Mitogen should fetch it from the host node. Ooh but if it's a module that's not in sys.modules then the issue might be that Mitogen's import helper code couldn't find it (there's a few different ways Mitogen tries to find modules and sys.modules is one of them). I ran into a similar error with trying to get collections support to work for Mitogen: Ansible collections do a bit of a hack on the import system so that you can import X but stuff doesn't exist in the usual places Python looks when you import something. I haven't used https://pypi.org/project/retrying/ before myself so I can't say for sure.

Would you be able to provide a sample playbook so I can reproduce the error inside of the bionic container you tried?

[kevinkirkup]

Let me see what I can do. It could be a few days b/c of the holiday.

[s1113950]

No rush! 😄