search

mitogen-hq / mitogen

Distributed self-replicating programs in Python
https://mitogen.networkgenomics.com/
BSD 3-Clause "New" or "Revised" License
2.34k stars 199 forks source link

network_cli connections failing with "could not recover task_vars" #766

Open craigers521 opened 3 years ago

craigers521 commented 3 years ago

Running ansible version 2.9.11 on macOS with python 3.8, with latest mitogen version from master, targeting cisco isr routers and other network_cli based connection devices. All playbooks fail with this error:

<10.119.122.254> attempting to start connection
<10.119.122.254> using connection plugin network_cli
<10.119.130.254> attempting to start connection
<10.119.130.254> using connection plugin network_cli
<10.119.122.254> local domain socket does not exist, starting it
<10.119.122.254> control socket path is /Users/cse015/.ansible/pc/92ada5d152
<10.119.122.254> local domain socket listeners started successfully
<10.119.122.254> loaded cliconf plugin ios from path /Users/cse015/python-virtual-environments/ansible29/lib/python3.8/site-packages/ansible/plugins/cliconf/ios.py for network_os ios
<10.119.122.254>
<10.119.122.254> local domain socket path is /Users/cse015/.ansible/pc/92ada5d152
<10.119.130.254> local domain socket does not exist, starting it
<10.119.130.254> control socket path is /Users/cse015/.ansible/pc/77c5115e9c
<10.119.130.254> local domain socket listeners started successfully
<10.119.130.254> loaded cliconf plugin ios from path /Users/cse015/python-virtual-environments/ansible29/lib/python3.8/site-packages/ansible/plugins/cliconf/ios.py for network_os ios
<10.119.130.254>
<10.119.130.254> local domain socket path is /Users/cse015/.ansible/pc/77c5115e9c
fatal: [10.119.122.254]: UNREACHABLE! => {
    "changed": false,
    "msg": "could not recover task_vars. This means some connection settings may erroneously be reset to their defaults. Please report a bug if you encounter this message.",
    "unreachable": true
}
fatal: [10.119.130.254]: UNREACHABLE! => {
    "changed": false,
    "msg": "could not recover task_vars. This means some connection settings may erroneously be reset to their defaults. Please report a bug if you encounter this message.",
    "unreachable": true

I also tried running with ansible 2.10 and the latest collectionSupport branch from the s1113950 fork and i was met with a different error: 

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 158, in run
    res = self._execute()
  File "/usr/local/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 613, in _execute
    self._connection = self._get_connection(cvars, templar)
  File "/usr/local/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 908, in _get_connection
    connection, plugin_load_context = self._shared_loader_obj.connection_loader.get_with_context(
  File "/Users/cse015/.ansible/plugins/mitogen-dev/ansible_mitogen/strategy.py", line 170, in wrap_connection_loader__get
    return ansible_mitogen.loaders.connection_loader__get(name, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/loader.py", line 826, in get_with_context
    obj.__init__(instance, *args, **kwargs)
  File "/Users/cse015/.ansible/collections/ansible_collections/ansible/netcommon/plugins/connection/network_cli.py", line 324, in __init__
    super(Connection, self).__init__(
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/connection/__init__.py", line 290, in __init__
    self._update_connection_state()
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/connection/__init__.py", line 368, in _update_connection_state
    control_path = ssh._create_control_path(
  File "/Users/cse015/.ansible/plugins/mitogen-dev/ansible_mitogen/plugins/connection/mitogen_ssh.py", line 67, in _create_control_path
    return Connection.vanilla_class._create_control_path(*args, **kwargs)
AttributeError: 'get_with_context_result' object has no attribute '_create_control_path'
fatal: [10.119.130.254]: FAILED! => {
    "msg": "Unexpected failure during module execution.",
    "stdout": ""
s1113950 commented 3 years ago

@craigers521 thanks for testing this on the #715 branch. Is there a simple way to reproduce this using another network_cli-based task? I don't have a cisco router available to test with.

isundaylee commented 3 years ago

I ran into this same issue AttributeError: 'get_with_context_result' object has no attribute '_create_control_path' for a vyos_config task. Ansible 2.10.6 with Mitogen 0.3.0rc1.

Not sure if this qualifies as a "simple way to reproduce": but Vyos can be installed on a virtual machine, which can be used to test network_cli tasks against.

s1113950 commented 3 years ago

@isundaylee do you have a vagrant file or something of the like to create a vyos installation on a vm? I don't have the time to look into reproducing with Vyos at the moment unfortunately.

lorephoenix commented 3 years ago

Hello I am having a similar issue running Ansible 2.10 using Mitogen 0.30rc1 ansible-galaxy collection list Collection Version

ansible.netcommon 1.4.1 ansible.posix 1.1.1 cisco.ios 1.2.1

The issue occur when gathering facts

Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 158, in run res = self._execute() File "/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 613, in _execute self._connection = self._get_connection(cvars, templar) File "/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 913, in _get_connection ansible_playbook_pid=to_text(os.getppid()) File "/home/deployer/ansible/plugins/mitogen-0.3.0rc1/ansible_mitogen/strategy.py", line 168, in wrap_connection_loaderget return ansible_mitogen.loaders.connection_loaderget(name, *args, kwargs) File "/usr/lib/python3.6/site-packages/ansible/plugins/loader.py", line 826, in get_with_context obj.init(instance, *args, *kwargs) File "/home/deployer/.ansible/collections/ansible_collections/ansible/netcommon/plugins/connection/network_cli.py", line 332, in init play_context, new_stdin, args, kwargs File "/usr/lib/python3.6/site-packages/ansible/plugins/connection/init.py", line 290, in init self._update_connection_state() File "/usr/lib/python3.6/site-packages/ansible/plugins/connection/init.py", line 371, in _update_connection_state self._ansible_playbook_pid File "/home/deployer/ansible/plugins/mitogen-0.3.0rc1/ansible_mitogen/plugins/connection/mitogen_ssh.py", line 67, in _create_control_path return Connection.vanilla_class._create_control_path(*args, **kwargs) AttributeError: 'get_with_context_result' object has no attribute '_create_control_path' fatal: [hostname]: FAILED! => { "msg": "Unexpected failure during module execution.", "stdout": "" }

MrLucifer11 commented 3 years ago

Hi,

I have the same issue with ansible 2.10.3 and Mitogen 0.30rc1. Has anyone found a solution please ?

XuNiLuS commented 3 years ago

Hello,

I have the same issue with Ansible 3.4.0 (Based 2.10.13) and Mitogen 0.30rc1. is there a solution?

arhue commented 2 years ago

Same issue with Ansible 2.13.1 and Mikrotik Router OS 7.5.

markafarrell commented 1 year ago

I have been able to reproduce this issue with the following https://github.com/markafarrell/mitogen-issue-766-repro

RomyxBaps commented 1 year ago

I have you find a solution ? I have the problem for a rsync tasks...

rawat-he commented 7 months ago

Hi,

I am facing the same issue. Below is the traceback

The full traceback is:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 158, in run
    res = self._execute()
  File "/usr/local/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 549, in _execute
    self._connection = self._get_connection(cvars, templar, current_connection)
  File "/usr/local/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 945, in _get_connection
    connection, plugin_load_context = self._shared_loader_obj.connection_loader.get_with_context(
  File "/usr/local/lib/python3.10/site-packages/ansible_mitogen/strategy.py", line 125, in wrap_connection_loader__get
    return ansible_mitogen.loaders.connection_loader__get(name, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/ansible/plugins/loader.py", line 904, in get_with_context
    obj.__init__(instance, *args, **kwargs)  # pylint: disable=unnecessary-dunder-call
  File "/data/collections/ansible_collections/ansible/netcommon/plugins/connection/netconf.py", line 232, in __init__
    super(Connection, self).__init__(
  File "/usr/local/lib/python3.10/site-packages/ansible/plugins/connection/__init__.py", line 289, in __init__
    self._update_connection_state()
  File "/usr/local/lib/python3.10/site-packages/ansible/plugins/connection/__init__.py", line 367, in _update_connection_state
    control_path = ssh._create_control_path(
  File "/usr/local/lib/python3.10/site-packages/ansible_mitogen/plugins/connection/mitogen_ssh.py", line 87, in _create_control_path
    return Connection.vanilla_class._create_control_path(*args, **kwargs)
AttributeError: 'get_with_context_result' object has no attribute '_create_control_path'

msg: 'Unexpected failure during module execution: ''get_with_context_result'' object has no attribute ''_create_control_path''

Ansible: 2.14.0 Target OS: Alpine Linux v3.17 Python: 3.10.13 mitogen: 0.3.6

May I know what was the fix?

Thanks

Renji-FR commented 4 months ago

Hi,

I have the same issue:

The full traceback is:
Traceback (most recent call last):
  File "/home/renji/virtualenvs/f5/network_9/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 165, in run
    res = self._execute()
  File "/home/renji/virtualenvs/f5/network_9/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 574, in _execute
    self._connection = self._get_connection(cvars, templar, current_connection)
  File "/home/renji/virtualenvs/f5/network_9/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 959, in _get_connection
    connection, plugin_load_context = self._shared_loader_obj.connection_loader.get_with_context(
  File "/home/renji/git/acorus-networks/netops-fortinet-ansible/ansible/plugins/mitogen-0.3.7/ansible_mitogen/strategy.py", line 125, in wrap_connection_loader__get
    return ansible_mitogen.loaders.connection_loader__get(name, *args, **kwargs)
  File "/home/renji/virtualenvs/f5/network_9/lib/python3.10/site-packages/ansible/plugins/loader.py", line 936, in get_with_context
    obj.__init__(instance, *args, **kwargs)  # pylint: disable=unnecessary-dunder-call
  File "/home/renji/virtualenvs/f5/network_9/lib/python3.10/site-packages/ansible_collections/ansible/netcommon/plugins/connection/network_cli.py", line 359, in __init__
    super(Connection, self).__init__(play_context, new_stdin, *args, **kwargs)
  File "/home/renji/virtualenvs/f5/network_9/lib/python3.10/site-packages/ansible_collections/ansible/netcommon/plugins/plugin_utils/connection_base.py", line 35, in __init__
    super(NetworkConnectionBase, self).__init__(play_context, new_stdin, *args, **kwargs)
  File "/home/renji/virtualenvs/f5/network_9/lib/python3.10/site-packages/ansible_collections/ansible/utils/plugins/plugin_utils/connection_base.py", line 47, in __init__
    self._update_connection_state()
  File "/home/renji/virtualenvs/f5/network_9/lib/python3.10/site-packages/ansible_collections/ansible/utils/plugins/plugin_utils/connection_base.py", line 101, in _update_connection_state
    control_path = ssh._create_control_path(
  File "/home/renji/git/acorus-networks/netops-fortinet-ansible/ansible/plugins/mitogen-0.3.7/ansible_mitogen/plugins/connection/mitogen_ssh.py", line 87, in _create_control_path
    return Connection.vanilla_class._create_control_path(*args, **kwargs)
AttributeError: 'get_with_context_result' object has no attribute '_create_control_path'
fatal: [fw01-backend-ch1-fed.vpn]: FAILED! => {
    "msg": "Unexpected failure during module execution: 'get_with_context_result' object has no attribute '_create_control_path'"
}
ansible                9.7.0
ansible-core       2.16.8
mitogen             0.3.7

ansible [core 2.16.8]
  config file = None
*****
  python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (*****)
  jinja version = 3.1.4
  libyaml = True

Any update to fix this issue please?

Renji-FR commented 4 months ago

I tested this path: https://github.com/mitogen-hq/mitogen/pull/976 and it works for me, why it was not merged in master?

Mitogen 0.3.7

Trailingslashes commented 1 month ago

I got around this error by installing 0.3.14 and removing the strategy option.

strategy_plugins = /opt/mitogen-0.3.14/ansible_mitogen/plugins/strategy
# strategy = mitogen_linear

Ansible [core 2.16.11] Python 3.11.2