pyinfra turns Python code into shell commands and runs them on your servers. Execute ad-hoc commands and write declarative operations. Target SSH servers, local machine and Docker containers. Fast and scales from one server to thousands.
A exception is raise when running some tasks on 13 hosts. I do not get the error when running only on 4 hosts, or only 1 hosts. I suspect the error shows more often as the number of hosts increases.
PyInfra v3.1.1
I noticed it happends with machines on which the Connecting to hosts part takes a bit of time (1-3 seconds). When I use hosts that connects in less than ~1 sec, there is not issue.
In some occasions, I only get the Key Error _print_spinner Exception, but then the process is able to continue without facing any other Exception.
EDIT: I confirm that I can trigger the issue when using only 2 hosts that are slow to connect to.
To Reproduce
I was not able to create a minimal reproducible example.
My set of files always reproduces the issue. But when I reduce it, the error disappear. So it may have something to do with a particular timing.
Operation code & usage
Target system information
Example using the @docker connector (helps isolate the problem)
Expected behavior
No exception
Meta
Include output of pyinfra --support.
If you are having issues with pyinfra or wish to make feature requests, please
check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
When adding an issue, be sure to include the following:
System: Linux
Platform: Linux-4.18.0-553.22.1.el8_10.x86_64-x86_64-with-glibc2.28
Release: 4.18.0-553.22.1.el8_10.x86_64
Machine: x86_64
pyinfra: v3.1.1
black: v24.10.0
black: v24.10.0
click: v8.1.7
configparser: v7.1.0
distro: v1.9.0
gevent: v24.10.2
jinja2: v3.1.4
packaging: v24.1
paramiko: v3.5.0
python-dateutil: v2.9.0.post0
pywinrm: v0.5.0
setuptools: v70.1.0
typeguard: v4.3.0
typing-extensions: v4.12.2
wheel: v0.43.0
Executable: <hidden path>/.tox/pyinfra/bin/pyinfra
Python: 3.11.8 (CPython, GCC 12.3.0)
--> Loading config...
--> Loading inventory...
--> Connecting to hosts...
No host key for host08 found in known_hosts
No host key for host07 found in known_hosts
No host key for host03 found in known_hosts
No host key for host08 found in known_hosts
No host key for host04 found in known_hosts
No host key for host09 found in known_hosts
No host key for host11 found in known_hosts
No host key for host07 found in known_hosts
No host key for host05 found in known_hosts
No host key for host10 found in known_hosts
No host key for host01 found in known_hosts
No host key for host06 found in known_hosts
No host key for host02 found in known_hosts
[host08] Connected
[host07] Connected
[host03] Connected
[host04] Connected
[host11] Connected
[host09] Connected
[host08] Connected
[host05] Connected
[host10] Connected
[host07] Connected
[host06] Connected
[host01] Connected
[host02] Connected
--> Preparing operations...
--> Preparing Operations...
Loading: my_task.py
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
del _limbo[self]
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
del _limbo[self]
~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError
[host07] Command socket/SSH error: SSHException('Channel is not open',)
[host07] Error: could not load fact: server.LinuxDistribution
[host08] Command socket/SSH error: SSHException('Channel is not open',)
[host08] Error: could not load fact: server.LinuxDistribution
--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "my_task.py", line 20, in <module>
local.include("tasks/task1.py")
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
exec_file(filename)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "<hidden_path>/tasks/task1.py", line 24, in <module>
do_something(
File "<hidden_path>/utils/task2.py", line 176, in do_something
if linux_distribution["major"] <= 8:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<=' not supported between instances of 'NoneType' and 'int'
During handling of the above exception, another exception occurred:
pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
del _limbo[self]
~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1049, in _bootstrap_inner
self._delete()
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1081, in _delete
del _active[get_ident()]
~~~~~~~^^^^^^^^^^^^^
KeyError: 140737278873728
2024-10-15T03:16:08Z <Greenlet at 0x7ffff3839080: <bound method Thread._bootstrap of <paramiko.Transport at 0xf3847e10 (unconnected)>>> failed with KeyError
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1049, in _bootstrap_inner
self._delete()
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1081, in _delete
del _active[get_ident()]
~~~~~~~^^^^^^^^^^^^^
KeyError: 140737278871008
2024-10-15T03:16:08Z <Greenlet at 0x7ffff38385e0: <bound method Thread._bootstrap of <paramiko.Transport at 0xf37dc490 (unconnected)>>> failed with KeyError
--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "my_task.py", line 20, in <module>
local.include("tasks/task1.py")
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
exec_file(filename)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "<hidden_path>/tasks/task1.py", line 24, in <module>
do_something(
File "<hidden_path>/utils/task2.py", line 174, in do_something
linux_distribution = host.get_fact(LinuxDistribution)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
return get_fact(state, host, cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
return _get_fact(
^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
status, output = host.run_shell_command(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
return self.connector.run_shell_command(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
return_code, combined_output = execute_command_with_sudo_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
return_code, output = execute_command()
^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
chan = self._transport.open_session(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
return self.open_channel(
^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1125, in open_channel
raise e
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 2201, in run
ptype, m = self.packetizer.read_message()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 496, in read_message
header = self.read_all(self.__block_size_in, check_rekey=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 324, in read_all
raise EOFError()
EOFError
During handling of the above exception, another exception occurred:
pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
del _limbo[self]
~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError
--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "my_task.py", line 20, in <module>
local.include("tasks/task1.py")
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
exec_file(filename)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "<hidden_path>/tasks/task1.py", line 24, in <module>
do_something(
File "<hidden_path>/utils/task2.py", line 174, in do_something
linux_distribution = host.get_fact(LinuxDistribution)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
return get_fact(state, host, cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
return _get_fact(
^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
status, output = host.run_shell_command(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
return self.connector.run_shell_command(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
return_code, combined_output = execute_command_with_sudo_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
return_code, output = execute_command()
^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
chan = self._transport.open_session(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
return self.open_channel(
^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
self._send_user_message(m)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
self._send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
self.packetizer.send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
self.write_all(out)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
raise EOFError()
EOFError
During handling of the above exception, another exception occurred:
pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
del _limbo[self]
~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError
--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "my_task.py", line 20, in <module>
local.include("tasks/task1.py")
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
exec_file(filename)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "<hidden_path>/tasks/task1.py", line 24, in <module>
do_something(
File "<hidden_path>/utils/task2.py", line 174, in do_something
linux_distribution = host.get_fact(LinuxDistribution)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
return get_fact(state, host, cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
return _get_fact(
^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
status, output = host.run_shell_command(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
return self.connector.run_shell_command(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
return_code, combined_output = execute_command_with_sudo_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
return_code, output = execute_command()
^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
chan = self._transport.open_session(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
return self.open_channel(
^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
self._send_user_message(m)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
self._send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
self.packetizer.send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
self.write_all(out)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
raise EOFError()
EOFError
During handling of the above exception, another exception occurred:
pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
del _limbo[self]
~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError
--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "my_task.py", line 20, in <module>
local.include("tasks/task1.py")
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
exec_file(filename)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "<hidden_path>/tasks/task1.py", line 24, in <module>
do_something(
File "<hidden_path>/utils/task2.py", line 174, in do_something
linux_distribution = host.get_fact(LinuxDistribution)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
return get_fact(state, host, cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
return _get_fact(
^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
status, output = host.run_shell_command(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
return self.connector.run_shell_command(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
return_code, combined_output = execute_command_with_sudo_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
return_code, output = execute_command()
^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
chan = self._transport.open_session(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
return self.open_channel(
^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
self._send_user_message(m)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
self._send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
self.packetizer.send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
self.write_all(out)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
raise EOFError()
EOFError
During handling of the above exception, another exception occurred:
pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
del _limbo[self]
~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError
--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "my_task.py", line 20, in <module>
local.include("tasks/task1.py")
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
exec_file(filename)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "<hidden_path>/tasks/task1.py", line 24, in <module>
do_something(
File "<hidden_path>/utils/task2.py", line 174, in do_something
linux_distribution = host.get_fact(LinuxDistribution)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
return get_fact(state, host, cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
return _get_fact(
^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
status, output = host.run_shell_command(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
return self.connector.run_shell_command(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
return_code, combined_output = execute_command_with_sudo_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
return_code, output = execute_command()
^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
chan = self._transport.open_session(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
return self.open_channel(
^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
self._send_user_message(m)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
self._send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
self.packetizer.send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
self.write_all(out)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
raise EOFError()
EOFError
During handling of the above exception, another exception occurred:
pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
del _limbo[self]
~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError
--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "my_task.py", line 20, in <module>
local.include("tasks/task1.py")
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
exec_file(filename)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "<hidden_path>/tasks/task1.py", line 24, in <module>
do_something(
File "<hidden_path>/utils/task2.py", line 174, in do_something
linux_distribution = host.get_fact(LinuxDistribution)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
return get_fact(state, host, cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
return _get_fact(
^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
status, output = host.run_shell_command(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
return self.connector.run_shell_command(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
return_code, combined_output = execute_command_with_sudo_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
return_code, output = execute_command()
^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
chan = self._transport.open_session(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
return self.open_channel(
^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
self._send_user_message(m)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
self._send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
self.packetizer.send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
self.write_all(out)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
raise EOFError()
EOFError
During handling of the above exception, another exception occurred:
pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
del _limbo[self]
~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError
--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "my_task.py", line 20, in <module>
local.include("tasks/task1.py")
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
exec_file(filename)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "<hidden_path>/tasks/task1.py", line 24, in <module>
do_something(
File "<hidden_path>/utils/task2.py", line 174, in do_something
linux_distribution = host.get_fact(LinuxDistribution)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
return get_fact(state, host, cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
return _get_fact(
^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
status, output = host.run_shell_command(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
return self.connector.run_shell_command(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
return_code, combined_output = execute_command_with_sudo_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
return_code, output = execute_command()
^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
chan = self._transport.open_session(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
return self.open_channel(
^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
self._send_user_message(m)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
self._send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
self.packetizer.send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
self.write_all(out)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
raise EOFError()
EOFError
During handling of the above exception, another exception occurred:
pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>
--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:
Traceback (most recent call last):
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
del _limbo[self]
~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "my_task.py", line 20, in <module>
local.include("tasks/task1.py")
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
exec_file(filename)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
raise UnexpectedExternalError(e, filename)
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError
--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "my_task.py", line 20, in <module>
local.include("tasks/task1.py")
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
exec_file(filename)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "<hidden_path>/tasks/task1.py", line 24, in <module>
do_something(
File "<hidden_path>/utils/task2.py", line 174, in do_something
linux_distribution = host.get_fact(LinuxDistribution)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
return get_fact(state, host, cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
return _get_fact(
^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
status, output = host.run_shell_command(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
return self.connector.run_shell_command(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
return_code, combined_output = execute_command_with_sudo_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
return_code, output = execute_command()
^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
chan = self._transport.open_session(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
return self.open_channel(
^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
self._send_user_message(m)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
self._send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
self.packetizer.send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
self.write_all(out)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
raise EOFError()
EOFError
During handling of the above exception, another exception occurred:
pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>
Traceback (most recent call last):
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
exec(PYTHON_CODES[filename], data)
File "<hidden_path>/tasks/task1.py", line 24, in <module>
do_something(
File "<hidden_path>/utils/task2.py", line 174, in do_something
linux_distribution = host.get_fact(LinuxDistribution)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
return get_fact(state, host, cls, args=args, kwargs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
return _get_fact(
^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
status, output = host.run_shell_command(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
return self.connector.run_shell_command(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
return_code, combined_output = execute_command_with_sudo_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
return_code, output = execute_command()
^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
chan = self._transport.open_session(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
return self.open_channel(
^^^^^^^^^^^^^^^^^^
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
self._send_user_message(m)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
self._send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
self.packetizer.send_message(data)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
self.write_all(out)
File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
raise EOFError()
EOFError
```
I pinpointed the issue with an upgrade of gevent dependency. Version 24.10.2 has the issue, but version 24.2.1 does not. Could you freeze that dependency while figuring out what is wrong ?
Describe the bug
A exception is raise when running some tasks on 13 hosts. I do not get the error when running only on 4 hosts, or only 1 hosts. I suspect the error shows more often as the number of hosts increases.
PyInfra v3.1.1
I noticed it happends with machines on which the
Connecting to hosts
part takes a bit of time (1-3 seconds). When I use hosts that connects in less than ~1 sec, there is not issue.In some occasions, I only get the Key Error _print_spinner Exception, but then the process is able to continue without facing any other Exception.
EDIT: I confirm that I can trigger the issue when using only 2 hosts that are slow to connect to.
To Reproduce
I was not able to create a minimal reproducible example. My set of files always reproduces the issue. But when I reduce it, the error disappear. So it may have something to do with a particular timing.
@docker
connector (helps isolate the problem)Expected behavior
No exception
Meta
Include output of
pyinfra --support
.-vv
and--debug
.--> Loading config... --> Loading inventory... --> Connecting to hosts... No host key for host08 found in known_hosts No host key for host07 found in known_hosts No host key for host03 found in known_hosts No host key for host08 found in known_hosts No host key for host04 found in known_hosts No host key for host09 found in known_hosts No host key for host11 found in known_hosts No host key for host07 found in known_hosts No host key for host05 found in known_hosts No host key for host10 found in known_hosts No host key for host01 found in known_hosts No host key for host06 found in known_hosts No host key for host02 found in known_hosts [host08] Connected [host07] Connected [host03] Connected [host04] Connected [host11] Connected [host09] Connected [host08] Connected [host05] Connected [host10] Connected [host07] Connected [host06] Connected [host01] Connected [host02] Connected
--> Preparing operations... --> Preparing Operations... Loading: my_task.py Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap self._bootstrap_inner() File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner del _limbo[self]