Exception when running on slow to connect hosts

[julienlavergne]

Describe the bug

A exception is raise when running some tasks on 13 hosts. I do not get the error when running only on 4 hosts, or only 1 hosts. I suspect the error shows more often as the number of hosts increases.

PyInfra v3.1.1

I noticed it happends with machines on which the Connecting to hosts part takes a bit of time (1-3 seconds). When I use hosts that connects in less than ~1 sec, there is not issue.

In some occasions, I only get the Key Error _print_spinner Exception, but then the process is able to continue without facing any other Exception.

EDIT: I confirm that I can trigger the issue when using only 2 hosts that are slow to connect to.

To Reproduce

I was not able to create a minimal reproducible example. My set of files always reproduces the issue. But when I reduce it, the error disappear. So it may have something to do with a particular timing.

• Operation code & usage
• Target system information
• Example using the @docker connector (helps isolate the problem)

Expected behavior

No exception

Meta

• Include output of pyinfra --support.

  
  If you are having issues with pyinfra or wish to make feature requests, please
  check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
  When adding an issue, be sure to include the following:
  
  System: Linux
    Platform: Linux-4.18.0-553.22.1.el8_10.x86_64-x86_64-with-glibc2.28
    Release: 4.18.0-553.22.1.el8_10.x86_64
    Machine: x86_64
  pyinfra: v3.1.1
    black: v24.10.0
    black: v24.10.0
    click: v8.1.7
    configparser: v7.1.0
    distro: v1.9.0
    gevent: v24.10.2
    jinja2: v3.1.4
    packaging: v24.1
    paramiko: v3.5.0
    python-dateutil: v2.9.0.post0
    pywinrm: v0.5.0
    setuptools: v70.1.0
    typeguard: v4.3.0
    typing-extensions: v4.12.2
    wheel: v0.43.0
  Executable: <hidden path>/.tox/pyinfra/bin/pyinfra
  Python: 3.11.8 (CPython, GCC 12.3.0)

• How was pyinfra installed (source/pip)?

  pip install

• Include pyinfra-debug.log (if one was created)
• Consider including output with -vv and --debug.

  
  aiodns==3.2.0,aiohappyeyeballs==2.4.3,aiohttp==3.10.10,aiosignal==1.3.1,attrs==24.2.0,bcrypt==4.2.0,black==24.10.0,Brotli==1.1.0,certifi==2024.8.30,cffi==1.17.1,charset-normalizer==3.4.0,click==8.1.7,configparser==7.1.0,cryptography==43.0.1,distro==1.9.0,frozenlist==1.4.1,gevent==24.10.2,greenlet==3.1.1,hvac==2.3.0,idna==3.10,Jinja2==3.1.4,MarkupSafe==3.0.1,multidict==6.1.0,mypy-extensions==1.0.0,packaging==24.1,paramiko==3.5.0,pathspec==0.12.1,pip==24.1,platformdirs==4.3.6,propcache==0.2.0,pycares==4.4.0,pycparser==2.22,pyinfra==3.1.1,PyNaCl==1.5.0,pyspnego==0.11.1,python-dateutil==2.9.0.post0,pywinrm==0.5.0,requests==2.32.3,requests_ntlm==1.3.0,setuptools==70.1.0,six==1.16.0,typeguard==4.3.0,typing_extensions==4.12.2,urllib3==2.2.3,wheel==0.43.0,xmltodict==0.14.1,yarl==1.15.2,zope.event==5.0,zope.interface==7.1.0

--> Loading config... --> Loading inventory... --> Connecting to hosts... No host key for host08 found in known_hosts No host key for host07 found in known_hosts No host key for host03 found in known_hosts No host key for host08 found in known_hosts No host key for host04 found in known_hosts No host key for host09 found in known_hosts No host key for host11 found in known_hosts No host key for host07 found in known_hosts No host key for host05 found in known_hosts No host key for host10 found in known_hosts No host key for host01 found in known_hosts No host key for host06 found in known_hosts No host key for host02 found in known_hosts [host08] Connected [host07] Connected [host03] Connected [host04] Connected [host11] Connected [host09] Connected [host08] Connected [host05] Connected [host10] Connected [host07] Connected [host06] Connected [host01] Connected [host02] Connected

--> Preparing operations... --> Preparing Operations... Loading: my_task.py Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap self._bootstrap_inner() File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner del _limbo[self]

KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError

    [host07] Command socket/SSH error: SSHException('Channel is not open',)
    [host07] Error: could not load fact: server.LinuxDistribution 
    [host08] Command socket/SSH error: SSHException('Channel is not open',)
    [host08] Error: could not load fact: server.LinuxDistribution 

--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 176, in do_something
    if linux_distribution["major"] <= 8:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<=' not supported between instances of 'NoneType' and 'int'

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1049, in _bootstrap_inner
    self._delete()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1081, in _delete
    del _active[get_ident()]
        ~~~~~~~^^^^^^^^^^^^^
KeyError: 140737278873728
2024-10-15T03:16:08Z <Greenlet at 0x7ffff3839080: <bound method Thread._bootstrap of <paramiko.Transport at 0xf3847e10 (unconnected)>>> failed with KeyError

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1049, in _bootstrap_inner
    self._delete()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1081, in _delete
    del _active[get_ident()]
        ~~~~~~~^^^^^^^^^^^^^
KeyError: 140737278871008
2024-10-15T03:16:08Z <Greenlet at 0x7ffff38385e0: <bound method Thread._bootstrap of <paramiko.Transport at 0xf37dc490 (unconnected)>>> failed with KeyError

--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1125, in open_channel
    raise e
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 2201, in run
    ptype, m = self.packetizer.read_message()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 496, in read_message
    header = self.read_all(self.__block_size_in, check_rekey=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 324, in read_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError

--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError

--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError

--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError

--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError

--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError

--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError

--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError
```

[julienlavergne]

I pinpointed the issue with an upgrade of gevent dependency. Version 24.10.2 has the issue, but version 24.2.1 does not. Could you freeze that dependency while figuring out what is wrong ?

[Fizzadar]

Hrm that's a big jump in versions, we should try pin down the exact version with the issue and ideally an issue on gevent's side to track against before disabling.