pyinfra turns Python code into shell commands and runs them on your servers. Execute ad-hoc commands and write declarative operations. Target SSH servers, local machine and Docker containers. Fast and scales from one server to thousands.
It seems there's an issue where trying to connect via SSH through two or more proxy jumps fails, resulting in a connection timeout error.
To Reproduce
Let's say we have the following SSH configuration file:
Host jumper1
Hostname 10.10.10.1
User devops
Host jumper2
Hostname 10.20.10.1
Port 30028
User devops
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
ProxyJump jumper1
Host test
Hostname 10.20.10.2
User devops
ProxyJump jumper2
When setting ssh_config_file in inventory.py and trying to connect to the test server (which belongs to the test_group) using the following command, a timeout error occurs.
$ pyinfra inventory.py --limit test_group exec -- hostname
--> Loading config...
--> Loading inventory...
--> Connecting to hosts...
[test] Could not connect ([Errno 60] Operation timed out)
--> Disconnecting from hosts...
--> pyinfra error: No hosts remaining!
When printing the config variable before and after the line, it was observed that the config.sock parameter had been changed to None in the second hop.
$ pyinfra inventory.py --limit test_group exec -- hostname
--> Loading config...
--> Loading inventory...
--> Connecting to hosts...
--- debug: New hop
Initial config={'port': 22, 'sock': None, 'username': 'devops'}
Updated config={'port': 22, 'sock': None, 'username': 'devops'}
--- debug: New hop
Initial config={'port': 30028, 'sock': <paramiko.Channel 0 (open) window=2097152 -> <paramiko.Transport at 0x33b6fc0 (cipher ***, *** bits) (active; 1 open channel(s))>>, 'username': 'devops'}
Updated config={'port': '30028', 'sock': None, 'username': 'devops'}
[test] Could not connect ([Errno 60] Operation timed out)
--> Disconnecting from hosts...
--> pyinfra error: No hosts remaining!
When I changed the line as follows:
config.update({k: v for k, v in kwargs.items() if k not in config})
I confirmed that the connection is established successfully, as shown in the following output. However, there seems to be a secondary issue where an EOF-related error occurs at the end.
$ pyinfra inventory.py --limit test_group exec -- hostname
--> Loading config...
--> Loading inventory...
--> Connecting to hosts...
---debug: New hop
Initial config={'port': 22, 'sock': None, 'username': 'devops'}
Updated config={'port': 22, 'sock': None, 'username': 'devops'}
--- debug: New hop
Initial config={'port': 30028, 'sock': <paramiko.Channel 0 (open) window=2097152 -> <paramiko.Transport at 0x6886d80 (cipher ****, **** bits) (active; 1 open channel(s))>>, 'username': 'devops'}
Updated config={'port': 30028, 'sock': <paramiko.Channel 0 (open) window=2097152 -> <paramiko.Transport at 0x6886d80 (cipher ****, **** bits) (active; 1 open channel(s))>>, 'username': 'devops'}
No host key for [10.20.10.1]:30028 found in known_hosts
--- debug: New hop
Initial config={'port': 22, 'allow_agent': False, 'look_for_keys': False, 'username': 'devops', 'timeout': 10, 'pkey': PKey(alg=****, bits=****, fp=****:****), 'sock': <paramiko.Channel 0 (open) window=2097152 in-buffer=32 -> <paramiko.Transport at 0x70b2810 (cipher ****, **** bits) (active; 1 open channel(s))>>}
Updated config={'port': 22, 'allow_agent': False, 'look_for_keys': False, 'username': 'devops', 'timeout': 10, 'pkey': PKey(alg=****, bits=****, fp=****:****), 'sock': <paramiko.Channel 0 (open) window=2097152 in-buffer=32 -> <paramiko.Transport at 0x70b2810 (cipher ****, **** bits) (active; 1 open channel(s))>>}
[test] Connected
--> Preparing operations...
[test] Ready: shell
--> Beginning operation run...
--> Starting operation: server.shell (hostname)
[test] harbor1
[test] Success
--> Results:
Operation Hosts Success Error No Change
server.shell (hostname) 1 1 - -
--> Disconnecting from hosts...
Exception ignored in atexit callback: <function _join_lingering_threads at 0x105eb3920>
Traceback (most recent call last):
File "****/paramiko/transport.py", line 149, in _join_lingering_threads
thr.stop_thread()
File "****/paramiko/transport.py", line 1920, in stop_thread
self.packetizer.close()
File "****/paramiko/packet.py", line 228, in close
self.__socket.close()
File "****/paramiko/channel.py", line 669, in close
self.transport._send_user_message(m)
File "****/paramiko/transport.py", line 1988, in _send_user_message
self._send_message(data)
File "****/paramiko/transport.py", line 1964, in _send_message
self.packetizer.send_message(data)
File "****/paramiko/packet.py", line 468, in send_message
self.write_all(out)
File "****/paramiko/packet.py", line 382, in write_all
raise EOFError()
EOFError:
Operation code & usage: Included above
Target system information: Apple M1 Pro
Example using the @docker connector (helps isolate the problem): I don't think it is relevant with this issue
Expected behavior
It should be able to connect through two or more jump proxies.
Meta
Include output of pyinfra --support.
$ pyinfra --support
If you are having issues with pyinfra or wish to make feature requests, please
check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
When adding an issue, be sure to include the following:
System: Darwin
Platform: macOS-15.0.1-arm64-arm-64bit
Release: 24.0.0
Machine: arm64
pyinfra: v3.1.1
click: v8.1.7
configparser: v7.1.0
distro: v1.9.0
gevent: v24.10.2
jinja2: v3.1.4
packaging: v24.1
paramiko: v3.5.0
python-dateutil: v2.9.0.post0
pywinrm: v0.5.0
setuptools: v75.1.0
typeguard: v4.3.0
typing-extensions: v4.12.2
Executable: ~/.pyenv/versions/test/bin/pyinfra
Python: 3.12.3 (CPython, Clang 15.0.0 (clang-1500.3.9.4))
Describe the bug
It seems there's an issue where trying to connect via SSH through two or more proxy jumps fails, resulting in a connection timeout error.
To Reproduce
Let's say we have the following SSH configuration file:
When setting
ssh_config_file
ininventory.py
and trying to connect to thetest
server (which belongs to thetest_group
) using the following command, a timeout error occurs.I think the problem might be occurring in the following line where
config
andkwargs
are merged, withsock=None
contained inkwargs
being overwritten toconfig
. https://github.com/pyinfra-dev/pyinfra/blob/9ce7ac44ef6e9d5d8d8a926836e8d294cccbff29/pyinfra/connectors/sshuserclient/client.py#L150When printing the
config
variable before and after the line, it was observed that theconfig.sock
parameter had been changed toNone
in the second hop.When I changed the line as follows:
I confirmed that the connection is established successfully, as shown in the following output. However, there seems to be a secondary issue where an
EOF
-related error occurs at the end.@docker
connector (helps isolate the problem): I don't think it is relevant with this issueExpected behavior
It should be able to connect through two or more jump proxies.
Meta
Include output of
pyinfra --support
.-vv
and--debug
.