pyinfra-dev / pyinfra

pyinfra turns Python code into shell commands and runs them on your servers. Execute ad-hoc commands and write declarative operations. Target SSH servers, local machine and Docker containers. Fast and scales from one server to thousands.
https://pyinfra.com
MIT License
3.91k stars 382 forks source link

Pyinfra tries to connect to first ip only. #780

Open romaia opened 2 years ago

romaia commented 2 years ago

Describe the bug

I have a situation my hosts have multiple IPs, but only one is accessible. Pyinfra seems to only try to connect to the first ip resolved. For instance:

myserver has address 172.16.0.4
myserver has address 10.100.48.54

If (by any chance) the order is reversed, pyinfra manages to connect. SSH also manages to connect: it will timeout for the first ip, but will try the second after that. I have also decreased ConnectTimeout in my ssh config to speed up the first timeout.

To Reproduce

Make one host have multiple ips, one being invalid or inacessible.

Expected behavior

pyinfra should try to connect to the second ip after the first timeout.

Meta

    System: Linux
      Platform: Linux-5.4.0-48-generic-x86_64-with-glibc2.29
      Release: 5.4.0-48-generic
      Machine: x86_64
    pyinfra: v1.6.3
    Executable: /home/stoq_pdv_user/.local/bin/pyinfra
    Python: 3.8.5 (CPython, GCC 9.3.0)

pip

$ pyinfra wlx138pdv070 fact facts.InstalledVersion  -vvv --debug
    [pyinfra_cli.main] Deploy directory remains as cwd
--> Loading config...
--> Loading inventory...
    [pyinfra_cli.inventory] Creating fake inventory...

--> Connecting to hosts...
    [pyinfra.api.connectors.ssh] Connecting to: wlx138pdv070 ({'allow_agent': True, 'look_for_keys': True, 'hostname': 'wlx138pdv070', '_pyinfra_force_forward_agent': None, '_pyinfra_ssh_config_file': None, 'timeout': 10})
    [wlx138pdv070] Could not connect (timed out)
    [pyinfra.api.state] Failing hosts: wlx138pdv070
--> pyinfra error: No hosts remaining!
sabuhish commented 2 years ago

Hi, any update or fix?

Fizzadar commented 2 years ago

This is an upstream issue with paramiko, which version do you have installed (pip show paramiko)?

Looking at the code it appears paramiko should try all available IPs: https://github.com/paramiko/paramiko/blob/361a60d44a00e469d8ff6e84519ae5afcbaa7e9f/paramiko/transport.py#L428-L450

romaia commented 2 years ago

I had 2.9 installed:

$ pip show paramiko
Name: paramiko
Version: 2.9.2
Summary: SSH2 protocol library
Home-page: https://paramiko.org
Author: Jeff Forcier
Author-email: jeff@bitprophet.org
License: LGPL
Location: /home/stoq_pdv_user/.local/lib/python3.8/site-packages
Requires: bcrypt, cryptography, pynacl
Required-by: pyinfra

I upgraded to the latest version (2.10.3) and it still fails to connect.

When I try using ssh, I get this (some information redacted):

$ ssh -v wlx048pdv070
OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/xxx/.ssh/config
debug1: /home/xxx/.ssh/config line 19: Applying options for wlx*
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: hostname canonicalisation enabled, will re-parse configuration
debug1: re-parsing configuration
debug1: Reading configuration data /home/xxx/.ssh/config
debug1: /home/xxx/.ssh/config line 19: Applying options for wlx*
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to wlx048pdv070 [172.16.0.4] port 22.
debug1: connect to address 172.16.0.4 port 22: Connection timed out
debug1: Connecting to wlx048pdv070 [10.100.48.64] port 22.
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
romaia commented 2 years ago

I tried adding some debugging code to this code you pointed to in paramiko, but it doesn't seem to be going through that.