paramiko.ssh_exception.ProxyCommandFailure: ProxyCommand

arob182 commented 2 years ago

ISSUE:

I tried following the thread, #2645, but could not reach a solution. My topology is typical.. Win10 ---> Jumpserver --> Device. When running my /.ssh/config script via powershell I can route through the proxy to end device with no issue. Even without specifying the path in powershell I route through the proxy (ex: ssh user@X.X.X.X brings me through the proxy to the switch). When I attempt to use netmiko I get thrown these errors. Below is my /.ssh/config, python script, and output. I am using Netmiko 3.4.0 and paramiko 2.8.1

Paramiko.ssh_exception.ProxyCommandFailure: ProxyCommand("ssh -F c:/users//.ssh/config -W :22 jumphost

permissionError: [WinError 5] Access is denied

/.ssh/config

host jumphost
IdentityFile ~/.ssh/id_rsa
User test
HostName domain.com

host * !jumphost
User localuser
ProxyCommand ssh -F c:/users//.ssh/config -W %h:%p jumphost

script

from netmiko import ConnectHandler
from getpass import getpass
import os

cisco_9300 = {
'device_type' : 'cisco_xe',
'host' : 'host',
'username' : 'localuser',
'password' : 'password',
'ssh_config_file' : r"c:\users\.ssh\config",
}

net_conenct = ConnectHandler(**cisco_9300)
output = net_conenct.sendcommand("show inv")
print(output)

Python Return Code

`(base) C:\Users\ >python C:\Users\ \Python\test.py
Exception: ProxyCommand("ssh -F c:/users/ /.ssh/config -W :22 jumphost") returned nonzero exit status: An operation was attempted on something that is not a socket
Traceback (most recent call last):
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\proxy.py", line 107, in recv
r, w, x = select([self.process.stdout], [], [], select_timeout)
OSError: [WinError 10038] An operation was attempted on something that is not a socket

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\transport.py", line 2039, in run
self._check_banner()
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\transport.py", line 2211, in _check_banner
buf = self.packetizer.readline(timeout)
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\packet.py", line 380, in readline
buf += self._read_timeout(timeout)
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\packet.py", line 607, in _read_timeout
x = self.__socket.recv(128)
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\proxy.py", line 119, in recv
raise ProxyCommandFailure(" ".join(self.cmd), e.strerror)
paramiko.ssh_exception.ProxyCommandFailure: ProxyCommand("ssh -F c:/users/ /.ssh/config -W :22 jumphost") returned nonzero exit status: An operation was attempted on something that is not a socket

Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Users\ \Anaconda3\lib\threading.py", line 973, in _bootstrap_inner
Traceback (most recent call last):
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\proxy.py", line 107, in recv
self.run()
r, w, x = select([self.process.stdout], [], [], select_timeout)
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\transport.py", line 2166, in run
OSError: [WinError 10038] An operation was attempted on something that is not a socket
self.sock.close()

During handling of the above exception, another exception occurred:

File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\proxy.py", line 122, in close
Traceback (most recent call last):
os.kill(self.process.pid, signal.SIGTERM)
File "C:\Users\ \Python\test.py", line 13, in
PermissionError: [WinError 5] Access is denied
net_conenct = ConnectHandler(**cisco_9300)
File "C:\Users\ \Anaconda3\lib\site-packages\netmiko\ssh_dispatcher.py", line 326, in ConnectHandler
return ConnectionClass(*args, **kwargs)
File "C:\Users\ \Anaconda3\lib\site-packages\netmiko\cisco\cisco_ios.py", line 17, in init
return super().init(*args, **kwargs)
File "C:\Users\ \Anaconda3\lib\site-packages\netmiko\base_connection.py", line 350, in init
self._open()
File "C:\Users\ \Anaconda3\lib\site-packages\netmiko\base_connection.py", line 355, in _open
self.establish_connection()
File "C:\Users\ \Anaconda3\lib\site-packages\netmiko\base_connection.py", line 935, in establish_connection
self.remote_conn_pre.connect(**ssh_connect_params)
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\client.py", line 406, in connect
t.start_client(timeout=timeout)
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\transport.py", line 660, in start_client
raise e
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\transport.py", line 2039, in run
self._check_banner()
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\transport.py", line 2211, in _check_banner
buf = self.packetizer.readline(timeout)
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\packet.py", line 380, in readline
buf += self._read_timeout(timeout)
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\packet.py", line 607, in _read_timeout
x = self.__socket.recv(128)
File "C:\Users\ \Anaconda3\lib\site-packages\paramiko\proxy.py", line 119, in recv
raise ProxyCommandFailure(" ".join(self.cmd), e.strerror)
paramiko.ssh_exception.ProxyCommandFailure: ProxyCommand("ssh -F c:/users/ /.ssh/config -W :22 jumphost") returned nonzero exit status: An operation was attempted on something that is not a socket

ktbyers commented 2 years ago

Why the double forward slash here?

ProxyCommand ssh -F c:/users//.ssh/config -W %h:%p jumphost

arob182 commented 2 years ago

@ktbyers I just deleted my company username from the directory. Sorry should have specified that.

ktbyers commented 2 years ago

@arob182 Okay, so you edited this error line here?

paramiko.ssh_exception.ProxyCommandFailure: ProxyCommand("ssh -F c:/users/ /.ssh/config -W :22 jumphost") returned nonzero exit status: An operation was attempted on something that is not a socket

And that extra space is not really in the output?

Just verifying.

arob182 commented 2 years ago

@ktbyers same with

paramiko.ssh_exception.ProxyCommandFailure: ProxyCommand("ssh -F c:/users/ /.ssh/config -W :22 jumphost") returned nonzero exit status: An operation was attempted on something that is not a socket

after the -W is the IP of the host put here 'host' : 'host',

arob182 commented 2 years ago

@arob182 Okay, so you edited this error line here?
paramiko.ssh_exception.ProxyCommandFailure: ProxyCommand("ssh -F c:/users/ /.ssh/config -W :22 jumphost") returned nonzero exit status: An operation was attempted on something that is not a socket
And that extra space is not really in the output?

Just verifying.

Correct, I can replace with a filler if that makes parsing this easier :)

arob182 commented 2 years ago

Here is a new run of the script with it filled < host> as the IP i am trying to reach and < user> for the name.

(base) C:\Users\<user>>python c:\users\<user>\python\test.py
Exception: ProxyCommand("ssh -F c:/users/<user>/.ssh/config -W <host>:22 jumphost") returned nonzero exit status: An operation was attempted on something that is not a socket
Traceback (most recent call last):
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\proxy.py", line 107, in recv
    r, w, x = select([self.process.stdout], [], [], select_timeout)
OSError: [WinError 10038] An operation was attempted on something that is not a socket

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\transport.py", line 2039, in run
    self._check_banner()
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\transport.py", line 2211, in _check_banner
    buf = self.packetizer.readline(timeout)
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\packet.py", line 380, in readline
    buf += self._read_timeout(timeout)
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\packet.py", line 607, in _read_timeout
    x = self.__socket.recv(128)
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\proxy.py", line 119, in recv
    raise ProxyCommandFailure(" ".join(self.cmd), e.strerror)
paramiko.ssh_exception.ProxyCommandFailure: ProxyCommand("ssh -F c:/users/<user>/.ssh/config -W <host>:22 jumphost") returned nonzero exit status: An operation was attempted on something that is not a socket

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\<user>\Anaconda3\lib\threading.py", line 973, in _bootstrap_inner
Traceback (most recent call last):
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\proxy.py", line 107, in recv
    r, w, x = select([self.process.stdout], [], [], select_timeout)
OSError: [WinError 10038] An operation was attempted on something that is not a socket
    self.run()

During handling of the above exception, another exception occurred:

  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\transport.py", line 2166, in run
Traceback (most recent call last):
  File "c:\users\<user>\python\test.py", line 13, in <module>
    self.sock.close()
    net_conenct = ConnectHandler(**cisco_9300)
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\proxy.py", line 122, in close
  File "C:\Users\<user>\Anaconda3\lib\site-packages\netmiko\ssh_dispatcher.py", line 326, in ConnectHandler
    os.kill(self.process.pid, signal.SIGTERM)
PermissionError: [WinError 5] Access is denied
    return ConnectionClass(*args, **kwargs)
  File "C:\Users\<user>\Anaconda3\lib\site-packages\netmiko\cisco\cisco_ios.py", line 17, in __init__
    return super().__init__(*args, **kwargs)
  File "C:\Users\<user>\Anaconda3\lib\site-packages\netmiko\base_connection.py", line 350, in __init__
    self._open()
  File "C:\Users\<user>\Anaconda3\lib\site-packages\netmiko\base_connection.py", line 355, in _open
    self.establish_connection()
  File "C:\Users\<user>\Anaconda3\lib\site-packages\netmiko\base_connection.py", line 935, in establish_connection
    self.remote_conn_pre.connect(**ssh_connect_params)
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\client.py", line 406, in connect
    t.start_client(timeout=timeout)
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\transport.py", line 660, in start_client
    raise e
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\transport.py", line 2039, in run
    self._check_banner()
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\transport.py", line 2211, in _check_banner
    buf = self.packetizer.readline(timeout)
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\packet.py", line 380, in readline
    buf += self._read_timeout(timeout)
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\packet.py", line 607, in _read_timeout
    x = self.__socket.recv(128)
  File "C:\Users\<user>\Anaconda3\lib\site-packages\paramiko\proxy.py", line 119, in recv
    raise ProxyCommandFailure(" ".join(self.cmd), e.strerror)
paramiko.ssh_exception.ProxyCommandFailure: ProxyCommand("ssh -F c:/users/<user>/.ssh/config -W <host>:22 jumphost") returned nonzero exit status: An operation was attempted on something that is not a socket

ktbyers commented 2 years ago

This error looks like some sort of a Windows permissions error:

permissionError: [WinError 5] Access is denied

That is where I would search and dig some more on.

I assume you have Windows Subsystem for Linux?

arob182 commented 2 years ago

@ktbyers Thanks ill dig into it some more. I do have WSL, but I am relatively new to utilizing it. I will also look into how to run the script from there.

ktbyers commented 2 years ago

Yeah, that permissionError/WinError is the main thing that looked odd to me.

That and this message also:

An operation was attempted on something that is not a socket

That is where I would probably start.

You might want to also try to just directly use the SSH config level from WSL and openssh (i.e. test wheter it works outside of Netmiko using WSL).

deveshmanish commented 2 years ago

Did you manage to get a solution?

shibu503 commented 1 year ago

I am receiving same error permissionError: [WinError 5] Access is denied. @arob182 did you manage to get the solution? Using windows 10, python version is 3.9.1, netmiko 4.1.1. Trying to use ssh proxy with Windows as script server, remote device is behind linux host.

ktbyers commented 1 year ago

@shibu503 @deveshmanish Can you try this fix?

https://github.com/paramiko/paramiko/pull/1924/files

And let me know if it works?

shibu503 commented 1 year ago

Hi Ktbyers,

Thanks for your prompt response.

I have already tried the above fix to resolve same issue as mentioned in https://github.com/ktbyers/netmiko/issues/2855. I was initially receiving below error:

"ProxyCommand("ssh -F ./config -W :22 jumphost") returned nonzero exit status: An operation was attempted on something that is not a socket"

After making changes in paramiko/proxy.py as mentioned in https://github.com/paramiko/paramiko/pull/1924/files, the error was gone however getting permissionError: [WinError 5] Access is denied. From windows cli, ssh -F C:\Users\ShibuJ.ssh\config , it works fine as I am able to login to device proxying via jump host.

Below is full error:

PS C:\Users\ShibuJ\Documents\python3-venv1\p3-project> & c:/Users/ShibuJ/Documents/python3-venv1/p3-project/Scripts/python.exe c:/Users/ShibuJ/Documents/python3-venv1/p3-project/test.py {'R8': Host: R8} Exception in thread Thread-1: Traceback (most recent call last): File "threading.py", line 954, in _bootstrap_inner File "C:\Users\ShibuJ\Documents\python3-venv1\p3-project\lib\site-packages\paramiko\transport.py", line 2226, in run self.sock.close() File "C:\Users\ShibuJ\Documents\python3-venv1\p3-project\lib\site-packages\paramiko\proxy.py", line 127, in close os.kill(self.process.pid, signal.SIGTERM) PermissionError: [WinError 5] Access is denied netmiko_send_config*****

R8 changed : False ** vvvv netmiko_send_config changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv ERROR Traceback (most recent call last): File "C:\Users\ShibuJ\Documents\python3-venv1\p3-project\lib\site-packages\nornir\core\task.py", line 99, in start r = self.task(self, self.params) File "C:\Users\ShibuJ\Documents\python3-venv1\p3-project\lib\site-packages\nornir_netmiko\tasks\netmiko_send_config.py", line 28, in netmiko_send_config net_connect = task.host.get_connection(CONNECTION_NAME, task.nornir.config) File "C:\Users\ShibuJ\Documents\python3-venv1\p3-project\lib\site-packages\nornir\core\inventory.py", line 494, in get_connection self.open_connection( File "C:\Users\ShibuJ\Documents\python3-venv1\p3-project\lib\site-packages\nornir\core\inventory.py", line 546, in open_connection conn_obj.open( File "C:\Users\ShibuJ\Documents\python3-venv1\p3-project\lib\site-packages\nornir_netmiko\connections\netmiko.py", line 59, in open connection = ConnectHandler(*parameters) File "C:\Users\ShibuJ\Documents\python3-venv1\p3-project\lib\site-packages\netmiko\ssh_dispatcher.py", line 351, in ConnectHandler return ConnectionClass(args, **kwargs) File "C:\Users\ShibuJ\Documents\python3-venv1\p3-project\lib\site-packages\netmiko\base_connection.py", line 434, in init self._open() File "C:\Users\ShibuJ\Documents\python3-venv1\p3-project\lib\site-packages\netmiko\base_connection.py", line 439, in _open
self.establish_connection() File "C:\Users\ShibuJ\Documents\python3-venv1\p3-project\lib\site-packages\netmiko\base_connection.py", line 1080, in establish_connection self.remote_conn = self.remote_conn_pre.invoke_shell( File "C:\Users\ShibuJ\Documents\python3-venv1\p3-project\lib\site-packages\paramiko\client.py", line 547, in invoke_shell
chan = self._transport.open_session() File "C:\Users\ShibuJ\Documents\python3-venv1\p3-project\lib\site-packages\paramiko\transport.py", line 920, in open_session
return self.open_channel( File "C:\Users\ShibuJ\Documents\python3-venv1\p3-project\lib\site-packages\paramiko\transport.py", line 1051, in open_channel raise e paramiko.ssh_exception.SSHException: Unable to open channel.

^^^^ END netmiko_send_config ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PS C:\Users\ShibuJ\Documents\python3-venv1\p3-project>

Let me know if anyother logs or output is needed. I am using nornir with netmiko for my code.

Using below version:

netmiko 4.1.1 nornir 3.3.0 nornir-netmiko 0.2.0 nornir-utils 0.2.0 paramiko 2.11.0

Thanks Shibu

ktbyers commented 1 year ago

@shibu503 Have you tried (in your SSH config file)?

ProxyCommand ssh -F C:\Users\ShibuJ.ssh\config -W %h:%p jumphost

Also please post your full SSH config file (after making the above change).

If your SSH config file is large and has other sensitive information in it, then you should make a simplified version of the file that you can test with and share here (also just obscure anything that is confidential).

shibu503 commented 1 year ago

Hi Ktbyers,

Below is SSH config file:

# The intermediate server
host 192.168.2.1
  # Use only the key specified in IdentityFile
  IdentitiesOnly yes
  # The SSH key to use to the intermediate server
  IdentityFile ~/.ssh/id_rsa
  User shibuj
  HostName 192.168.2.1

host * !192.168.2.1
  User shibuj
  KexAlgorithms diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
  # -F forces usage of this SSH config file
  ProxyCommand ssh -F C:/Users/ShibuJ/.ssh/config -W %h:%p 192.168.2.1

Also from windows powershell, I can successfully login to remove device via ssh -F C:\Users\ShibuJ.ssh\config x.x.x.x.

Thanks Shibu John

ktbyers commented 1 year ago

And same error using the above (as the previous error)?

shibu503 commented 1 year ago

Hi Ktbyers,

Same error, also the I am using user has administrator privileges for the windows pc.

Thanks Shibu John

youngbobnetwork commented 1 year ago

@shibu503 did you ever come to a conclusion on this? A year later I decided to try netmiko again but still hit the wall of I don’t think I can get this to work.

ktbyers / netmiko

paramiko.ssh_exception.ProxyCommandFailure: ProxyCommand #2695

Below is full error:

Using below version: