search

ktbyers / netmiko

Multi-vendor library to simplify Paramiko SSH connections to network devices
MIT License
3.56k stars 1.3k forks source link

Palo Alto with SSH Proxy is not working properly #2905

Open achhabr1 opened 2 years ago

achhabr1 commented 2 years ago

SSH Conn object to Palo Alto vFW keeps resetting and throwing errors without any obvious issues. Attaching logs and the the commands i tried

`(python38-venv) ac043s@NJML01AC043S ~ % python Python 3.8.9 (default, Apr 13 2022, 08:48:07) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin Type "help", "copyright", "credits" or "license" for more information.

from netmiko import ConnLogOnly conn = ConnLogOnly(device_type="paloalto_panos", host="ga-u350-14-vfw", username="admin", password="xxxxx", ssh_config_file= "~/.ssh/ssh_config_netmiko", conn_timeout=10, session_log="output.txt") output = conn.send_command("request license info") Traceback (most recent call last): File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/proxy.py", line 79, in send self.process.stdin.write(content) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "", line 1, in File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/utilities.py", line 592, in wrapper_decorator return func(self, *args, *kwargs) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1659, in send_command search_pattern = self._prompt_handler(auto_find_prompt) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1544, in _prompt_handler prompt = self.find_prompt() File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/paloalto/paloalto_panos.py", line 87, in find_prompt return super().find_prompt(delay_factor=delay_factor, pattern=pattern) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1347, in find_prompt self.write_channel(self.RETURN) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 97, in wrapper_decorator return_val = func(self, args, **kwargs) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 111, in wrapper_decorator func(self, out_data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 537, in write_channel self.channel.write_channel(out_data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/channel.py", line 74, in write_channel self.remote_conn.sendall(write_bytes(out_data, encoding=self.encoding)) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/channel.py", line 1208, in _send self.transport._send_user_message(m) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/transport.py", line 1863, in _send_user_message self._send_message(data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/transport.py", line 1839, in _send_message self.packetizer.send_message(data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/packet.py", line 431, in send_message self.write_all(out) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/packet.py", line 336, in write_all n = self.__socket.send(out) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/proxy.py", line 85, in send raise ProxyCommandFailure(" ".join(self.cmd), e.strerror) paramiko.ssh_exception.ProxyCommandFailure: ProxyCommand("ssh -F /Users/ac043s/.ssh/ssh_config_netmiko -W ga-u350-14-vfw:22 jumphost") returned nonzero exit status: Broken pipe

output = conn.send_command("request license info") Traceback (most recent call last): File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/proxy.py", line 79, in send self.process.stdin.write(content) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "", line 1, in File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/utilities.py", line 592, in wrapper_decorator return func(self, *args, *kwargs) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1659, in send_command search_pattern = self._prompt_handler(auto_find_prompt) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1544, in _prompt_handler prompt = self.find_prompt() File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/paloalto/paloalto_panos.py", line 87, in find_prompt return super().find_prompt(delay_factor=delay_factor, pattern=pattern) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1335, in find_prompt self.write_channel(self.RETURN) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 97, in wrapper_decorator return_val = func(self, args, **kwargs) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 111, in wrapper_decorator func(self, out_data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 537, in write_channel self.channel.write_channel(out_data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/channel.py", line 74, in write_channel self.remote_conn.sendall(write_bytes(out_data, encoding=self.encoding)) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/channel.py", line 1208, in _send self.transport._send_user_message(m) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/transport.py", line 1863, in _send_user_message self._send_message(data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/transport.py", line 1839, in _send_message self.packetizer.send_message(data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/packet.py", line 431, in send_message self.write_all(out) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/packet.py", line 336, in write_all n = self.__socket.send(out) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/proxy.py", line 85, in send raise ProxyCommandFailure(" ".join(self.cmd), e.strerror) paramiko.ssh_exception.ProxyCommandFailure: ProxyCommand("ssh -F /Users/ac043s/.ssh/ssh_config_netmiko -W ga-u350-14-vfw:22 jumphost") returned nonzero exit status: Broken pipe

conn = ConnLogOnly(device_type="paloalto_panos", host="ga-u350-14-vfw", username="admin", password="xxxxxx", ssh_config_file= "~/.ssh/ssh_config_netmiko", conn_timeout=10, session_log="output.txt") output = conn.send_command("request license info") output '\n\nCurrent GMT Date: August 19, 2022\n\nLicense entry:\nFeature: Threat Prevention\nDescription: Threat Prevention\nSerial: 015300000007296\nIssued: November 21, 2017\nExpires: December 20, 2036\nExpired?: no\nBase license: PA-VM\n\nLicense entry:\nFeature: DNS Security\nDescription: Palo Alto Networks DNS Security License\nSerial: 015300000007296\nIssued: February 15, 2019\nExpires: December 20, 2036\nExpired?: no\nBase license: PA-VM\n\nLicense entry:\nFeature: WildFire License\nDescription: WildFire signature feed, integrated WildFire logs, WildFire API\nSerial: 015300000007296\nIssued: November 21, 2017\nExpires: December 20, 2036\nExpired?: no\nBase license: PA-VM\n\nLicense entry:\nFeature: GlobalProtect Gateway\nDescription: GlobalProtect Gateway License\nSerial: 015300000007296\nIssued: November 21, 2017\nExpires: December 20, 2036\nExpired?: no\nBase license: PA-VM\n\nLicense entry:\nFeature: Premium\nDescription: 24 x 7 phone support; advanced replacement hardware service\nSerial: 015300000007296\nIssued: November 21, 2017\nExpires: December 20, 2036\nExpired?: no\nBase license: PA-VM\n\nLicense entry:\nFeature: PA-VM\nDescription: Standard VM-50\nSerial: 015300000007296\nIssued: November 21, 2017\nExpires: Never\nExpired?: no\n\nLicense entry:\nFeature: PAN-DB URL Filtering\nDescription: Palo Alto Networks URL Filtering License\nSerial: 015300000007296\nIssued: November 21, 2017\nExpires: December 20, 2036\nExpired?: no\nBase license: PA-VM\n\n' output = conn.send_command("request license info") output = conn.send_command("request license info") output = conn.send_command("ping count 10 source 10.0.49.148 host 8.8.8.8") Traceback (most recent call last): File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/proxy.py", line 79, in send self.process.stdin.write(content) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "", line 1, in File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/utilities.py", line 592, in wrapper_decorator return func(self, *args, *kwargs) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1659, in send_command search_pattern = self._prompt_handler(auto_find_prompt) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1544, in _prompt_handler prompt = self.find_prompt() File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/paloalto/paloalto_panos.py", line 87, in find_prompt return super().find_prompt(delay_factor=delay_factor, pattern=pattern) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1347, in find_prompt self.write_channel(self.RETURN) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 97, in wrapper_decorator return_val = func(self, args, **kwargs) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 111, in wrapper_decorator func(self, out_data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 537, in write_channel self.channel.write_channel(out_data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/channel.py", line 74, in write_channel self.remote_conn.sendall(write_bytes(out_data, encoding=self.encoding)) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/channel.py", line 1208, in _send self.transport._send_user_message(m) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/transport.py", line 1863, in _send_user_message self._send_message(data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/transport.py", line 1839, in _send_message self.packetizer.send_message(data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/packet.py", line 431, in send_message self.write_all(out) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/packet.py", line 336, in write_all n = self.__socket.send(out) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/proxy.py", line 85, in send raise ProxyCommandFailure(" ".join(self.cmd), e.strerror) paramiko.ssh_exception.ProxyCommandFailure: ProxyCommand("ssh -F /Users/ac043s/.ssh/ssh_config_netmiko -W ga-u350-14-vfw:22 jumphost") returned nonzero exit status: Broken pipe

output = conn.send_command("request license info") Traceback (most recent call last): File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/proxy.py", line 79, in send self.process.stdin.write(content) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "", line 1, in File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/utilities.py", line 592, in wrapper_decorator return func(self, *args, *kwargs) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1659, in send_command search_pattern = self._prompt_handler(auto_find_prompt) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1544, in _prompt_handler prompt = self.find_prompt() File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/paloalto/paloalto_panos.py", line 87, in find_prompt return super().find_prompt(delay_factor=delay_factor, pattern=pattern) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1335, in find_prompt self.write_channel(self.RETURN) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 97, in wrapper_decorator return_val = func(self, args, **kwargs) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 111, in wrapper_decorator func(self, out_data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 537, in write_channel self.channel.write_channel(out_data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/channel.py", line 74, in write_channel self.remote_conn.sendall(write_bytes(out_data, encoding=self.encoding)) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/channel.py", line 1208, in _send self.transport._send_user_message(m) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/transport.py", line 1863, in _send_user_message self._send_message(data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/transport.py", line 1839, in _send_message self.packetizer.send_message(data) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/packet.py", line 431, in send_message self.write_all(out) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/packet.py", line 336, in write_all n = self.__socket.send(out) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/paramiko/proxy.py", line 85, in send raise ProxyCommandFailure(" ".join(self.cmd), e.strerror) paramiko.ssh_exception.ProxyCommandFailure: ProxyCommand("ssh -F /Users/ac043s/.ssh/ssh_config_netmiko -W ga-u350-14-vfw:22 jumphost") returned nonzero exit status: Broken pipe

conn = ConnLogOnly(device_type="paloalto_panos", host="ga-u350-14-vfw", username="admin", password="Juniper123", ssh_config_file= "~/.ssh/ssh_config_netmiko", conn_timeout=10, session_log="output.txt") output = conn.send_command("request license info") Traceback (most recent call last): File "", line 1, in File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/utilities.py", line 592, in wrapper_decorator return func(self, *args, **kwargs) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1721, in send_command raise ReadTimeout(msg) netmiko.exceptions.ReadTimeout: Pattern not detected: 'admin@u350\-14\-s2\-vfw>\ admin@u350\-14\-s2\-vfw>' in output.

Things you might try to fix this:

  1. Explicitly set your pattern using the expect_string argument.
  2. Increase the read_timeout to a larger value.

You can also look at the Netmiko session_log or debug log for more information.

output = conn.send_command("ping count 10 source 10.0.49.148 host 8.8.8.8") Traceback (most recent call last): File "", line 1, in File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/utilities.py", line 592, in wrapper_decorator return func(self, *args, **kwargs) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1721, in send_command raise ReadTimeout(msg) netmiko.exceptions.ReadTimeout: Pattern not detected: 'admin@u350\-14\-s2\-vfw>\ admin@u350\-14\-s2\-vfw>' in output.

Things you might try to fix this:

  1. Explicitly set your pattern using the expect_string argument.
  2. Increase the read_timeout to a larger value.

You can also look at the Netmiko session_log or debug log for more information.

output = conn.send_command("ping count 10 source 10.0.49.148 host 8.8.8.8") Traceback (most recent call last): File "", line 1, in File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/utilities.py", line 592, in wrapper_decorator return func(self, *args, **kwargs) File "/Users/ac043s/python38-venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1721, in send_command raise ReadTimeout(msg) netmiko.exceptions.ReadTimeout: Pattern not detected: 'admin@u350\-14\-s2\-vfw>\ admin@u350\-14\-s2\-vfw>' in output.

Things you might try to fix this:

  1. Explicitly set your pattern using the expect_string argument.
  2. Increase the read_timeout to a larger value.

You can also look at the Netmiko session_log or debug log for more information.

`

achhabr1 commented 2 years ago

output.txt

achhabr1 commented 2 years ago

netmiko 4.1.2

ktbyers commented 2 years ago

@achhabr1 Does your SSH proxy setup work outside of Netmiko.

Can you post your SSH proxy setup here?

achhabr1 commented 2 years ago

(python38-venv) ac043s@NJML01AC043S .ssh % cat ssh_config_netmiko host jumphost IdentitiesOnly yes IdentityFile ~/.ssh/id_rsa User ac043s HostName 135.25.157.179

host * !jumphost User ac043s

Force usage of this SSH config file

ProxyCommand ssh -F ~/.ssh/ssh_config_netmiko -W %h:%p jumphost

Alternate solution using netcat

ProxyCommand ssh -F ./ssh_config_netmiko jumphost nc %h %p

(python38-venv) ac043s@NJML01AC043S .ssh %

(python38-venv) ac043s@NJML01AC043S .ssh % ssh -J ac043s@135.25.157.179 admin@ga-u350-14-vfw Warning: Permanently added '135.25.157.179' (RSA) to the list of known hosts.

#####################################################################

Welcome to Middletown Labs

All connections are monitored and recorded

Disconnect IMMEDIATELY if you are not an authorized user!!

#####################################################################

WARNING THIS IS A PRIVATE SERVER FACILITY TO BE ACCESSED AND USED FOR COMPANY BUSINESS PURPOSES ONLY. ACCESS TO IT FOR ANY REASON MUST BE SPECIFICALLY AUTHORIZED. VIOLATORS WILL BE PROSECUTED TO THE FULLEST EXTENT OF LOCAL, STATE AND FEDERAL LAWS.

Warning: Permanently added 'fd:192:168:53:0:9:14:5' (RSA) to the list of known hosts. (admin@fd:192:168:53:0:9:14:5) Password: Last login: Fri Aug 19 19:39:24 2022 from fd:192:168:53::4

Number of failed attempts since last successful login: 0

admin@u350-14-s2-vfw>

admin@u350-14-s2-vfw> show system info

hostname: u350-14-s2-vfw ip-address: 10.192.168.254 public-ip-address: unknown netmask: 255.255.255.0 default-gateway: 10.192.168.1 ip-assignment: static ipv6-address: fd:192:168:53:0:9:14:5/64 ipv6-link-local-address: fe80::428f:9dff:feea:7eae/64 ipv6-default-gateway: fd:192:168:53::5 mac-address: 40:8f:9d:ea:7e:ae time: Mon Aug 22 13:12:47 2022 uptime: 2 days, 17:53:02 family: vm model: PA-VM serial: 015300000007296 vm-mac-base: E4:A7:49:34:B7:00 vm-mac-count: 256 vm-uuid: B7D3D9EF796AC06714F476778922FA24 vm-cpuid: E4516B9CF914DAC022EDB57AAAC8158F vm-license: VM-50 vm-mode: KVM cloud-mode: non-cloud sw-version: 9.1.9 global-protect-client-package-version: 0.0.0 app-version: 8354-6456 app-release-date: av-version: 3565-4076 av-release-date: threat-version: 8354-6456 threat-release-date: wf-private-version: 0 wf-private-release-date: unknown url-db: paloaltonetworks wildfire-version: 515954-518954 wildfire-release-date: url-filtering-version: 20220822.20198 global-protect-datafile-version: unknown global-protect-datafile-release-date: unknown global-protect-clientless-vpn-version: 0 global-protect-clientless-vpn-release-date: logdb-version: 9.1.22 vm_series: vm_series-2.0.6 platform-family: vm vpn-disable-mode: off multi-vsys: off operational-mode: normal device-certificate-status: None

admin@u350-14-s2-vfw>

achhabr1 commented 2 years ago

i feel issue is with command timeout settings. I tired to adjust it as per the recommendations but nothing seemed to worked consistently

ktbyers commented 2 years ago

It looks like the end device is responding slowly. I would try adding a global_delay_factor and set it either to 2, 4, or 8. This is an argument to ConnectHandler.