Closed narcotico-g closed 3 years ago
In your case 3, you would need to include the timeout in inventory as part of a connection option. With what you are trying above, that timeout argument will not be used as the Nornir-Netmiko task plugin is expecting send_command
arguments (and timeout isn't one of those arguments).
Format in Nornir inventory should be:
device_name:
connection_options:
netmiko:
extras:
timeout: 4
I thought that I had closed this issue.
Yes, with that It was solved. I had not found details about inventory for netmiko plugin (only old posts at that moment).
I dont use github habitually. So I am not sure if I should open a new post or reuse this one at this moment. It is similar case.
Now, after update to Nornir 3 and use Netmiko plugin I get a lot of timeout errors (SSH timeout mainly, but banner too) with slow/old devices. It looks the timeout specified in inventory is not working because I get timeout in less time than timeout specified (test with 10, 20, 40 and without specified one), However I get timeout in few sec: 5-11 sec).
01-26-2021 12:02:16.968 backup:DEBUG --> Task : starting backup... 01-26-2021 12:02:21.999 backup:ERROR --> Task: cannot connect by SSH (Timeout)
netmiko 3.3.2 nornir-netmiko 0.1.1 nornir-utils 0.1.1 paramiko 2.7.2
Thanks
@narcotico-g You are going to need to put a full exception stack trace and your code.
Regards, Kirk
After preparing a test to show you I have got some conclusion but I do not know the cause.
The real code I ran with Nornir 2 and previous version of Netmiko (I do not remember exact number) it was run successfully with 20-30 threads. After update, It showed up errors I said before (timeout excepcions, banner, socket.error...).
I prepared shorter code for showing you and the results are the next ones (only 7 devices for the test):
Could you find a cause/explanation?
I send code and test. This time, the exception is the same, but after update I get these ones and others.
Test inventory
Host1:
hostname: <host1 IP>
platform: hp_procurve
username: <user>
password: <pass>
secret: <pass>
data:
bakcom: show runn
id: 10
backup: yes
connection_options:
netmiko:
extras: {"secret": <pass>}
Host2:
hostname: <host2 IP>
platform: hp_procurve
username: <user>
password: <pass>
secret: <pass>
data:
bakcom: show runn
id: 11
backup: yes
connection_options:
netmiko:
extras: {"secret": <pass>}
Host3:
hostname: <host3 IP>
platform: hp_procurve
username: <user>
password: <pass>
secret: <pass>
data:
bakcom: show runn
id: 12
backup: yes
connection_options:
netmiko:
extras: {"secret": <pass>}
Host4:
hostname: <host4 IP>
platform: hp_procurve
username: <user>
password: <pass>
secret: <pass>
data:
bakcom: show runn
id: 13
backup: yes
connection_options:
netmiko:
extras: {"secret": <pass>}
Host5:
hostname: <host5 IP>
platform: hp_procurve
username: <user>
password: <pass>
secret: <pass>
data:
bakcom: show runn
id: 14
backup: yes
connection_options:
netmiko:
extras: {"secret": <pass>}
Host6:
hostname: <host6 IP>
platform: hp_procurve
username: <user>
password: <pass>
secret: <pass>
data:
bakcom: show runn
id: 15
backup: yes
connection_options:
netmiko:
extras: {"secret": <pass>}
Host7:
hostname: <host7 IP>
platform: hp_procurve
username: <user>
password: <pass>
secret: <pass>
data:
bakcom: show runn
id: 16
backup: yes
connection_options:
netmiko:
extras: {"secret": <pass>}
Test ran:
def getConfig (task):
delay_factor = 4 if (task.host.platform == "huawei") else 2
try:
logger.error (task.host.data)
r = task.run (task=netmiko_send_command, name="Getting config", command_string=task.host.data['bakcom'], delay_factor = delay_factor)
except Exception as e:
logger.error ("Host '" + task.host.name + "': exception in getConfig")
logger.debug ("Host '" + task.host.name + "': exception in getConfig. Showing traceback... ", exc_info = True)
if __name__ == '__main__':
try:
short_options = "dl:n:h"
long_options = ["debug", "list", "hostname", "help"]
argument_list = sys.argv[1:]
arguments, values = getopt.getopt (argument_list, short_options, long_options)
for current_argument, current_value in arguments:
if current_argument in ("-d", "--debug"): DEBUG = True
elif current_argument in ("-h", "--help"):
print ("Params: \n -n, --hostname <device> \t Run in specified device with debug enabled \n -l , --list <file> \t\t File with hostnames devices separated by comma")
sys.exit()
logger = initializeLog (ERRORLOG, DEBUGLOG, DEBUG)
nr = InitNornir (config_file="./config.nornir3.yaml")
res=nr.run (task=getConfig, name="Devices backup")
logger.debug ("Time elapsed (seg): " + str(time.time() - START_TIME))
except getopt.error as e:
print ("Params Error: " + str(e))
Config nornir
runner:
plugin: threaded
options:
num_workers: 30
inventory:
plugin: SimpleInventory
options:
host_file: "inventory/hosts.yaml"
logging:
enabled: False
c Traceback and errors:
01-27-2021 12:13:40.590 test:ERROR --> {'bakcom': 'show runn', 'id': 10, 'backup': 'yes'}
01-27-2021 12:13:40.591 test:ERROR --> {'bakcom': 'show runn', 'id': 11, 'backup': 'yes'}
01-27-2021 12:13:40.592 test:ERROR --> {'bakcom': 'show runn', 'id': 12, 'backup': 'yes'}
01-27-2021 12:13:40.593 test:ERROR --> {'bakcom': 'show runn', 'id': 13, 'backup': 'yes'}
01-27-2021 12:13:40.594 test:ERROR --> {'bakcom': 'show runn', 'id': 15, 'backup': 'yes'}
01-27-2021 12:13:40.594 test:ERROR --> {'bakcom': 'show runn', 'id': 16, 'backup': 'yes'}
01-27-2021 12:13:40.595 test:ERROR --> {'bakcom': 'show runn', 'id': 14, 'backup': 'yes'}
01-27-2021 12:13:45.627 task:ERROR --> Host 'Host1': task 'Getting config' failed with traceback:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 920, in establish_connection
self.remote_conn_pre.connect(**ssh_connect_params)
File "/usr/local/lib/python3.6/site-packages/paramiko/client.py", line 349, in connect
retry_on_signal(lambda: sock.connect(addr))
File "/usr/local/lib/python3.6/site-packages/paramiko/util.py", line 283, in retry_on_signal
return function()
File "/usr/local/lib/python3.6/site-packages/paramiko/client.py", line 349, in <lambda>
retry_on_signal(lambda: sock.connect(addr))
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/nornir/core/task.py", line 98, in start
r = self.task(self, **self.params)
File "/usr/local/lib/python3.6/site-packages/nornir_netmiko/tasks/netmiko_send_command.py", line 26, in netmiko_send_command
net_connect = task.host.get_connection(CONNECTION_NAME, task.nornir.config)
File "/usr/local/lib/python3.6/site-packages/nornir/core/inventory.py", line 448, in get_connection
extras=conn.extras,
File "/usr/local/lib/python3.6/site-packages/nornir/core/inventory.py", line 499, in open_connection
configuration=configuration,
File "/usr/local/lib/python3.6/site-packages/nornir_netmiko/connections/netmiko.py", line 59, in open
connection = ConnectHandler(**parameters)
File "/usr/local/lib/python3.6/site-packages/netmiko/ssh_dispatcher.py", line 312, in ConnectHandler
return ConnectionClass(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 346, in __init__
self._open()
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 351, in _open
self.establish_connection()
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 942, in establish_connection
raise NetmikoTimeoutException(msg)
netmiko.ssh_exception.NetmikoTimeoutException: TCP connection to device failed.
Common causes of this problem are:
1. Incorrect hostname or IP address.
2. Wrong TCP port.
3. Intermediate firewall blocking access.
Device settings: hp_procurve <host1 IP>:22
01-27-2021 12:13:45.627 test:ERROR --> Host 'Host1': exception in getConfig
01-27-2021 12:13:45.629 task:ERROR --> Host 'Host3': task 'Getting config' failed with traceback:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 920, in establish_connection
self.remote_conn_pre.connect(**ssh_connect_params)
File "/usr/local/lib/python3.6/site-packages/paramiko/client.py", line 349, in connect
retry_on_signal(lambda: sock.connect(addr))
File "/usr/local/lib/python3.6/site-packages/paramiko/util.py", line 283, in retry_on_signal
return function()
File "/usr/local/lib/python3.6/site-packages/paramiko/client.py", line 349, in <lambda>
retry_on_signal(lambda: sock.connect(addr))
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/nornir/core/task.py", line 98, in start
r = self.task(self, **self.params)
File "/usr/local/lib/python3.6/site-packages/nornir_netmiko/tasks/netmiko_send_command.py", line 26, in netmiko_send_command
net_connect = task.host.get_connection(CONNECTION_NAME, task.nornir.config)
File "/usr/local/lib/python3.6/site-packages/nornir/core/inventory.py", line 448, in get_connection
extras=conn.extras,
File "/usr/local/lib/python3.6/site-packages/nornir/core/inventory.py", line 499, in open_connection
configuration=configuration,
File "/usr/local/lib/python3.6/site-packages/nornir_netmiko/connections/netmiko.py", line 59, in open
connection = ConnectHandler(**parameters)
File "/usr/local/lib/python3.6/site-packages/netmiko/ssh_dispatcher.py", line 312, in ConnectHandler
return ConnectionClass(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 346, in __init__
self._open()
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 351, in _open
self.establish_connection()
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 942, in establish_connection
raise NetmikoTimeoutException(msg)
netmiko.ssh_exception.NetmikoTimeoutException: TCP connection to device failed.
Common causes of this problem are:
1. Incorrect hostname or IP address.
2. Wrong TCP port.
3. Intermediate firewall blocking access.
Device settings: hp_procurve <host3 IP>:22
01-27-2021 12:13:45.630 test:ERROR --> Host 'Host3': exception in getConfig
01-27-2021 12:13:45.631 task:ERROR --> Host 'Host4': task 'Getting config' failed with traceback:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 920, in establish_connection
self.remote_conn_pre.connect(**ssh_connect_params)
File "/usr/local/lib/python3.6/site-packages/paramiko/client.py", line 349, in connect
retry_on_signal(lambda: sock.connect(addr))
File "/usr/local/lib/python3.6/site-packages/paramiko/util.py", line 283, in retry_on_signal
return function()
File "/usr/local/lib/python3.6/site-packages/paramiko/client.py", line 349, in <lambda>
retry_on_signal(lambda: sock.connect(addr))
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/nornir/core/task.py", line 98, in start
r = self.task(self, **self.params)
File "/usr/local/lib/python3.6/site-packages/nornir_netmiko/tasks/netmiko_send_command.py", line 26, in netmiko_send_command
net_connect = task.host.get_connection(CONNECTION_NAME, task.nornir.config)
File "/usr/local/lib/python3.6/site-packages/nornir/core/inventory.py", line 448, in get_connection
extras=conn.extras,
File "/usr/local/lib/python3.6/site-packages/nornir/core/inventory.py", line 499, in open_connection
configuration=configuration,
File "/usr/local/lib/python3.6/site-packages/nornir_netmiko/connections/netmiko.py", line 59, in open
connection = ConnectHandler(**parameters)
File "/usr/local/lib/python3.6/site-packages/netmiko/ssh_dispatcher.py", line 312, in ConnectHandler
return ConnectionClass(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 346, in __init__
self._open()
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 351, in _open
self.establish_connection()
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 942, in establish_connection
raise NetmikoTimeoutException(msg)
netmiko.ssh_exception.NetmikoTimeoutException: TCP connection to device failed.
Common causes of this problem are:
1. Incorrect hostname or IP address.
2. Wrong TCP port.
3. Intermediate firewall blocking access.
Device settings: hp_procurve <host4 IP>:22
01-27-2021 12:13:45.633 task:ERROR --> Host 'Host2': task 'Getting config' failed with traceback:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 920, in establish_connection
self.remote_conn_pre.connect(**ssh_connect_params)
File "/usr/local/lib/python3.6/site-packages/paramiko/client.py", line 349, in connect
retry_on_signal(lambda: sock.connect(addr))
File "/usr/local/lib/python3.6/site-packages/paramiko/util.py", line 283, in retry_on_signal
return function()
File "/usr/local/lib/python3.6/site-packages/paramiko/client.py", line 349, in <lambda>
retry_on_signal(lambda: sock.connect(addr))
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/nornir/core/task.py", line 98, in start
r = self.task(self, **self.params)
File "/usr/local/lib/python3.6/site-packages/nornir_netmiko/tasks/netmiko_send_command.py", line 26, in netmiko_send_command
net_connect = task.host.get_connection(CONNECTION_NAME, task.nornir.config)
File "/usr/local/lib/python3.6/site-packages/nornir/core/inventory.py", line 448, in get_connection
extras=conn.extras,
File "/usr/local/lib/python3.6/site-packages/nornir/core/inventory.py", line 499, in open_connection
configuration=configuration,
File "/usr/local/lib/python3.6/site-packages/nornir_netmiko/connections/netmiko.py", line 59, in open
connection = ConnectHandler(**parameters)
File "/usr/local/lib/python3.6/site-packages/netmiko/ssh_dispatcher.py", line 312, in ConnectHandler
return ConnectionClass(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 346, in __init__
self._open()
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 351, in _open
self.establish_connection()
File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 942, in establish_connection
raise NetmikoTimeoutException(msg)
netmiko.ssh_exception.NetmikoTimeoutException: TCP connection to device failed.
Common causes of this problem are:
1. Incorrect hostname or IP address.
2. Wrong TCP port.
3. Intermediate firewall blocking access.
Device settings: hp_procurve <host2 IP>:22
01-27-2021 12:13:45.633 test:ERROR --> Host 'Host4': exception in getConfig
01-27-2021 12:13:45.634 test:ERROR --> Host 'Host2': exception in getConfig
Are you using any TACACS+ for authentication or do you potentially have a firewall blocking access to the end device(s)?
The case is the same as previous versión.
I do not use TACACS+, but I use Radius. About this, with some of these devices I can see the login prompt is slow (I open SSH client and I have to wait 3-5 seg for the SSH connection is established or the prompt shows up). But it happens with other devices too.
There are firewalls but permit these connections.
I Will check firewalls logs/deny anyway for discarding.
I have checked but I can not see deny traffic. For example, in a connection with error I got:
start_time="2021-01-28 11:51:20" duration=21 policy_id=xxx service=SSH proto=6 src zone=xxx dst zone=xxx action=Permit sent=2225 rcvd=2143 src=xxx dst=xxx src_port=35720 dst_port=22 src-xlated ip=xxx port=35720 dst-xlated ip=xxx port=22 session_id=1838060 reason=Close - AGE OUT
In serial way, I get same errors with slow devices (maybe less quantity, but it happens too). I "think" as cause some timer or behaviour that has changed because when I got these errors (before upgrade if I config more than 40 workers) ALWAYS happens after a lot of seconds (1 min or more) and now it happens after 5 or 6 sec.
To try limit the scenario, I will prepare to running with netmiko directly meanwhile.
Maybe try increasing the netmiko conn_timeout
to 20 seconds.
This would be under netmiko > connection_options in Nornir inventory.
eos:
platform: eos
connection_options:
netmiko:
platform: arista_eos
extras:
conn_timeout: 20
That is an Arista example, but should show the pattern.
Regards, Kirk
Last day I was reading all netmiko realease notes because I got same errors working only with Netmiko and I can see that timeout.
After modifying new timers it looks it is working fine with Netmiko.
Feel free for closing this post.
Thanks for your support and development of Netmiko.
Regards.
Hello,
When I try to connect with netmiko to an offline or dead host, it takes too long. This is solved sending custom timeout and it is working fine.
However, using this timeout with nornir is not working (it takes the same time as netmiko with default timeout) or I am doing something wrong.
Simple examples for testing:
1) Netmiko default (about 100 sec), ok:
2) Netmiko with timeout=4 (about 4 sec). Ok
3) Nornir with timeout=4 (about 100 sec). Why?
I use this code to test (nornir inventory with 1 host, same IP). 1)
2)
3)
Thanks.