Random tracebacks all of a sudden, can't figure out why

[scripty-mctestface]

I'm pretty perplexed on this one. I have several fairly basic python scripts that use netmiko to backup my network devices which have been working great for several years now. A few weeks ago I started to get random traceback errors that I can't figure out why. The obvious thing would be to see if anything has changed. From what I can tell nothing has changed. I haven't modified my scripts, added/removed any devices, upgrade my OS, or upgraded python, pip, etc. The tracebacks happen on random devices, and one of the tracebacks even bombed on a f.write(config.split) command. Most of the tracebacks happen due to "OSError: Search pattern never detected in send_command". In the past, this has only happened to me on slow/legacy devices and it would be the same device every time. Now it's random and cross platform, including Arista, Cisco, Palo Alto.

I'm on a fairly old version of Netmiko (3.4.0) and using Python 3.6.12. I don't mind upgrading but I'm stuck using CentOS 7.8 and can't figure out how to do so. I am using pip 21.1.1. My python environment is setup from the Red Hat Software Collection (https://developers.redhat.com/blog/2018/08/13/install-python3-rhel).

[ktbyers]

Can you post your full exception stack trace?

And then also a section of your code around where the exception happens.

[scripty-mctestface]

Sure,

Here is the traceback:

Traceback (most recent call last):
  File "daily_switch_check.py", line 297, in <module>
    a_lldp = net_connect_arista.send_command('show lldp neighbor')
  File "/home/netmiko/.local/lib/python3.6/site-packages/netmiko/utilities.py", line 500, in wrapper_decorator
    return func(self, *args, **kwargs)
  File "/home/netmiko/.local/lib/python3.6/site-packages/netmiko/base_connection.py", line 1537, in send_command
    search_pattern
OSError: Search pattern never detected in send_command: 1\ \ \ \ \ \ \ Uplink_Core\ \ \ \ \ \ \ \ \ n\/a\ \ \ \ \ \ \ \ \ Po1\ \ \ \ \ \ \ \ \ \ n\/a\ \ \ \ \ \ \ \ \ \ \ \ \ \?\/\-

This is the specific loop:

for a_device in all_arista_devices:
    try:
        arista_host = a_device.get('host')
        net_connect_arista = ConnectHandler(**a_device)
        a_config = net_connect_arista.send_command('show running-config')
        a_version = net_connect_arista.send_command('show version')
        a_inventory = net_connect_arista.send_command('show inventory')
        a_environment = net_connect_arista.send_command('show environment all')
        a_sys_env = net_connect_arista.send_command('show system environment all')
        a_interface = net_connect_arista.send_command('show interfaces status')
        a_int_counters = net_connect_arista.send_command('show interfaces counters')
        a_int_count_dis = net_connect_arista.send_command('show interfaces counters discards')
        a_int_count_err = net_connect_arista.send_command('show interfaces counters errors')
        a_show_int = net_connect_arista.send_command('show interfaces')
        a_transceiver = net_connect_arista.send_command('show interfaces transceiver')
        a_ip_int = net_connect_arista.send_command('show ip interface brief')
        a_arp = net_connect_arista.send_command('show arp')
        a_mac = net_connect_arista.send_command('show mac address-table')
        a_mlag = net_connect_arista.send_command('show mlag')
        a_mlag_int = net_connect_arista.send_command('show mlag interfaces')
        a_vlan = net_connect_arista.send_command('show vlan')
        a_stp = net_connect_arista.send_command('show spanning-tree')
        a_lldp = net_connect_arista.send_command('show lldp neighbor')
        a_vrrp = net_connect_arista.send_command('show vrrp brief')
        a_bgp = net_connect_arista.send_command('show ip bgp summary')
        a_ospf = net_connect_arista.send_command('show ip ospf neighbor')
        a_route = net_connect_arista.send_command('show ip route')
        a_pim = net_connect_arista.send_command('show ip pim neighbor')
        a_mroute = net_connect_arista.send_command('show ip mroute')
        a_igmp = net_connect_arista.send_command('show ip igmp groups')
        a_snoop = net_connect_arista.send_command('show ip igmp snooping groups')
        a_ptp = net_connect_arista.send_command('show ptp')
        f = open(f'/home/netmiko/daily_switch_check/{arista_host}_switch_check_{date}.txt', mode='w')
        f.write('=' * 30)
        f.write(' show running-config ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_config)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show version ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_version)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show inventory ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_inventory)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show environment all ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_environment)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show system environment all ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_sys_env)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show interfaces status ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_interface)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show interfaces counters ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_int_counters)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show interfaces counters discards ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_int_count_dis)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show interfaces counters errors ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_int_count_err)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show interfaces  ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_show_int)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show interfaces transceiver ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_transceiver)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show ip interface brief ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_ip_int)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show arp ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_arp)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show mac address-table ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_mac)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show mlag ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_mlag)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show mlag interfaces ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_mlag_int)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show vlan ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_vlan)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show spanning-tree ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_stp)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show lldp neighbor ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_lldp)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show vrrp brief ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_vrrp)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show ip bgp summary ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_bgp)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show ip ospf neighbor ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_ospf)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show ip route ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_route)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show ip pim neighbor ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_pim)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show ip mroute ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_mroute)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show ip igmp groups ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_igmp)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show ip igmp snooping groups ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_snoop)
        f.write('\n')
        f.write('\n')
        f.write('=' * 30)
        f.write(' show ptp ')
        f.write('=' * 30)
        f.write('\n')
        f.write('\n')
        f.write(a_ptp)
        f.close()
        net_connect_arista.disconnect()
    except (NetMikoTimeoutException):
        print ('\n' + 'Connection to device timed-out: ' + arista_host)
        continue
    except (AuthenticationException):
        print ('\n' + 'Authentication failure: ' + arista_host)
        continue

[scripty-mctestface]

So one thing that I do is that I have multiple python scripts that reference the device dictionaries from my main backup script via symbolic links, that way I only have to update a dictionary/device in one place. The above script is simply importing the modules from my backup script. Oddly enough, the backup script is randomly failing as well on some days, but that error seems to be consistent:

Traceback (most recent call last):
  File "network_backup.py", line 117, in <module>
    f.write(arista_config.split("\n",5)[5])
IndexError: list index out of range

Code Snippet:

for a_device in all_arista_devices:
    try:
        arista_host = a_device.get('host')
        net_connect_arista = ConnectHandler(**a_device)
        arista_config = net_connect_arista.send_command('show running-config')
        f = open(f'/home/netmiko/network_backup/backups/{arista_host}_backup.txt', mode='w')
        f.write(arista_config.split("\n",5)[5])
        f.close()
        net_connect_arista.disconnect()
    except (NetMikoTimeoutException):
        print ('\n' + 'Connection to device timed-out: ' + arista_host)
        continue
    except (AuthenticationException):
        print ('\n' + 'Authentication failure: ' + arista_host)
        continue

The reason I start on the 5th line is that I found that avoids this issue in where a network config changes each day by an empty line. (I commit the changes into git after the backup is completed).

[ktbyers]

Which version of Netmiko are you using?

[scripty-mctestface]

I'm using 3.4.0. Here is what I have installed with pip:

$ pip3 list
Package             Version
------------------- -------
bcrypt              3.2.0
cffi                1.14.5
cryptography        3.4.7
future              0.18.2
importlib-resources 5.1.2
netmiko             3.4.0
ntc-templates       2.0.0
paramiko            2.7.2
pip                 21.1.1
pycparser           2.20
PyNaCl              1.4.0
pyserial            3.5
scp                 0.13.3
setuptools          56.1.0
six                 1.16.0
tenacity            7.0.0
textfsm             1.1.0
virtualenv          15.1.0
zipp                3.4.1

[ktbyers]

Netmiko 3.X is no more. You would need to reproduce the problem on Netmiko 4.X (i.e. any fix would need to be on the Netmiko 4.x code base so we would first need to reproduce the issue there).

[scripty-mctestface]

Got it, thanks for looking into this Kirk! I can't seem to find an easy way to update Python 3.6, pip, or netmiko on this host. Elsewhere I see that Python virtual environments are the way to go here so I will go that route.

$ pip3 install -U netmiko --trusted-host pypi.org
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: http://pypi.org/simple
Requirement already satisfied: netmiko in ./.local/lib/python3.6/site-packages (3.4.0)
Requirement already satisfied: setuptools>=38.4.0 in ./.local/lib/python3.6/site-packages (from netmiko) (56.1.0)
Requirement already satisfied: ntc-templates in ./.local/lib/python3.6/site-packages (from netmiko) (2.0.0)
Requirement already satisfied: scp>=0.13.2 in ./.local/lib/python3.6/site-packages (from netmiko) (0.13.3)
Requirement already satisfied: importlib-resources in ./.local/lib/python3.6/site-packages (from netmiko) (5.1.2)
Requirement already satisfied: pyserial in ./.local/lib/python3.6/site-packages (from netmiko) (3.5)
Requirement already satisfied: paramiko>=2.6.0 in ./.local/lib/python3.6/site-packages (from netmiko) (2.7.2)
Requirement already satisfied: tenacity in ./.local/lib/python3.6/site-packages (from netmiko) (7.0.0)
Requirement already satisfied: pynacl>=1.0.1 in ./.local/lib/python3.6/site-packages (from paramiko>=2.6.0->netmiko) (1.4.0)
Requirement already satisfied: bcrypt>=3.1.3 in ./.local/lib/python3.6/site-packages (from paramiko>=2.6.0->netmiko) (3.2.0)
Requirement already satisfied: cryptography>=2.5 in ./.local/lib/python3.6/site-packages (from paramiko>=2.6.0->netmiko) (3.4.7)
Requirement already satisfied: six>=1.4.1 in ./.local/lib/python3.6/site-packages (from bcrypt>=3.1.3->paramiko>=2.6.0->netmiko) (1.16.0)
Requirement already satisfied: cffi>=1.1 in ./.local/lib/python3.6/site-packages (from bcrypt>=3.1.3->paramiko>=2.6.0->netmiko) (1.14.5)
Requirement already satisfied: pycparser in ./.local/lib/python3.6/site-packages (from cffi>=1.1->bcrypt>=3.1.3->paramiko>=2.6.0->netmiko) (2.20)
Requirement already satisfied: zipp>=0.4 in ./.local/lib/python3.6/site-packages (from importlib-resources->netmiko) (3.4.1)
Requirement already satisfied: textfsm<2.0.0,>=1.1.0 in ./.local/lib/python3.6/site-packages (from ntc-templates->netmiko) (1.1.0)
Requirement already satisfied: future in ./.local/lib/python3.6/site-packages (from textfsm<2.0.0,>=1.1.0->ntc-templates->netmiko) (0.18.2)

[ktbyers]

PY3.6 is end-of-life (unfortunately for you).

Usually you update it at the OS level (either apt/yum or downloading/installing newer Python on macOS or Windows).

Yeah, after you get a newer Python installed, then I would create a virtual environment (using that new Python) and install Netmiko and related dependencies there.

[ktbyers]

If you are still running into this issue on current Netmiko just re-open the issue.