search

ktbyers / netmiko

Multi-vendor library to simplify Paramiko SSH connections to network devices
MIT License
3.61k stars 1.31k forks source link

Unable to interact with a specific Ubiquiti Edge Switch - Others are fine. #3471

Open kaussaq opened 3 months ago

kaussaq commented 3 months ago

Description of Issue/Question

Issue connecting to a specific Ubiquiti Edgeswitch advising Pattern not detected, despite find prompt showing the exact pattern the error is asking for.

I'm fairly new to this so I apologise in advance if I've missed something basic.

Please could you advise, as I've tried a few of the ubiquiti_edge* options and they all seem to do the same thing.

Setup

Windows 11, python 3.12.4, Microsoft Visual Studio Code 1.91.1

(UBNT EdgeSwitch) >show version

Switch: 1

System Description............................. EdgePoint Switch 16-Port, 1.6.0.4900860, Linux 3.6.5-f4a26ed5, 1.0.0.4857129 Machine Type................................... EdgePoint Switch 16-Port Machine Model.................................. EP-S16 Software Version............................... 1.6.0.4900860

Netmiko version

(Paste verbatim output from pip freeze | grep netmiko between quotes below)

I'm using windows so grep is not usable: netmiko==4.4.0

Netmiko device_type (if relevant to the issue)

(Paste device_type between quotes below)


'device_type': 'ubiquiti_edgeswitch'

Steps to Reproduce the Issue

connect to device using netmiko.

Try to run command show run

Error Traceback

(Paste the complete traceback of the exception between quotes below)

Traceback (most recent call last):
  File "c:\Users\TomBuckfield\OneDrive - Connexin LTD\ScriptyBoys\EdgeBackupProject\testing.py", line 15, in <module>
    output = net_connect.send_command('show run')
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\TomBuckfield\AppData\Local\Programs\Python\Python312\Lib\site-packages\netmiko\base_connection.py", line 111, in wrapper_decorator
    return_val = func(self, *args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\TomBuckfield\AppData\Local\Programs\Python\Python312\Lib\site-packages\netmiko\utilities.py", line 596, in wrapper_decorator
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\TomBuckfield\AppData\Local\Programs\Python\Python312\Lib\site-packages\netmiko\base_connection.py", line 1826, in send_command
    raise ReadTimeout(msg)
netmiko.exceptions.ReadTimeout:
Pattern not detected: '\\(UBNT\\ EdgeSwitch\\)\\ \\#' in output.

Things you might try to fix this:
1. Explicitly set your pattern using the expect_string argument.
2. Increase the read_timeout to a larger value.

You can also look at the Netmiko session_log or debug log for more information.

Relevant Python code

(Please try to essentialize your Python code to the minimum code needed to reproduce the issue) (Paste the code between the quotes below)


switch = {
    'device_type': 'ubiquiti_edgeswitch',
    'host': 'ip address would be here',
    'username': username,
    'password': password,
    'secret': password,
}

net_connect = ConnectHandler(**switch)
output = net_connect.find_prompt()
print(output)
output = net_connect.send_command('show run')
print(output)
net_connect.disconnect()
ktbyers commented 3 months ago

You might want to check the session log to see better what is going on.

Basically Netmiko is saying it never saw the prompt get returned on the show run call. I believe the default read_timeout is 10s (so the prompt wasn't seen in that time).

kaussaq commented 3 months ago

Thanks for the recommendation, with that being said, it looks like the command runs but the output isn't finished before it times out.

I'm guessing the read timeout needs to be higher than the time it takes for the command to finish on the switch itself, is that correct in your eyes?

ktbyers commented 3 months ago

Yeah, if show run takes longer than 10s, you will need to increase the read_timeout=x (where x is roughly 2X the number of seconds expected for the command).

kaussaq commented 3 months ago

Just logged onto the switch, ran the show run command and it outputted probably one of the smallest configs of all of the devices I'm working through without issue.

Set read timeout to 300s and still getting the same error.

Any other suggestions at all please? Can't see it being the switch at fault here as it's responsive when I SSH Directly?

ktbyers commented 3 months ago

Have you looked at the session log or the Netmiko debug log?

kaussaq commented 3 months ago

I'm still looking at the session log.

This time it took the full 300 seconds to output the below, less than it did without the read timeout:

| __|| | (c) 2010-2018 | | / |/ |/ \ Ubiquiti Networks, Inc. | || (| | (| | / |_._|\. |_| https://www.ubnt.com |___/

Welcome to EdgeSwitch

By logging in, accessing or using the Ubiquiti (UBNT) product, you acknowledge that you have read and understood the Ubiquiti Licence Agreement (available in the GUI and at https://www.ubnt.com/eula/) and agree to be bound by its terms.

(UBNT EdgeSwitch) > (UBNT EdgeSwitch) > (UBNT EdgeSwitch) > (UBNT EdgeSwitch) >enable

(UBNT EdgeSwitch) # (UBNT EdgeSwitch) # (UBNT EdgeSwitch) #terminal length 0

(UBNT EdgeSwitch) # (UBNT EdgeSwitch) # (UBNT EdgeSwitch) # (UBNT EdgeSwitch) #show runexit

I'm just looking at getting debug logging on my test file.

kaussaq commented 3 months ago

Just adding the debug log, and it's getting stuck after the first portion of the output.

It is showing pattern found though, starting off the show run, then has 2000 lines of DEBUG:netmiko:read_channel:

DEBUG:netmiko:Pattern found: (#|>) (UBNT EdgeSwitch) # DEBUG:netmiko:read_channel: DEBUG:netmiko:[find_prompt()]: prompt is (UBNT EdgeSwitch) # DEBUG:netmiko:In disable_paging DEBUG:netmiko:Command: terminal length 0

DEBUG:netmiko:write_channel: b'terminal length 0\n' DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: terminal length 0

(UBNT EdgeSwitch) # DEBUG:netmiko:Pattern found: (terminal\ length\ 0) terminal length 0 DEBUG:netmiko:terminal length 0 DEBUG:netmiko:Exiting disable_paging DEBUG:netmiko:read_channel: DEBUG:netmiko:Clear buffer detects data in the channel DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:write_channel: b'\n' DEBUG:netmiko:read_channel: DEBUG:netmiko:write_channel: b'\n' DEBUG:netmiko:read_channel: DEBUG:netmiko:write_channel: b'\n' DEBUG:netmiko:read_channel: (UBNT EdgeSwitch) # DEBUG:netmiko:read_channel: DEBUG:netmiko:[find_prompt()]: prompt is (UBNT EdgeSwitch) # DEBUG:netmiko:write_channel: b'show run\n' DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: (UBNT EdgeSwitch) # DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: (UBNT EdgeSwitch) # DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: show run DEBUG:netmiko:Pattern found: (show\ run) (UBNT EdgeSwitch) # (UBNT EdgeSwitch) #show run DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel:

!Current Configuration: ! !System Description "EdgePoint Switch 16-Port, 1.6.0.4900860, Linux 3.6.5-f4a26ed5, 1.0.0.4857129" !System Software Version "1.6.0.4900860" !System Up Time "396 days 14 hrs 33 mins 3 secs" !Additional Packages QOS,IPv6 Management,Routing !Current SNTP Synchronized Time: SNTP Last Attempt Status Is Not Successful !

network protocol none network parms redacted no network ipv6 enable vlan database redacted exit

network mgmt_vlan no ip http server ip http secure-se DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel:

ktbyers commented 3 months ago

Debug basically is showing that your device stops outputting data for some reason.

I.E. Netmiko keeps looking for the next part of the show run output and it is not showing up.

How long does the show run take to execute if you run it manually (roughly).

kaussaq commented 3 months ago

Thought so.

Less than 15 seconds when directly logged in, that includes hitting space bar to get to the end of the output also.

ktbyers commented 3 months ago

I would try:

import time
# other imports

switch = {
    'device_type': 'ubiquiti_edgeswitch',
    'host': 'ip address would be here',
    'username': username,
    'password': password,
    'secret': password,
}

net_connect = ConnectHandler(**switch)
output = net_connect.find_prompt()
print(output)
# sleep for a few seconds
time.sleep(4)
output = net_connect.send_command('show run', expect_string="#", read_timeout=30)
print(output)
net_connect.disconnect()
kaussaq commented 3 months ago

Just tried the above, output in console:

Pattern not detected: '#' in output.

Things you might try to fix this:

Explicitly set your pattern using the expect_string argument. Increase the read_timeout to a larger value. You can also look at the Netmiko session_log or debug log for more information.

Debug now stops after the 'show run':

DEBUG:netmiko:read_channel: (UBNT EdgeSwitch) # DEBUG:netmiko:read_channel: DEBUG:netmiko:[find_prompt()]: prompt is (UBNT EdgeSwitch) # DEBUG:netmiko:write_channel: b'show run\n' DEBUG:netmiko:read_channel: (UBNT EdgeSwitch) # (UBNT EdgeSwitch) # DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: show run DEBUG:netmiko:Pattern found: (show\ run) (UBNT EdgeSwitch) # (UBNT EdgeSwitch) #show run DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel:

Session log producing:

Welcome to EdgeSwitch

By logging in, accessing or using the Ubiquiti (UBNT) product, you acknowledge that you have read and understood the Ubiquiti Licence Agreement (available in the GUI and at https://www.ubnt.com/eula/) and agree to be bound by its terms.

(UBNT EdgeSwitch) > (UBNT EdgeSwitch) > (UBNT EdgeSwitch) > (UBNT EdgeSwitch) >enable

(UBNT EdgeSwitch) # (UBNT EdgeSwitch) # (UBNT EdgeSwitch) #terminal length 0

(UBNT EdgeSwitch) # (UBNT EdgeSwitch) # (UBNT EdgeSwitch) # (UBNT EdgeSwitch) #show runexit

I cant make sense as to why Netmiko can't see the output of the command but can see the prompt and the command being entered.

ktbyers commented 3 months ago

What do you see in your log/session_log if you try this?

output = net_connect.send_command_timing('show run')

Rest of the code can be the same.

kaussaq commented 3 months ago

Session log shows the same as before unfortunately:

| __|| | (c) 2010-2018 | | / |/ |/ \ Ubiquiti Networks, Inc. | || (| | (| | / |_._|\. |_| https://www.ubnt.com |___/

Welcome to EdgeSwitch

By logging in, accessing or using the Ubiquiti (UBNT) product, you acknowledge that you have read and understood the Ubiquiti Licence Agreement (available in the GUI and at https://www.ubnt.com/eula/) and agree to be bound by its terms.

(UBNT EdgeSwitch) > (UBNT EdgeSwitch) > (UBNT EdgeSwitch) > (UBNT EdgeSwitch) >enable

(UBNT EdgeSwitch) # (UBNT EdgeSwitch) # (UBNT EdgeSwitch) #terminal length 0

(UBNT EdgeSwitch) # (UBNT EdgeSwitch) # (UBNT EdgeSwitch) # (UBNT EdgeSwitch) #show runexit

The script finished earlier though and debug log reflects that also:

(UBNT EdgeSwitch) # DEBUG:netmiko:Pattern found: (terminal\ length\ 0) terminal length 0 DEBUG:netmiko:terminal length 0 DEBUG:netmiko:Exiting disable_paging DEBUG:netmiko:read_channel: DEBUG:netmiko:Clear buffer detects data in the channel DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:write_channel: b'\n' DEBUG:netmiko:read_channel: DEBUG:netmiko:write_channel: b'\n' DEBUG:netmiko:read_channel: DEBUG:netmiko:write_channel: b'\n' DEBUG:netmiko:read_channel: (UBNT EdgeSwitch) # DEBUG:netmiko:read_channel: DEBUG:netmiko:[find_prompt()]: prompt is (UBNT EdgeSwitch) # DEBUG:netmiko:write_channel: b'show run\n' DEBUG:netmiko:read_channel: (UBNT EdgeSwitch) # (UBNT EdgeSwitch) #show run DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:write_channel: b'\n' DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:write_channel: b'exit\n' DEBUG:paramiko.transport:EOF in transport thread

I also believe there's about that many lines in the actual full config, just obviously isn't displaying the result in each line.

ktbyers commented 3 months ago

What about?

import time
# other imports

switch = {
    'device_type': 'ubiquiti_edgeswitch',
    'host': 'ip address would be here',
    'username': username,
    'password': password,
    'secret': password,
    'default_enter': "\r\n",        # ADDED THIS
}

net_connect = ConnectHandler(**switch)
output = net_connect.find_prompt()
print(output)
# sleep for a few seconds
time.sleep(4)
output = net_connect.send_command('show run', expect_string="#", read_timeout=30)
print(output)
net_connect.disconnect()
ktbyers commented 3 months ago

Note I added the default_enter argument to the switch dictionary.

kaussaq commented 3 months ago

I get a syntax error doing this:

switch = { 'device_type': 'ubiquiti_edgeswitch', 'host': '...', 'username': username, 'password': password, 'secret': password, 'session_log': f 'default_enter': "\r\n", }

Error:

'default_enter': "\r\n", ^^^^^^^^^^^^^^^ SyntaxError: invalid syntax

ktbyers commented 3 months ago

Your dictionary needs commas after every key-value pair (your session_log doesn't have one):

switch = {
'device_type': 'ubiquiti_edgeswitch',
'host': '...',
'username': username,
'password': password,
'secret': password,
'session_log': file_name,  # Needs a comma
'default_enter': "\r\n",
}
kaussaq commented 3 months ago

Of course it does, I deleted that by accident, apologies.

It printed the full output the first time, when I ran it again I'm now back at square one with the error:

Pattern not detected: '#' in output.

Things you might try to fix this:

  1. Explicitly set your pattern using the expect_string argument.
  2. Increase the read_timeout to a larger value.

You can also look at the Netmiko session_log or debug log for more information.

ktbyers commented 3 months ago

If you run it a few more times (the code that worked once), does it work some of the time?

kaussaq commented 3 months ago

Sorry for delayed response - I've been tweaking this today now that I've had some sleep, and managed to get it reliably working in the below format:

switch = { 'device_type': 'ubiquiti_edgeswitch', 'host': '...', 'username': username, 'password': password, 'secret': password, 'default_enter': "\r\n", }

net_connect = ConnectHandler(**switch) output = net_connect.find_prompt() print(output) output = net_connect.send_command("show run") print(output) net_connect.disconnect()

It seems 'default_enter': "\r\n" is the deciding factor here.

For future reference, what is it that this argument is doing that would change the outcome of what's being displayed/picked up by netmiko?

I'm a little out of my depth here I think.