After upgrading to v.4.3.0 some ERS switches stopped working

[romanstech]

After upgrading to v.4.3.0 some ERS switches stopped working

Setup

Netmiko Avaya ERS switches 5xxx

Netmiko version

netmiko==4.3.0

Netmiko device_type (if relevant to the issue)

avaya_ers

Steps to Reproduce the Issue

Error Traceback

Pattern not detected: '(?:sername|ssword|Ctrl-Y|Press ENTER to continue|(?m:[>#]\\s*$)|Menu)' in output.

Relevant Python code

Need to add/edit rows with 'plus' sign

prompt_pattern = r"(?m:[>#]\s*$)"  # force re.Multiline
+ enable_pattern = "#"

enter_msg = "Press ENTER to continue"
+ menu = "Menu"
+ command_line_interface = "ommand"

pattern = (
+            rf"(?:{uname}|{password}|{cntl_y}|{enter_msg}|{self.prompt_pattern}|{self.enable_pattern}|{menu}|{command_line_interface})"
        )

+ elif menu in new_data or command_line_interface in new_data:
                self.write_channel(CTRL_C)

Without these changes, Netmiko can't connect to some Avaya ERS switches. And these issues sometimes appear randomly. After adding the changes all started working properly.

[ktbyers]

@romanstech Can you show what the new ERS looks like when logging in via SSH manually (in other words, I want to see what the SSH login looks like when it fails)?

[romanstech]

After SSH authentication and disabled banner with Ctrl-Y (I've disabled it on every Avaya switch) we receive:

Use arrow keys to highlight option, press <Return> or <Enter> to select option.
                       Ethernet Routing Switch 5510-48T Main Menu

                        IP Configuration/Setup...
                        SNMP Configuration...
                        System Characteristics...
                        Switch Configuration...
                        Console/Comm Port Configuration...
                        Display Hardware Units...
                        Spanning Tree Configuration...
                        TELNET/SNMP/SSH/Web Access Configuration...
                        Software Download...
                        Configuration File...
                        Display System Log...
                        Reset...
                        Reset to Default Settings...
                        Shutdown Command...
                        Command Line Interface...
                        Logout...

Use arrow keys to highlight option, press <Return> or <Enter> to select option.

Then we need to send C character to call

Command Line Interface...

So I've added check:

command_line_interface = "ommand"

to be sure that the line exists and we can continue. After C we receive standard prompt (user has full permissions so we don't need enable command):

avaya#

From my tests prompt pattern:

prompt_pattern = r"(?m:[>#]\s*$)"  # force re.Multiline

doesn't work well. In previous versions, we had separate enable prompt defined:

enable_pattern = "#"

Then I've added it back and all returned to work properly.

Now my scripts run for 3 days with zero issues.

[shivaraj-arch]

Yes old and new switches give different outputs. when output is uncertain, sshclient of paramiko is the best option. invoke a ssh shell console

ssh = paramiko.SSHClient() #add autopolicy
console = ssh.invoke_shell()
console.keep_this = ssh

which can be then used to send , sleep and recv.

console.send("c\n")
time.sleep(3)
out = str(console.recv(2000).decode('utf-8')

[ktbyers]

Okay, someone will need to submit a PR to fix that...pretty bad choice on ERS's part (i.e. they are making there device's harder to automate, as automation becomes more and more common).

[ktbyers]

@romanstech Can you post the full exception stack trace from the failure?

Also can you post the fix that you did to Netmiko that is working.

You mention this:

From my tests prompt pattern:

prompt_pattern = r"(?m:[>#]\s*$)"  # force re.Multiline

doesn't work well. In previous versions, we had separate enable prompt defined:

enable_pattern = "#"

But prompt pattern would match on a #

>>> import re
>>> prompt_pattern = r"(?m:[>#]\s*$)"
>>> re.search(prompt_pattern, "#")
<re.Match object; span=(0, 1), match='#'>

[romanstech]

Sorry for delay.

Debug:

2024-02-12 13:01:03,670 DEBUG netmiko read_channel: 
2024-02-12 13:01:03,680 DEBUG netmiko read_channel: 
2024-02-12 13:01:03,691 DEBUG netmiko read_channel: 
2024-02-12 13:01:03,701 DEBUG netmiko read_channel: 
2024-02-12 13:01:03,712 DEBUG netmiko read_channel: 
Ethernet Routing Switch 5510-48T Main Menu IP Configuration/Setup...  SNMP Configuration...  System Characteristics... Switch Configuration... Console/Comm Port Configuration... Display Hardware Units... Spanning Tree Configuration... TELNET/SNMP/SSH/Web Access Configuration... Software Download... Configuration File... Display System Log... 
2024-02-12 13:01:03,713 DEBUG netmiko Pattern found: ((?:sername|ssword|Ctrl-Y|Press ENTER to continue|(?m:[>#]\s*$)|Menu|ommand)) 
Ethernet Routing Switch 5510-48T Main Menu
2024-02-12 13:01:03,713 DEBUG netmiko write_channel: b'c'
2024-02-12 13:01:03,713 DEBUG netmiko read_channel: Reset... Reset to Default Settings... Shutdown Command... Command Line Interface... Logout...Use arrow keys to highlight option, press <Return> or <Enter> to select option.[?25l
2024-02-12 13:01:03,713 DEBUG netmiko Pattern found: ((?:sername|ssword|Ctrl-Y|Press ENTER to continue|(?m:[>#]\s*$)|Menu|ommand))  IP Configuration/Setup...  SNMP Configuration...  System Characteristics... Switch Configuration... Console/Comm Port Configuration... Display Hardware Units... Spanning Tree Configuration... TELNET/SNMP/SSH/Web Access Configuration... Software Download... Configuration File... Display System Log... Reset... Reset to Default Settings... Shutdown Command... Command
2024-02-12 13:01:03,714 DEBUG netmiko write_channel: b'c'
2024-02-12 13:01:03,714 DEBUG netmiko read_channel: 
2024-02-12 13:01:03,724 DEBUG netmiko read_channel: 
2024-02-12 13:01:03,734 DEBUG netmiko read_channel: 
2024-02-12 13:01:03,745 DEBUG netmiko read_channel: 
2024-02-12 13:01:03,755 DEBUG netmiko read_channel: 
2024-02-12 13:01:03,765 DEBUG netmiko read_channel: [?25h[?25h
ae5510bc-rocket#c
2024-02-12 13:01:03,776 DEBUG netmiko read_channel: 
2024-02-12 13:01:03,786 DEBUG netmiko read_channel: 
2024-02-12 13:01:08,697 DEBUG netmiko read_channel: 
2024-02-12 13:01:08,707 DEBUG netmiko read_channel: 
2024-02-12 13:01:08,717 ERROR netmiko.ssh_dispatcher An unknown exception occurred during connection:

Pattern not detected: '(?:sername|ssword|Ctrl-Y|Press ENTER to continue|(?m:[>#]\\s*$)|Menu|ommand)' in output.

Things you might try to fix this:
1. Adjust the regex pattern to better identify the terminating string. Note, in
many situations the pattern is automatically based on the network device's prompt.
2. Increase the read_timeout to a larger value.

You can also look at the Netmiko session_log or debug log for more information.

All I did to make it work:

enable_pattern = "#" and rf"(?:{uname}|{password}|{cntl_y}|{enter_msg}|{self.prompt_pattern}|{self.enable_pattern}|{menu}|{command_line_interface})"

Debug after my changes: ` 2024-02-12 13:19:24,005 DEBUG netmiko read_channel: 2024-02-12 13:19:24,016 DEBUG netmiko read_channel: [?25lEthernet Routing Switch 5510-48T Main Menu IP Configuration/Setup...  SNMP Configuration...  System Characteristics... Switch Configuration... Console/Comm Port Configuration... Display Hardware Units... Spanning Tree Configuration... TELNET/SNMP/SSH/Web Access Configuration... Software Download... Configuration File... Display System Log... Reset... Reset to Default Settings... Shutdown Command... Command Line Interface... Logout...Use arrow keys to highlight option, press or to select option.[?25l Ethernet Routing Switch 5510-48T Main Menu IP Configuration/Setup...  SNMP Configuration...  System Characteristics... Switch Configuration... Console/Comm Port Configuration... Display Hardware Units... Spanning Tree Configuration... TELNET/SNMP/SSH/Web Access Configuration... Software Download... Configuration File... Display System Log... Reset... Reset to Default Settings... Shutdown Command... Command Line Interface... Logout...Use arrow keys to highlight option, press or to select option.[?25l 2024-02-12 13:19:24,016 DEBUG netmiko Pattern found: ((?:sername|ssword|Ctrl-Y|Press ENTER to continue|(?m:[>#]\s$)|#|Menu|ommand)) [?25lEthernet Routing Switch 5510-48T Main Menu 2024-02-12 13:19:24,017 DEBUG netmiko write_channel: b'c' 2024-02-12 13:19:24,017 DEBUG netmiko read_channel: 2024-02-12 13:19:24,017 DEBUG netmiko Pattern found: ((?:sername|ssword|Ctrl-Y|Press ENTER to continue|(?m:[>#]\s$)|#|Menu|ommand))  IP Configuration/Setup...  SNMP Configuration...  System Characteristics... Switch Configuration... Console/Comm Port Configuration... Display Hardware Units... Spanning Tree Configuration... TELNET/SNMP/SSH/Web Access Configuration... Software Download... Configuration File... Display System Log... Reset... Reset to Default Settings... Shutdown Command... Command 2024-02-12 13:19:24,017 DEBUG netmiko write_channel: b'c' 2024-02-12 13:19:24,017 DEBUG netmiko read_channel: 2024-02-12 13:19:24,017 DEBUG netmiko Pattern found: ((?:sername|ssword|Ctrl-Y|Press ENTER to continue|(?m:[>#]\s$)|#|Menu|ommand)) Line Interface... Logout...Use arrow keys to highlight option, press or to select option.[?25l Ethernet Routing Switch 5510-48T Main Menu 2024-02-12 13:19:24,017 DEBUG netmiko write_channel: b'c' 2024-02-12 13:19:24,017 DEBUG netmiko read_channel: 2024-02-12 13:19:24,018 DEBUG netmiko Pattern found: ((?:sername|ssword|Ctrl-Y|Press ENTER to continue|(?m:[>#]\s$)|#|Menu|ommand))  IP Configuration/Setup...  SNMP Configuration...  System Characteristics... Switch Configuration... Console/Comm Port Configuration... Display Hardware Units... Spanning Tree Configuration... TELNET/SNMP/SSH/Web Access Configuration... Software Download... Configuration File... Display System Log... Reset... Reset to Default Settings... Shutdown Command... Command 2024-02-12 13:19:24,018 DEBUG netmiko write_channel: b'c' 2024-02-12 13:19:24,018 DEBUG netmiko read_channel: 2024-02-12 13:19:24,028 DEBUG netmiko read_channel: 2024-02-12 13:19:24,038 DEBUG netmiko read_channel: 2024-02-12 13:19:24,048 DEBUG netmiko read_channel: 2024-02-12 13:19:24,059 DEBUG netmiko read_channel: 2024-02-12 13:19:24,069 DEBUG netmiko read_channel: [?25h[?25h ae5510bc-rocket#c 2024-02-12 13:19:24,069 DEBUG netmiko Pattern found: ((?:sername|ssword|Ctrl-Y|Press ENTER to continue|(?m:[>#]\s*$)|#|Menu|ommand)) Line Interface... Logout...Use arrow keys to highlight option, press or to select option.[?25l[?25h[?25h ae5510bc-rocket# 2024-02-12 13:19:24,070 DEBUG netmiko write_channel: b'\n' 2024-02-12 13:19:24,181 DEBUG netmiko read_channel: cc ^ % Invalid input detected at '^' marker.

2024-02-12 13:19:24,182 DEBUG netmiko Clear buffer detects data in the channel 2024-02-12 13:19:24,203 DEBUG netmiko read_channel: 2024-02-12 13:19:24,203 DEBUG netmiko write_channel: b'\n' 2024-02-12 13:19:24,203 DEBUG netmiko read_channel: 2024-02-12 13:19:24,214 DEBUG netmiko read_channel: 2024-02-12 13:19:24,224 DEBUG netmiko read_channel: 2024-02-12 13:19:24,234 DEBUG netmiko read_channel: ae5510bc-rocket#

2024-02-12 13:19:24,234 DEBUG netmiko Parenthesis found in pattern.

pattern: (#|>)

This can be problemtic when used in read_until_pattern().

You should ensure that you use either non-capture groups i.e. '(?:' or that the parenthesis completely wrap the pattern '(pattern)' 2024-02-12 13:19:24,234 DEBUG netmiko Pattern found: (#|>) ae5510bc-rocket# 2024-02-12 13:19:24,245 DEBUG netmiko read_channel: 2024-02-12 13:19:24,245 DEBUG netmiko Clear buffer detects data in the channel 2024-02-12 13:19:24,265 DEBUG netmiko read_channel: 2024-02-12 13:19:24,265 DEBUG netmiko [find_prompt()]: prompt is ae5510bc-rocket# 2024-02-12 13:19:24,265 DEBUG netmiko In disable_paging 2024-02-12 13:19:24,265 DEBUG netmiko Command: terminal length 0

2024-02-12 13:19:24,266 DEBUG netmiko write_channel: b'terminal length 0\n' `