search

ktbyers / netmiko

Multi-vendor library to simplify Paramiko SSH connections to network devices
MIT License
3.6k stars 1.31k forks source link

Handling prompts in send_command() output #2103

Closed mohitbhardwaj5659 closed 2 years ago

mohitbhardwaj5659 commented 3 years ago

Hi Kirk ,

I am trying to send the below command to the nexus 9k switch Command

install all nxos bootflash:///nxos.7.0.3.I5.2.bin Output

Installer will perform compatibility check first. Please wait. Installer is forced disruptive It stucks here and do not show anything .

I am using the same command manually on the switch and i am getting the result

Installer will perform compatibility check first. Please wait. Installer is forced disruptive

Verifying image bootflash:/nxos.7.0.3.I5.2.bin for boot variable "nxos". [####################] 100% -- SUCCESS

Verifying image type. [####################] 100% -- SUCCESS

Preparing "nxos" version info using image bootflash:/nxos.7.0.3.I5.2.bin. [####################] 100% -- SUCCESS

Preparing "bios" version info using image bootflash:/nxos.7.0.3.I5.2.bin. [####################] 100% -- SUCCESS

I am using the below function to send the command. Please suggest if any parameter need to be provided.

def install_and_reload(self,client,net_connect): ''' reloading the switch with the image. ''' try: logger.info("installing image and reloading the device %s",client['host']) install_reload="install all nxos bootflash:///{}".format(self.image) print(install_reload) installing_and_reloading=net_connect.send_command_timing(install_reload,strip_prompt= False,strip_command = False) print(installing_and_reloading) except Exception as error: logger.error("Error in reloading the switch",error) sys.exit(0)

ktbyers commented 3 years ago

In Netmiko 3.3.2 for NX-OS, you need to set both:

fast_cli = False
global_delay_factor = X      # where X is your multiplier

Regards,

Kirk

mohitbhardwaj5659 commented 3 years ago

Hi Kirk,

I have tried with the above recommendation but is still throwing this errror. install all nxos bootflash:///nxos.7.0.3.I5.2.bin --- Logging error --- Traceback (most recent call last): File "/home/mohit/NORNIR_POC/nornir-network-automation/upgrade_with_netmiko/nexus_upgrade_final.py", line 278, in install_and_reload installing_and_reloading=net_connect.send_command(install_reload) File "/usr/local/lib/python3.6/site-packages/netmiko/utilities.py", line 429, in wrapper_decorator return func(self, *args, **kwargs) File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 1525, in send_command search_pattern.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib64/python3.6/logging/init.py", line 994, in emit msg = self.format(record) File "/usr/lib64/python3.6/logging/init.py", line 840, in format return fmt.format(record) File "/usr/lib64/python3.6/logging/init.py", line 577, in format record.message = record.getMessage() File "/usr/lib64/python3.6/logging/init.py", line 338, in getMessage msg = msg % self.args TypeError: not all arguments converted during string formatting Call stack: File "calling_nexus_upgrade.py", line 109, in nexus_upgarde() File "calling_nexus_upgrade.py", line 104, in nexus_upgarde after_upgrade(creating_object) File "calling_nexus_upgrade.py", line 77, in after_upgrade creating_object.install_and_reload(item,net_connect) File "/home/mohit/NORNIR_POC/nornir-network-automation/upgrade_with_netmiko/nexus_upgrade_final.py", line 281, in install_and_reload logger.error("Error in reloading the switch",error) Message: 'Error in reloading the switch' Arguments: (OSError('Search pattern never detected in send_command: ODIN\-S1\#',),)

I have added that in the connecthandler and it is still throwing me the error.

    def Connect(self):
            device_list=[]
            connection =''
            for i in range(0,len(self.IP_ADDRESS)):
                    connection="connection_{}".format(i)
                    #print(connection)
                    connection= {
                       "device_type": self.device_type,
                       "host": self.IP_ADDRESS[i],
                       "username":self.username,
                       "password":self.password,
                       "global_delay_factor": 4,
                       "fast_cli": False
                         }
                    device_list.append(connection)
            return device_list
ktbyers commented 3 years ago

What command are you sending in send_comand() and how long are you expecting that command to take (to complete)?

It looks like you re trying to reload the switch, you probably should use send_command_timing() to reload the switch. Also does the reload operation prompt you here? If so, you will need to handle the prompting.

Regards, Kirk

mohitbhardwaj5659 commented 3 years ago

This is the ouput which i have ran manually on nexus 9k.

ODIN-S1# install all nxos bootflash:///nxos.7.0.3.I5.2.bin Installer will perform compatibility check first. Please wait. Installer is forced disruptive

Verifying image bootflash:/nxos.7.0.3.I5.2.bin for boot variable "nxos". [####################] 100% -- SUCCESS

Verifying image type. [####################] 100% -- SUCCESS

Preparing "nxos" version info using image bootflash:/nxos.7.0.3.I5.2.bin. [####################] 100% -- SUCCESS

Preparing "bios" version info using image bootflash:/nxos.7.0.3.I5.2.bin. [####################] 100% -- SUCCESS

Warning: the running-config contains configuration that is incompatible with the new image and will be lost (loose incompatibility). Please run 'show incompatibility-all nxos ' command for more details . [####################] 100% -- SUCCESS

Notifying services about system upgrade. [####################] 100% -- SUCCESS

Compatibility check is done: Module bootable Impact Install-type Reason

 1       yes      disruptive         reset  Incompatible image for ISSU
27       yes      disruptive         reset  Incompatible image for ISSU

Images will be upgraded according to following table: Module Image Running-Version(pri:alt) New-Version Upg-Required

 1       lcn9k                                    9.3(4)           7.0(3)I5(2)           yes
27        nxos                                    9.3(4)           7.0(3)I5(2)           yes
27        bios     v07.67(01/30/2020):v07.34(08/11/2015)    v07.59(08/26/2016)            no

Switch will be reloaded for disruptive upgrade. Do you want to continue with the installation (y/n)? [n] n

When I ran the command through the code it is getting stuck here and bypasses rest of the step and throws no error .

install all nxos bootflash:///nxos.7.0.3.I5.2.bin Installer will perform compatibility check first. Please wait. Installer is forced disruptive

Verifying image bootflash:/nxos.7.0.3.I5.2.bin for boot variable "nxos". [# ] 0%

def install_and_reload(self,client,net_connect): ''' reloading the switch with the image. ''' try: logger.info("installing image and reloading the device %s",client['host']) install_reload="install all nxos bootflash:///{}".format(self.image) print(install_reload) installing_and_reloading=net_connect.send_command_timing(install_reload) print(installing_and_reloading) except Exception as error: logger.error("Error in reloading the switch",error) sys.exit(0)

i Will handle the prompt with the expect string parameter but it is not going till that point.

mohitbhardwaj5659 commented 3 years ago

This is the debug ouput which i have found from the test log. Hope that helps in narrowing down the problem.

2021-01-11 11:12:50,372 DEBUG starting thread (client mode): 0x34796128 2021-01-11 11:12:50,373 DEBUG Local version/idstring: SSH-2.0-paramiko_2.7.2 2021-01-11 11:12:50,503 DEBUG Remote version/idstring: SSH-2.0-OpenSSH_7.5 PKIX[10.1] 2021-01-11 11:12:50,504 INFO Connected (version 2.0, client OpenSSH_7.5) 2021-01-11 11:12:50,507 DEBUG Kex agreed: ecdh-sha2-nistp256 2021-01-11 11:12:50,507 DEBUG HostKey agreed: ssh-rsa 2021-01-11 11:12:50,507 DEBUG Cipher agreed: aes128-ctr 2021-01-11 11:12:50,507 DEBUG MAC agreed: hmac-sha2-256 2021-01-11 11:12:50,507 DEBUG Compression agreed: none 2021-01-11 11:12:50,518 DEBUG kex engine KexNistp256 specified hash_algo 2021-01-11 11:12:50,519 DEBUG Switch to new keys ... 2021-01-11 11:12:50,520 DEBUG Adding ssh-rsa host key for 10.192.103.81: b'1cf0b95486e9a0d1a4ac9f246c63f2f1' 2021-01-11 11:12:50,623 DEBUG userauth is OK 2021-01-11 11:12:50,642 INFO Auth banner: b'User Access Verification\n' 2021-01-11 11:12:50,642 INFO Authentication (password) successful! 2021-01-11 11:12:50,643 DEBUG [chan 0] Max packet in: 32768 bytes 2021-01-11 11:12:50,649 DEBUG Received global request "hostkeys-00@openssh.com" 2021-01-11 11:12:50,649 DEBUG Rejecting "hostkeys-00@openssh.com" global request from server. 2021-01-11 11:12:50,690 DEBUG [chan 0] Max packet out: 32768 bytes 2021-01-11 11:12:50,690 DEBUG Secsh channel 0 opened. 2021-01-11 11:12:50,692 DEBUG [chan 0] Sesch channel 0 request ok 2021-01-11 11:12:50,695 DEBUG [chan 0] Sesch channel 0 request ok 2021-01-11 11:12:55,104 DEBUG read_channel: ^M Cisco Nexus Operating System (NX-OS) Software^M TAC support: http://www.cisco.com/tac^M Copyright (C) 2002-2020, Cisco and/or its affiliates.^M All rights reserved.^M The copyrights to certain works contained in this software are^M owned by other third parties and used and distributed under their own^M licenses, such as open source. This software is provided "as is," and unless^M otherwise stated, there is no warranty, express or implied, including but not^M limited to warranties of merchantability and fitness for a particular purpose.^M Certain components of this software are licensed under^M the GNU General Public License (GPL) version 2.0 or ^M GNU General Public License (GPL) version 3.0 or the GNU^M Lesser General Public License (LGPL) Version 2.1 or ^M Lesser General Public License (LGPL) Version 2.0. ^M A copy of each such license is available at^M http://www.opensource.org/licenses/gpl-2.0.php and^M http://opensource.org/licenses/gpl-3.0.html and^M http://www.opensource.org/licenses/lgpl-2.1.php and^M http://www.gnu.org/licenses/old-licenses/library.txt.^M ^MODIN-S1# 2021-01-11 11:12:55,505 DEBUG read_channel: 2021-01-11 11:13:03,513 DEBUG read_channel: 2021-01-11 11:13:03,514 DEBUG write_channel: b'terminal width 511\n' 2021-01-11 11:13:03,514 DEBUG Pattern is: terminal width 511 2021-01-11 11:13:03,516 DEBUG _read_channel_expect read_data: terminal width 511^M^M

2021-01-11 11:13:03,516 DEBUG Pattern found: terminal width 511 terminal width 511^M^M

2021-01-11 11:13:03,517 DEBUG In disable_paging 2021-01-11 11:13:03,517 DEBUG Command: terminal length 0

2021-01-11 11:13:03,517 DEBUG write_channel: b'terminal length 0\n' 2021-01-11 11:13:03,517 DEBUG Pattern is: terminal\ length\ 0 2021-01-11 11:13:03,519 DEBUG _read_channel_expect read_data: terminal length 0^M

2021-01-11 11:13:03,519 DEBUG Pattern found: terminal\ length\ 0 terminal length 0^M

2021-01-11 11:13:03,519 DEBUG terminal length 0^M

2021-01-11 11:13:03,519 DEBUG Exiting disable_paging 2021-01-11 11:13:03,920 DEBUG read_channel: ^MODIN-S1# terminal length 0^M^M ^MODIN-S1# 2021-01-11 11:13:03,920 DEBUG Clear buffer detects data in the channel 2021-01-11 11:13:04,722 DEBUG read_channel: 2021-01-11 11:13:04,722 DEBUG write_channel: b'\n' 2021-01-11 11:13:05,123 DEBUG read_channel: ^M^M ^MODIN-S1# 2021-01-11 11:13:05,925 DEBUG read_channel: 2021-01-11 11:13:05,925 DEBUG [find_prompt()]: prompt is ODIN-S1# 2021-01-11 11:13:05,925 INFO connecting to the device 10.192.103.81 2021-01-11 11:13:05,926 INFO installing image and reloading the device 10.192.103.81 2021-01-11 11:13:05,927 DEBUG write_channel: b'install all nxos bootflash:///nxos.7.0.3.I5.2.bin\n' 2021-01-11 11:13:05,927 DEBUG send_command_timing current output: 2021-01-11 11:13:06,328 DEBUG read_channel: install all nxos bootflash:///nxos.7.0.3.I5.2.bin^M^M Installer will perform compatibility check first. Please wait. ^M Installer is forced disruptive^M

2021-01-11 11:13:06,728 DEBUG read_channel: 2021-01-11 11:13:14,736 DEBUG read_channel: ^M Verifying image bootflash:/nxos.7.0.3.I5.2.bin for boot variable "nxos".^M [# ] 0% 2021-01-11 11:13:15,137 DEBUG read_channel: 2021-01-11 11:13:23,145 DEBUG read_channel: 2021-01-11 11:13:23,145 DEBUG send_command_timing final output: Installer will perform compatibility check first. Please wait. Installer is forced disruptive

Verifying image bootflash:/nxos.7.0.3.I5.2.bin for boot variable "nxos". [# ] 0%

ktbyers commented 3 years ago

There are two items you need to handle here:

  1. The amount of time to wait for a given operation to complete (before Netmiko gives up). Netmiko will wait 100 seconds if fast_cli=False. You can multiply this by 2, 4, 8, et cetera by setting either global_delay_factor or delay_factor. So you need to make sure you have allocated enough time here.
  2. You need to handle (in your code) any prompting that occurs. For example:
Switch will be reloaded for disruptive upgrade.
Do you want to continue with the installation (y/n)? [n] n

Here is an example of handling prompting using the expect_string argument.

Regards,

Kirk

mohitbhardwaj5659 commented 3 years ago

I have set the global delay factor to 4 in the connecthandler creation paramter to the install and check hardware compatibility function.

def Connect(self): device_list=[] connection ='' for i in range(0,len(self.IPADDRESS)): connection="connection{}".format(i)

print(connection)

                    connection= {
                       "device_type": self.device_type,
                       "host": self.IP_ADDRESS[i],
                       "username":self.username,
                       "password":self.password,
                       "global_delay_factor": 4,
                       "fast_cli": False
                         }
                    device_list.append(connection)

I am passing the net_connet parameter to the below function with the global_delay_factor set but it is still failing.

def install_and_reload(self,client,net_connect): ''' reloading the switch with the image. ''' try: logger.info("installing image and reloading the device %s",client['host']) install_reload="install all nxos bootflash:///{}".format(self.image) print(install_reload) installing_and_reloading=net_connect.send_command_timing(install_reload) print(installing_and_reloading)

if "Do you want to continue with the installation (y/n)" in installing_and_reloading:

                    #       installing_and_reloading += net_connect.send_command_timing("\n")
                    #       #print(installing_and_reloading)
                    #else:
                    #       print("################################# FAIL TO INSTALL AND RELOAD##############")
                    print("#########################CHECKING INSTALL AND RELOAD######################")
            except Exception as error:
                    logger.error("Error in reloading the switch",error)
                    sys.exit(0)

One thing to note i have also use checking hardware compatibilities function and it is showing me the correct result with the same set of parameter.

    def checking_hardware_compatibilities(self,client,net_connect):
            '''
            checking and writing whether software upgrade is disruptive or non-disruptive.
            '''
            try:
                    logger.info("checking hardware_compatibilities for switch %s",client['host'])
                    install_capabilities="show install all impact nxos bootflash:///"+str(self.image)
                    print(install_capabilities)
                    checking_install_output=net_connect.send_command(install_capabilities)
                    #print(net_connect.find_prompt())
                    print(checking_install_output)

                    pattern = re.compile("non-disruptive")
                    result = pattern.findall('non-disruptive')
                    if result[0] == "non-disruptive":
                            logging.info("Switch with ip %s will perform a non disruptive upgrade",client['host'])
                    else:
                            logging.info("Switch with ip %s will perform a disruptive upgrade",client['host'])

                    file_checking_install_output=open("hardware_compatibility_result.txt"
                                                           +str(client['host']),"w")
                    file_checking_install_output.write(checking_install_output)
                    file_checking_install_output.close()
                    time.sleep(2)
            except Exception as error:
                    logger.error("not able to perform hardware compatibility check exiting the upgrade process",error)
                    sys.exit(0)
mohitbhardwaj5659 commented 3 years ago

Hello Kirk,

Thanks for your guidance.

Somehow i have managed to run the install command in which was stuck . now i need to handle the prompt for continuation of the upgrade process.

I have used the send_command_timing function to send the install command that doesnt have "expect_string" parameter in that . But for handling the prompt i am using net_connect.send_command() but it is throwing me the error which i have pasted below

function

def install_and_reload(self,client,net_connect):
    '''
    reloading the switch with the image.
    '''
    try:
        logger.info("installing image and reloading the device %s",client['host'])
        install_reload="install all nxos bootflash:///"+str(self.image)
        print(install_reload)
        installing_and_reloading=net_connect.send_command_timing(install_reload)
        installing_and_reloading += net_connect.send_command_timing("\n")
        splitting = installing_and_reloading.split("\n")
        print(splitting)
        if splitting [-2] == "Do you want to continue with the installation (y/n)?  [n] ":
            logger.info("Reloading the device %s",client['host'])
            installing_and_reloading += net_connect.send_command("y")
            installing_and_reloading += net_connect.send_command("\n")
            print(installing_and_reloading)
        else:
            logger.info("FAIL TO RELOAD")
    except Exception as error:
        logger.error("Error in reloading the switch",error)
        sys.exit(0)

Error

2021-01-12 10:32:06,846 INFO Reloading the device 10.192.103.82 Installer will perform compatibility check first. Please wait. Installer is forced disruptive

Verifying image bootflash:/nxos.7.0.3.I5.2.bin for boot variable "nxos". [# ] 0%[####################] 100% -- SUCCESS

Verifying image type. [# ] 0%[####################] 100% -- SUCCESS

Preparing "nxos" version info using image bootflash:/nxos.7.0.3.I5.2.bin. [# ] 0%[####################] 100% -- SUCCESS

Preparing "bios" version info using image bootflash:/nxos.7.0.3.I5.2.bin. [# ] 0%[####################] 100% -- SUCCESS

Warning: the running-config contains configuration that is incompatible with the new image and will be lost (loose incompatibility). Please run 'show incompatibility-all nxos ' command for more details . [# ] 0%[####################] 100% -- SUCCESS

Performing module support checks. [####################] 100% -- SUCCESS

Notifying services about system upgrade. [####################] 100% -- SUCCESS

Compatibility check is done: Module bootable Impact Install-type Reason

 1       yes      disruptive         reset  Incompatible image for ISSU

Images will be upgraded according to following table: Module Image Running-Version(pri:alt) New-Version Upg-Required

 1        nxos                                    9.3(4)           7.0(3)I5(2)           yes
 1        bios     v07.59(08/26/2016):v07.34(08/11/2015)    v07.59(08/26/2016)            no

Switch will be reloaded for disruptive upgrade.

Do you want to continue with the installation (y/n)? [n] ^ % Invalid command at '^' marker.

I need to handle this prompt.

****Do you want to continue with the installation (y/n)? [n]

mohitbhardwaj5659 commented 3 years ago

Hello Kirk ,

Can you please help me handling the prompt .

I am getting stuck while sending the below command. installing_and_reloading += net_connect.send_command("y")

Error Do you want to continue with the installation (y/n)? [n] n ^ % Invalid command at '^' marker.

Vaibhav

ktbyers commented 3 years ago

Here are some examples on doing that:

https://github.com/ktbyers/netmiko/blob/develop/EXAMPLES.md#handling-commands-that-prompt-expect_string

And

https://github.com/ktbyers/netmiko/blob/develop/EXAMPLES.md#handling-commands-that-prompt-timing

Regards, Kirk

panli200 commented 2 years ago

Hi Kirk : Some command has dynamic output. I have an issue that I want print the output of running a command, however, I only get the beginning of output. Can you tell me how to get all output? Thank you ahead!

aramidetosin commented 7 months ago

@mohitbhardwaj5659 how did you end up fixing this ?

adityamum commented 6 months ago

Hi, even I am passing global_delay_factor as well as fast_cli ..my output is not getting completed. but when I pass read_timout to 10000 in send_command its giving whole output but I am unable to send "y" when the user prompt comes..pls help

ktbyers commented 6 months ago

@adityamum Please open a new issue and include the relevant section of your code and the issue that you are seeing.