search

CiscoTestAutomation / unicon.plugins

pyATS Unicon Plugins
https://developer.cisco.com/pyats/
Apache License 2.0
22 stars 46 forks source link

Reload is inconsistent #72

Open sngx13 opened 1 year ago

sngx13 commented 1 year ago

Sending e.g device.reload() doesn't always produce expected result. It's sitting and waiting for something at the reload confirm prompt and eventually timeouts and prompt recovery kicks in which effectively sends a new line and allows for reload to proceed.

YAML:

test_bed = """
devices:
  $HOSTNAME:
    os: iosxe
    platform: iosxe
    type: router
    credentials:
      default:
        username: $USERNAME
        password: $PASSWORD
      terminal_server:
        username: $USERNAME
        password: $PASSWORD
      fallback:
        username: cisco
        password: cisco
    connections:
      a:
        protocol: telnet
        ip: $CONSOLE_IP
        port: $CONSOLE_PORT
        login_creds: [terminal_server, default, fallback]
        arguments:
          connection_timeout: 120
          mit: True
          cred_action:
            terminal_server:
              post: sendline()
        settings:
          SENDLINE_AFTER_CRED: terminal_server
          ESCAPE_CHAR_CHATTY_TERM_WAIT: 0.5
          ESCAPE_CHAR_PROMPT_WAIT: 1
          ESCAPE_CHAR_CHATTY_TERM_WAIT_RETRIES: 15
          EXEC_TIMEOUT: 30
          CONFIG_TIMEOUT: 30
          CONFIG_TRANSITION_WAIT: 0.5
          RELOAD_TIMEOUT: 420
          RELOAD_WAIT: 300
          POST_RELOAD_WAIT: 60
          RELOAD_RECONNECT_ATTEMPTS: 5
          SLEEP_PRE_LAUNCH: 0.2
          POST_DISCONNECT_WAIT_SEC: 0
          GRACEFUL_DISCONNECT_WAIT_SEC: 0.2
          CONSOLE_TIMEOUT: 120
"""

LOG output:

2022-11-28 11:47:58,156: %UNICON-INFO: +++ DUT-C1117-4PM logfile /opt/autom8_app/autom8_app/logs/pyats/DUT-C1117-4PM.log +++

2022-11-28 11:47:58,156: %UNICON-INFO: +++ Unicon plugin iosxe (unicon.plugins.iosxe) +++
Trying 80.194.79.121...

2022-11-28 11:47:58,169: %UNICON-INFO: +++ connection to spawn: telnet 80.194.79.121 10014, id: 140694475231472 +++

2022-11-28 11:47:58,169: %UNICON-INFO: connection to DUT-C1117-4PM
Connected to 80.194.79.121.
Escape character is '^]'.

Login: automation
Password:

2022-11-28 11:47:58,282: %UNICON-INFO: Executing post credential command: sendline()

DUT-C1117-4PM#

2022-11-28 11:47:58,396: %UNICON-INFO: Learned hostname(s): 'DUT-C1117-4PM'.

2022-11-28 11:47:58,460: %UNICON-INFO: +++ DUT-C1117-4PM with via 'a': enable +++

2022-11-28 11:47:58,526: %UNICON-INFO: +++ DUT-C1117-4PM with via 'a': executing command 'write erase' +++
write erase
************************************************************************************************************
Erasing Nvram will not clear license trust code.
************************************************************************************************************
Erasing the nvram filesystem will remove all configuration files! Continue? [confirm]
[OK]
Erase of nvram: complete
DUT-C1117-4PM#

2022-11-28 11:48:00,922: %UNICON-INFO: +++ DUT-C1117-4PM with via 'a': reload +++

2022-11-28 11:48:00,985: %UNICON-INFO: +++ reloading DUT-C1117-4PM  with reload_command 'reload' and timeout is 420 seconds +++
reload

System configuration has been modified. Save? [yes/no]: n
WARNING:
Boot variable either does not exist or buffer is too small
This may impact autoboot of the router. Proceed with caution
Do you wish to proceed with reload anyway[confirm]

2022-11-28 11:55:00,991: %UNICON-WARNING: Timeout of 420 seconds has been reached.
Prompt Recovery has commenced. Total timeout occurs in 100 seconds.

2022-11-28 11:55:00,994: %UNICON-INFO: Sending prompt recovery command: b'\r'

Proceed with reload? [confirm]
2022-11-28 11:55:11,007: %UNICON-INFO: Sending prompt recovery command: b'\x15'

2022-11-28 11:55:21,028: %UNICON-INFO: Sending prompt recovery command: b'\x1a'

2022-11-28 11:55:31,036: %UNICON-INFO: Sending prompt recovery command: b'\r'
Rom image verified correctly
sngx13 commented 1 year ago

Hi, It's been a while, have you managed to reproduce this?

DUT-C1111-4P#

2022-12-19 10:55:23,421: %UNICON-INFO: Learned hostname(s): 'DUT-C1111-4P'.

2022-12-19 10:55:23,485: %UNICON-INFO: +++ DUT-C1111-4P with via 'console': enable +++

2022-12-19 10:55:23,550: %UNICON-INFO: +++ DUT-C1111-4P with via 'console': executing command 'write erase' +++
write erase
************************************************************************************************************
Erasing Nvram will not clear license trust code.
************************************************************************************************************
Erasing the nvram filesystem will remove all configuration files! Continue? [confirm]
[OK]
Erase of nvram: complete
DUT-C1111-4P#

2022-12-19 10:55:25,929: %UNICON-INFO: +++ DUT-C1111-4P with via 'console': reload +++

2022-12-19 10:55:25,994: %UNICON-INFO: +++ reloading DUT-C1111-4P  with reload_command 'reload' and timeout is 300 seconds +++
reload

System configuration has been modified. Save? [yes/no]: n
WARNING:
Boot variable either does not exist or buffer is too small
This may impact autoboot of the router. Proceed with caution
Do you wish to proceed with reload anyway[confirm]

2022-12-19 11:00:26,003: %UNICON-WARNING: Timeout of 300 seconds has been reached.
Prompt Recovery has commenced. Total timeout occurs in 100 seconds.

2022-12-19 11:00:26,007: %UNICON-INFO: Sending prompt recovery command: b'\r'

Proceed with reload? [confirm]

2022-12-19 11:00:36,027: %UNICON-INFO: Sending prompt recovery command: b'\x15'

Getting stuck at [confirm] prompt until recovery kicks in.

One interesting observation though. when the following line appears in syslog, reload is always 100% successful: _%IOSXE_RP_CFG_NOT-3-BOOT_VAR_NOTSET: Either the boot variable does not exist or the buffer is too small.Keep it blank.

2022-12-19 12:27:22,902: %UNICON-INFO: Executing post credential command: sendline()

DUT-C1111-4P#

2022-12-19 12:27:23,195: %UNICON-INFO: Learned hostname(s): 'DUT-C1111-4P'.

2022-12-19 12:27:23,259: %UNICON-INFO: +++ DUT-C1111-4P with via 'console': enable +++

2022-12-19 12:27:23,833: %UNICON-INFO: +++ DUT-C1111-4P with via 'console': executing command 'write erase' +++
write erase
************************************************************************************************************
Erasing Nvram will not clear license trust code.
************************************************************************************************************
Erasing the nvram filesystem will remove all configuration files! Continue? [confirm]
[OK]
Erase of nvram: complete
DUT-C1111-4P#

2022-12-19 12:27:26,241: %UNICON-INFO: +++ DUT-C1111-4P with via 'console': reload +++

2022-12-19 12:27:26,306: %UNICON-INFO: +++ reloading DUT-C1111-4P  with reload_command 'reload' and timeout is 400 seconds +++
reload
WARNING:
Boot variable either does not exist or buffer is too small
This may impact autoboot of the router. Proceed with caution
Do you wish to proceed with reload anyway[confirm]
*Dec 19 12:27:25.919: %SYS-7-NV_BLOCK_INIT: Initialized the geometry of nvram
Proceed with reload? [confirm]

*Dec 19 12:27:27.783: %IOSXE_RP_CFG_NOT-3-BOOT_VAR_NOT_SET: Either the boot variable does not exist or the buffer is too small.Keep it blank.
*Dec 19 12:27:31.050: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.Rom image verified correctly
ketulsuthar commented 1 year ago

Hi,

https://pubhub.devnetcloud.com/media/unicon/docs/user_guide/services/generic_services.html#:~:text=as%20shown%20here.-,execute,-%C2%B6

https://pubhub.devnetcloud.com/media/pyats-getting-started/docs/bestpractices/lib.html#:~:text=smaller%20verifications%20together.-,Dialog,-example

can you handle the Dialog the right way?

above link will help you

thanks

sngx13 commented 1 year ago

Hi, I'm using defaults, shouldn't those work? Thanks

On Mon, 19 Dec 2022, 17:40 ketul, @.***> wrote:

Hi,

https://pubhub.devnetcloud.com/media/unicon/docs/user_guide/services/generic_services.html#:~:text=as%20shown%20here.-,execute,-%C2%B6

https://pubhub.devnetcloud.com/media/pyats-getting-started/docs/bestpractices/lib.html#:~:text=smaller%20verifications%20together.-,Dialog,-example

can you handle the Dialog the right way?

above link will help you

thanks

— Reply to this email directly, view it on GitHub https://github.com/CiscoTestAutomation/unicon.plugins/issues/72#issuecomment-1358014022, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD2CGUE3ROWXXGJWJEGI3QLWOCMYDANCNFSM6AAAAAASNIZTZE . You are receiving this because you authored the thread.Message ID: @.***>

sngx13 commented 1 year ago

Where can i find dialog handling? At least the default ones?

2022-12-20 11:01:10,419: %UNICON-INFO: +++ DUT-C1111-4P with via 'console': reload +++

2022-12-20 11:01:10,482: %UNICON-INFO: +++ reloading DUT-C1111-4P  with reload_command 'reload' and timeout is 420 seconds +++
reload

System configuration has been modified. Save? [yes/no]: n
WARNING:
Boot variable either does not exist or buffer is too small
This may impact autoboot of the router. Proceed with caution
Do you wish to proceed with reload anyway[confirm]

This may be down to the fact that default dialog does not expect the warning message and isn't handling this properly:

WARNING: Boot variable either does not exist or buffer is too small This may impact autoboot of the router. Proceed with caution

Could this be added to next release?

Update Just confirmed my theory, when boot system bootflash:xxxx is added, problem goes away i.e WARNING message no longer appears and Unicon handles it properly. Can we handle this by default as there are instances when people are not setting boot vars but instead delete the old IOS?