search

ktbyers / net_upgrade

Automate OS Upgrade Process
Apache License 2.0
4 stars 1 forks source link

OS Upgrade Process Automation #1

Open ktbyers opened 7 years ago

ktbyers commented 7 years ago

Disclaimer: this is exploratory at this point (it is TBD whether anything actually gets built)

Starting Step:

Create an abstract process for IOS/IOS-XE, IOS-XR, NX-OS, JunOS, and Arista.

Basically a series of steps of what an OS upgrade for each of these platforms would consist of. Obviously, there will be a lot of variations here depending on what the hardware is.

High level questions:

Is there an auto-detection mechanism(s)? For hardware, for inventory characteristics.

What is the control channel? I am assuming that SSH will be used as the general purpose control channel (at least initially). What part of this can be accomplished via APIs?

What is the transfer mechanism? I am assuming SCP as the standard transfer mechanism. Does there need to be others. If they are others, I would have a pretty strong preference to have them be initiated from the network device (i.e. they are things that just are executed from the SSH control channel).

Pre-transfer requirements (i.e. gathering information pre-transfer). Space available, current config, current image, boot settings.

Post-transfer requirements. MD5 or checksum on file. Is this required/optional, what should the default be?

Mechanics of specifying image to be used.

Reload mechanics. Default will definitely be off here.

Post-transfer requirements (i.e. information gathered post-transfer) and what is done with the information. Are all modules/interfaces still present?

High-availability contexts and what are their requirements. Other platform specific idiosyncrasies.

Any other caveats like ROM requirements, sanity checking if ROM is supported, memory requirements.

Not saying any of these will/should be included, but just specifying some of the issues.

Version sequencing issues (i.e. when you have to upgrade through intermediate versions).

Similar issues for ROM/BIOS/FPGA/memory requirements.

Probably need some sort of generic requirements.yml file that will state the requirements for upgrading from version a to version b that the engineer can specify (somehow).

Other items:

Version sequencing issues (i.e. when you have to upgrade through intermediate versions).

Similar issues for ROM/BIOS/FPGA/memory requirements.

Probably need some sort of generic requirements.yml file that will state the requirements for upgrading from version a to version b that the engineer can specify (somehow).

Fallback transfer mechanism.

Mechanism to delete files to free up space.

ktbyers commented 7 years ago

See previous issue here:

https://github.com/ktbyers/netmiko/issues/592

ktbyers commented 7 years ago

Review what already exists:

Review what ntc-install-os already does:

https://github.com/networktocode/ntc-ansible/blob/master/library/ntc_install_os.py

And here:

https://github.com/networktocode/pyntc/tree/master/pyntc/devices

Also look if any other existing Python libraries out there that do this.

Other things to look at:

http://anastarsha.com/automating-cisco-device-upgrades-with-ansible/

IOS-XR specific:

http://www.networkhints.com/2014/12/upgrading-ios-xr-with-python-cisco-aut.html

https://networklore.com/ansible-ios-upgrade/

Look into this more:

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/pnp/configuration/15-mt/pnp-15-mt-book.html

Juniper Ansible Module for this.

ktbyers commented 7 years ago

General Process

Platforms to support

IOS/IOS-XE NX-OS IOS-XR Junos Arista ASA

Pre File Transfer

  1. Checks/Information Gathering/Archive Current Config Things that could possibly be checked: a. Checks should be modular (i.e. people could add/augment with additional checks easily) b. Check Current Image / MD5-Checksum-File Size c. Hardware Checks (what is the hardware) d. Memory Size e. File System (is space available). f. Nothing will be done to free space (at least initially) g. Interim versions (would just abort if an interim upgrade was required)

Transfer File

  1. Use an SSH control channel (telnet supported to extent supported by Netmiko)
  2. Use SCP by default (from server to client)
  3. Add options for additional transfer mechanism. a. These will all be driven from the network device using the control channel. b. Possibilities: FTP, HTTP/HTTPS, TFTP c. Think about making this modular again so people can easily augment to transfer in a different way.

Set Boot Variable (or equivalent)

  1. Use SSH control channel
  2. Any other pre-reboot verifications

Reboot

  1. Will have to be invoked separately / will never be called automatically

Post reload verification

  1. New image running
  2. Any config changes
  3. Other verifications.
bwks commented 7 years ago

Some notes on Juniper upgrades

EX series NSSU

Assuming that the Juniper Virtual Chassis stack has been deployed as per standard template.

Minimum requirements:

MX series NSSU

Minimum requirements, assuming redundant RE

nickrusso42518 commented 7 years ago

On the topic of "other verifications", would add a comment that it would be good to include licensing changes, especially invalidations. Maybe a gather function pre-upgrade, and a check against those values post-upgrade.