cdot65 / pan-os-upgrade

An efficient tool to execute configuration backups, network state snapshots, system readiness checks, and operating system upgrades of Palo Alto Networks firewalls and Panorama appliances.
https://cdot65.github.io/pan-os-upgrade/
Apache License 2.0
39 stars 7 forks source link

Implement disk RAID check before upgrades on larger platforms #127

Open cdot65 opened 4 months ago

cdot65 commented 4 months ago

Is your feature request related to a problem? Please describe.

When upgrading PAN-OS on large firewalls with RAID storage using the pan-os-upgrade utility, reboot timeouts can occur due to the extended duration of the reboot process, which can take up to an hour. This issue arises when the upgraded firewall undergoes a RAID check during the reboot, causing significant delays. Currently, the utility does not have a mechanism to detect and handle RAID checks before initiating the upgrade process, leading to potential timeouts and incomplete upgrades.

Describe the solution you'd like

Enhance the pan-os-upgrade utility to include disk and RAID checks before performing an upgrade on firewalls with RAID storage. The utility should:

  1. Before initiating the upgrade, run the show system raid command on the firewall to retrieve the RAID status and configuration.
  2. Parse the output of the command and check the counter since the last RAID check took place.
  3. If the counter indicates that a RAID check is overdue or likely to occur during the upgrade process, display a warning message to the user and prompt them to either proceed with the upgrade or abort the process.
  4. Implement a log parsing function to search through the firewall logs using grep or a similar method to determine if the upgraded firewall is currently undergoing a RAID check.
  5. If a RAID check is detected during the upgrade process, remove the peer firewall (if applicable) from the upgrade "revisit" list to avoid initiating an upgrade while the RAID check is in progress.
  6. If no RAID check is detected or if the RAID check completes successfully, proceed with the normal upgrade process.

Describe alternatives you've considered

An alternative approach could be to force the RAID check to occur before the upgrade by triggering it manually. However, this might not always be feasible or desirable, as it could lead to extended downtime and may not align with the planned maintenance window.

Additional context

Here are a few additional points to consider:

By implementing this feature, the pan-os-upgrade utility will proactively detect and handle scenarios where RAID checks may interfere with the upgrade process on firewalls with RAID storage. This will help avoid reboot timeouts, ensure successful upgrades, and provide a smoother experience for users managing large-scale firewall upgrades.