Separate pre plan sanity check from upgrade plan

[chanchiwai-ray]

Run pre plan sanity check before plan generation or upgrade step, and stop the plan generation or upgrade if the pre plan sanity check failed.

[chanchiwai-ray]

Need to improve the UI later, maybe new PR?

[chanchiwai-ray]

Example output:

$ cou plan # failed verification
Full execution log: '/home/raychan/.local/share/cou/log/cou-20240916115354.log'  
Connected to 'ceph' ✔
Analyzing cloud... ✔
Verifying cloud... \2024-09-16 11:53:56 [ERROR] 'noout' is expected to be unset for ceph-mon
Verifying cloud... ✔
2024-09-16 11:53:56 [ERROR] 'noout' should be set for ceph cluster
2024-09-16 11:53:56 [ERROR] Units in blocked: ceph-osd/0, ceph-osd/2, ceph-osd/1
2024-09-16 11:53:56 [ERROR] Running upgrades will not be possible until problems indicated in the errors are resolved

$ cou plan # passed verification
Full execution log: '/home/raychan/.local/share/cou/log/cou-20240913174023.log'  
Connected to 'ceph' ✔
Analyzing cloud... ✔
Verifying cloud... ✔
Generating upgrade plan... ✔
Upgrade cloud from 'yoga' to 'zed'
        Back up MySQL databases
        Archive old database data on nova-cloud-controller
        Control Plane principal(s) upgrade plan
                Upgrade plan for 'ceph-mon' to 'zed'
                        Upgrade software packages of 'ceph-mon' from the current APT repositories
                                Ψ Upgrade software packages on unit 'ceph-mon/0'
                        Ensure that the 'require-osd-release' option matches the 'ceph-osd' version
                        Change charm config of 'ceph-mon' 'source' to 'cloud:jammy-zed'
                        Wait for up to 2400s for model 'ceph' to reach the idle state
                        Verify that the workload of 'ceph-mon' has been upgraded on units: ceph-mon/0
        Remaining Data Plane principal(s) upgrade plan
                Upgrade plan for 'ceph-osd' to 'zed'
                        Verify that all 'nova-compute' units has been upgraded
                        Upgrade software packages of 'ceph-osd' from the current APT repositories
                                Ψ Upgrade software packages on unit 'ceph-osd/0'
                                Ψ Upgrade software packages on unit 'ceph-osd/1'
                                Ψ Upgrade software packages on unit 'ceph-osd/2'
                        Change charm config of 'ceph-osd' 'source' to 'cloud:jammy-zed'
                        Wait for up to 300s for app 'ceph-osd' to reach the idle state
                        Verify that the workload of 'ceph-osd' has been upgraded on units: ceph-osd/0, ceph-osd/1, ceph-osd/2
        Ensure ceph-mon's 'require-osd-release' option matches the 'ceph-osd' version

2024-09-13 17:40:24 [WARNING] Setting noout for ceph cluster during upgrade is optional but recommended. You can use `--set-noout` to add this optional step. For more information, Please refer to set-noout action: https://charmhub.io/ceph-mon/actions#set-noout

$ cou upgrade # failed verification
Full execution log: '/home/raychan/.local/share/cou/log/cou-20240916115556.log'  
Connected to 'ceph' ✔
Analyzing cloud... ✔
Verifying cloud... \2024-09-16 11:55:58 [ERROR] 'noout' is expected to be unset for ceph-mon
Verifying cloud... ✔
2024-09-16 11:55:58 [ERROR] 'noout' should be set for ceph cluster
2024-09-16 11:55:58 [ERROR] Units in blocked: ceph-osd/0, ceph-osd/2, ceph-osd/1
2024-09-16 11:55:58 [ERROR] Running upgrades will not be possible until problems indicated in the errors are resolved
Running post upgrade sanity check...
2024-09-16 11:55:59 [ERROR] 'noout' is expected to be unset for ceph-mon
Detected ceph 'noout' set, please unset noout for ceph cluster.
Post upgrade sanity check completed.

[chanchiwai-ray]

rebased