CRAYSAT-1706: To stop waiting if BOS session is deleted

annapoorna-s-alt commented 2 months ago

Summary and Scope

Stop waiting if BOS session is deleted If a BOS session is deleted when sat bootsys is running, then the bootsys session should end early instead of repeatedly showing a 404 error message.

Issues and Related PRs

CRAYSAT-1706

Testing

List the environments in which these changes were tested.

Tested on:

drax

Test description:

Start the BOS session with sat bootsys boot/shutdown/reboot --stage bos-operations and then delete the BOS session while it's in progress. It should exit early instead of repeatedly showing a 404 error message.

Risks and Mitigations

Low risk

Pull Request Checklist

[x] Version number(s) incremented, if applicable
[x] Copyrights updated
[x] License file intact
[x] Target branch correct
[x] CHANGELOG.md updated
[x] Testing is appropriate and complete, if applicable
[ ] HPC Product Announcement prepared, if applicable

annapoorna-s-alt commented 2 months ago

Here is the latest output

annapoorna-s-alt commented 2 months ago

Final output on drax

haasken-hpe commented 2 months ago

Testing output looks good, but there appears to be some strange blank spaces introduced in the output. I assume that is due to a copy-paste issue.

If you make further changes, I think it's acceptable to test with just one type of action (e.g. shutdown, boot, or reboot) rather than all three to save some time. I think the code path is essentially the same in SAT. It creates a session in BOS and then waits on the session to be done.

annapoorna-s-alt commented 2 months ago

Tested on drax with shutdown of managed nodes

ncn-m001:~/sann # sat bootsys shutdown --stage bos-operations --bos-shutdown-timeout 1800 --bos-templates $SESSION_TEMPLATES
Proceed with shutdown of nodes using BOS? [yes,no] yes
Proceeding with shutdown of nodes using BOS.
INFO: Using session templates provided by --bos-templates/bos_templates option: ['compute-24.3.1.x86_64-csm-1.5.2-final', 'uan-24.3.1.x86_64-csm-1.5.2-final']
INFO: Starting shutdown operation on BOS session templates: compute-24.3.1.x86_64-csm-1.5.2-final, uan-24.3.1.x86_64-csm-1.5.2-final.
INFO: Waiting up to 1800 seconds for sessions to complete.
INFO: Waiting for BOS session 2d79d94d-c6f3-4783-ba55-585f4bee7228 to reach target state complete. Session template: uan-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session 2d79d94d-c6f3-4783-ba55-585f4bee7228 to reach target state complete. Session template: uan-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
ERROR: Could not wait for condition "session 2d79d94d-c6f3-4783-ba55-585f4bee7228 reached target state complete": Failed to query session status: Session 2d79d94d-c6f3-4783-ba55-585f4bee7228 does not exist.
ERROR: Operation 'shutdown' failed on BOS session template 'uan-24.3.1.x86_64-csm-1.5.2-final': BOS session with id 2d79d94d-c6f3-4783-ba55-585f4bee7228 and session template uan-24.3.1.x86_64-csm-1.5.2-final failed.
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Still waiting on session(s) for template(s): compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Waiting for BOS session faaa9c2a-7574-488a-adbb-d537dfc42dc1 to reach target state complete. Session template: compute-24.3.1.x86_64-csm-1.5.2-final
INFO: Session faaa9c2a-7574-488a-adbb-d537dfc42dc1: 100.00% components succeeded, 0.00% components failed
INFO: Shutdown with BOS session template compute-24.3.1.x86_64-csm-1.5.2-final completed.
INFO: Session faaa9c2a-7574-488a-adbb-d537dfc42dc1: 100.00% components succeeded, 0.00% components failed
INFO: Session 2d79d94d-c6f3-4783-ba55-585f4bee7228: 0.00% components succeeded, 0.00% components failed
ERROR: Shutdown failed or timed out for session template: uan-24.3.1.x86_64-csm-1.5.2-final

Cray-HPE / sat