✨Clusters-keeper: terminate broken EC2s🚨

sanderegg commented 4 months ago

What do these changes do?

[x] cluster heartbeats are set only after the cluster effectively started, and PRIMARY_EC2_INSTANCES_MAX_START_TIME keeps now a hard-coded 2minutes value. not an ENV for now at least.
[x] if a started primary machine does not respond after PRIMARY_EC2_INSTANCES_MAX_START_TIME it will be terminated with a warning in the logs, if the dv-2 still wants a machine a new one will then be started
[x] if a primary machine suddenly stops responding (e.g. the dask-scheduler is somehow broken), it will be removed with its workers after 5 minutes.
[x] some refactoring and missing tests

🚨Needs some testing in master BEFORE it goes anywhere else. as this is an AWS only feature.

Related issue/s

fixes https://github.com/ITISFoundation/osparc-simcore/issues/5436

Dev-ops checklist
- [x] No ENV changes or I properly updated ENV (read the instruction)

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 88.2%. Comparing base (cafbf96) to head (b95b64c). Report is 221 commits behind head on master.

Additional details and impacted files

[![Impacted file tree graph](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851/graphs/tree.svg?width=650&height=150&src=pr&token=h1rOE8q7ic&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation)](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) ```diff @@ Coverage Diff @@ ## master #5851 +/- ## ========================================= + Coverage 84.5% 88.2% +3.6% ========================================= Files 10 974 +964 Lines 214 43820 +43606 Branches 25 234 +209 ========================================= + Hits 181 38652 +38471 - Misses 23 5119 +5096 - Partials 10 49 +39 ``` | [Flag](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [integrationtests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `65.1% <ø> (?)` | | | [unittests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `85.6% <100.0%> (+1.0%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [packages/aws-library/src/aws\_library/ec2/client.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851?src=pr&el=tree&filepath=packages%2Faws-library%2Fsrc%2Faws_library%2Fec2%2Fclient.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-cGFja2FnZXMvYXdzLWxpYnJhcnkvc3JjL2F3c19saWJyYXJ5L2VjMi9jbGllbnQucHk=) | `100.0% <100.0%> (ø)` | | | [packages/aws-library/src/aws\_library/ec2/models.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851?src=pr&el=tree&filepath=packages%2Faws-library%2Fsrc%2Faws_library%2Fec2%2Fmodels.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-cGFja2FnZXMvYXdzLWxpYnJhcnkvc3JjL2F3c19saWJyYXJ5L2VjMi9tb2RlbHMucHk=) | `88.8% <100.0%> (ø)` | | | [...c/simcore\_service\_clusters\_keeper/core/settings.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851?src=pr&el=tree&filepath=services%2Fclusters-keeper%2Fsrc%2Fsimcore_service_clusters_keeper%2Fcore%2Fsettings.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvY2x1c3RlcnMta2VlcGVyL3NyYy9zaW1jb3JlX3NlcnZpY2VfY2x1c3RlcnNfa2VlcGVyL2NvcmUvc2V0dGluZ3MucHk=) | `100.0% <100.0%> (ø)` | | | [...imcore\_service\_clusters\_keeper/modules/clusters.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851?src=pr&el=tree&filepath=services%2Fclusters-keeper%2Fsrc%2Fsimcore_service_clusters_keeper%2Fmodules%2Fclusters.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvY2x1c3RlcnMta2VlcGVyL3NyYy9zaW1jb3JlX3NlcnZpY2VfY2x1c3RlcnNfa2VlcGVyL21vZHVsZXMvY2x1c3RlcnMucHk=) | `100.0% <100.0%> (ø)` | | | [...lusters\_keeper/modules/clusters\_management\_core.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851?src=pr&el=tree&filepath=services%2Fclusters-keeper%2Fsrc%2Fsimcore_service_clusters_keeper%2Fmodules%2Fclusters_management_core.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvY2x1c3RlcnMta2VlcGVyL3NyYy9zaW1jb3JlX3NlcnZpY2VfY2x1c3RlcnNfa2VlcGVyL21vZHVsZXMvY2x1c3RlcnNfbWFuYWdlbWVudF9jb3JlLnB5) | `100.0% <100.0%> (ø)` | | | [...rc/simcore\_service\_clusters\_keeper/modules/dask.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851?src=pr&el=tree&filepath=services%2Fclusters-keeper%2Fsrc%2Fsimcore_service_clusters_keeper%2Fmodules%2Fdask.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvY2x1c3RlcnMta2VlcGVyL3NyYy9zaW1jb3JlX3NlcnZpY2VfY2x1c3RlcnNfa2VlcGVyL21vZHVsZXMvZGFzay5weQ==) | `97.1% <100.0%> (ø)` | | | [...rc/simcore\_service\_clusters\_keeper/rpc/clusters.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851?src=pr&el=tree&filepath=services%2Fclusters-keeper%2Fsrc%2Fsimcore_service_clusters_keeper%2Frpc%2Fclusters.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvY2x1c3RlcnMta2VlcGVyL3NyYy9zaW1jb3JlX3NlcnZpY2VfY2x1c3RlcnNfa2VlcGVyL3JwYy9jbHVzdGVycy5weQ==) | `100.0% <100.0%> (ø)` | | | [.../simcore\_service\_clusters\_keeper/utils/clusters.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851?src=pr&el=tree&filepath=services%2Fclusters-keeper%2Fsrc%2Fsimcore_service_clusters_keeper%2Futils%2Fclusters.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvY2x1c3RlcnMta2VlcGVyL3NyYy9zaW1jb3JlX3NlcnZpY2VfY2x1c3RlcnNfa2VlcGVyL3V0aWxzL2NsdXN0ZXJzLnB5) | `100.0% <100.0%> (ø)` | | | [...r/src/simcore\_service\_clusters\_keeper/utils/ec2.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851?src=pr&el=tree&filepath=services%2Fclusters-keeper%2Fsrc%2Fsimcore_service_clusters_keeper%2Futils%2Fec2.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvY2x1c3RlcnMta2VlcGVyL3NyYy9zaW1jb3JlX3NlcnZpY2VfY2x1c3RlcnNfa2VlcGVyL3V0aWxzL2VjMi5weQ==) | `100.0% <100.0%> (ø)` | | ... and [972 files with indirect coverage changes](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5851/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation)