✨Autoscaling: add a delay before draining a node

sanderegg commented 1 month ago

What do these changes do?

This PR adds a delay (EC2_INSTANCES_TIME_BEFORE_DRAINING) before draining active nodes (e.g. EC2 in the cluster that are available to run services) so that in case a dynamic sidecar would fail on start it has time to restart and is not rejected.

driving tests:

test_set_node_found_empty
test_cluster_scaling_up_and_down

bonus:

some cleanup, test coverage still at 97%

Related issue/s

fixes https://github.com/ITISFoundation/osparc-simcore/issues/5842

Dev-ops checklist
- new ENV EC2_INSTANCES_TIME_BEFORE_DRAINING and WORKERS_EC2_INSTANCES_TIME_BEFORE_DRAINING that take in a timedelta between 10 seconds and 1 minute (auto-capped)
- https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/598
- [x] No ENV changes or I properly updated ENV (read the instruction)

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 94.54545% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 87.6%. Comparing base (cafbf96) to head (5cffffd). Report is 213 commits behind head on master.

Additional details and impacted files

[![Impacted file tree graph](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5843/graphs/tree.svg?width=650&height=150&src=pr&token=h1rOE8q7ic&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation)](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5843?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) ```diff @@ Coverage Diff @@ ## master #5843 +/- ## ========================================= + Coverage 84.5% 87.6% +3.1% ========================================= Files 10 1367 +1357 Lines 214 56790 +56576 Branches 25 1284 +1259 ========================================= + Hits 181 49801 +49620 - Misses 23 6715 +6692 - Partials 10 274 +264 ``` | [Flag](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5843/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [integrationtests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5843/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `65.1% <ø> (?)` | | | [unittests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5843/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `85.6% <94.5%> (+1.0%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5843?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [...gs-library/src/settings\_library/docker\_registry.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5843?src=pr&el=tree&filepath=packages%2Fsettings-library%2Fsrc%2Fsettings_library%2Fdocker_registry.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-cGFja2FnZXMvc2V0dGluZ3MtbGlicmFyeS9zcmMvc2V0dGluZ3NfbGlicmFyeS9kb2NrZXJfcmVnaXN0cnkucHk=) | `95.6% <100.0%> (ø)` | | | [...g/src/simcore\_service\_autoscaling/core/settings.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5843?src=pr&el=tree&filepath=services%2Fautoscaling%2Fsrc%2Fsimcore_service_autoscaling%2Fcore%2Fsettings.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvYXV0b3NjYWxpbmcvc3JjL3NpbWNvcmVfc2VydmljZV9hdXRvc2NhbGluZy9jb3JlL3NldHRpbmdzLnB5) | `100.0% <100.0%> (ø)` | | | [...oscaling/src/simcore\_service\_autoscaling/models.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5843?src=pr&el=tree&filepath=services%2Fautoscaling%2Fsrc%2Fsimcore_service_autoscaling%2Fmodels.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvYXV0b3NjYWxpbmcvc3JjL3NpbWNvcmVfc2VydmljZV9hdXRvc2NhbGluZy9tb2RlbHMucHk=) | `100.0% <100.0%> (ø)` | | | [.../simcore\_service\_autoscaling/utils/utils\_docker.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5843?src=pr&el=tree&filepath=services%2Fautoscaling%2Fsrc%2Fsimcore_service_autoscaling%2Futils%2Futils_docker.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvYXV0b3NjYWxpbmcvc3JjL3NpbWNvcmVfc2VydmljZV9hdXRvc2NhbGluZy91dGlscy91dGlsc19kb2NrZXIucHk=) | `100.0% <100.0%> (ø)` | | | [...c/simcore\_service\_clusters\_keeper/core/settings.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5843?src=pr&el=tree&filepath=services%2Fclusters-keeper%2Fsrc%2Fsimcore_service_clusters_keeper%2Fcore%2Fsettings.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvY2x1c3RlcnMta2VlcGVyL3NyYy9zaW1jb3JlX3NlcnZpY2VfY2x1c3RlcnNfa2VlcGVyL2NvcmUvc2V0dGluZ3MucHk=) | `96.2% <100.0%> (ø)` | | | [.../simcore\_service\_clusters\_keeper/utils/clusters.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5843?src=pr&el=tree&filepath=services%2Fclusters-keeper%2Fsrc%2Fsimcore_service_clusters_keeper%2Futils%2Fclusters.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvY2x1c3RlcnMta2VlcGVyL3NyYy9zaW1jb3JlX3NlcnZpY2VfY2x1c3RlcnNfa2VlcGVyL3V0aWxzL2NsdXN0ZXJzLnB5) | `97.5% <ø> (ø)` | | | [...e\_service\_autoscaling/modules/auto\_scaling\_core.py](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5843?src=pr&el=tree&filepath=services%2Fautoscaling%2Fsrc%2Fsimcore_service_autoscaling%2Fmodules%2Fauto_scaling_core.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation#diff-c2VydmljZXMvYXV0b3NjYWxpbmcvc3JjL3NpbWNvcmVfc2VydmljZV9hdXRvc2NhbGluZy9tb2R1bGVzL2F1dG9fc2NhbGluZ19jb3JlLnB5) | `94.2% <87.5%> (ø)` | | ... and [1336 files with indirect coverage changes](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/5843/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation)