✨Computational backend: refactoring of dv-2 computational scheduler (Part 2)

sanderegg commented 2 weeks ago

What do these changes do?

requires https://github.com/ITISFoundation/osparc-simcore/pull/6696 This PR removes:
all in process data kept concerning the computational runs (now only checks from the comp_runs table),
--> fixes cancellation of task when multiple replicas of the dv-2 are running
--> fixes potential issue where metadata could be lost if multiple dv-2 are running

NOTE: all replicas might still schedule the same pipelines which is inefficient, this will be the subject of PR part 3

Related issue/s

How to test

Dev-ops checklist

[x] No ENV changes or I properly updated ENV (read the instruction)

codecov[bot] commented 2 weeks ago

Codecov Report

Attention: Patch coverage is 92.45283% with 4 lines in your changes missing coverage. Please review.

Project coverage is 88.64%. Comparing base (9915ebd) to head (13af039). Report is 1 commits behind head on master.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #6711 +/- ## ========================================== + Coverage 87.74% 88.64% +0.90% ========================================== Files 1570 1271 -299 Lines 63326 54555 -8771 Branches 2119 897 -1222 ========================================== - Hits 55564 48360 -7204 + Misses 7435 6061 -1374 + Partials 327 134 -193 ``` | [Flag](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [integrationtests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `64.86% <82.69%> (-0.02%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `86.29% <92.45%> (+0.59%)` | :arrow_up: | | [Components](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [api](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_aws_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_dask_task_models_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_models_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `92.01% <100.00%> (ø)` | | | [pkg_notifications_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_postgres_database](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_service_integration](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `71.44% <ø> (ø)` | | | [pkg_service_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_settings_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_simcore_sdk](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `85.27% <ø> (ø)` | | | [agent](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `96.98% <ø> (ø)` | | | [api_server](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `89.88% <ø> (ø)` | | | [autoscaling](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `95.22% <ø> (ø)` | | | [catalog](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `89.42% <ø> (ø)` | | | [clusters_keeper](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `98.72% <ø> (ø)` | | | [dask_sidecar](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `91.32% <ø> (ø)` | | | [datcore_adapter](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `94.05% <ø> (ø)` | | | [director](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `58.38% <ø> (-0.05%)` | :arrow_down: | | [director_v2](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `90.81% <92.30%> (-0.05%)` | :arrow_down: | | [dynamic_scheduler](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `96.59% <ø> (ø)` | | | [dynamic_sidecar](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `89.78% <ø> (ø)` | | | [efs_guardian](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `90.01% <ø> (ø)` | | | [invitations](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `93.51% <ø> (ø)` | | | [osparc_gateway_server](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `85.15% <ø> (ø)` | | | [payments](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `92.87% <ø> (ø)` | | | [resource_usage_tracker](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `90.65% <ø> (-0.15%)` | :arrow_down: | | [storage](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `89.76% <ø> (ø)` | | | [webclient](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [webserver](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `88.71% <ø> (-0.02%)` | :arrow_down: | ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation). Last update [9915ebd...13af039](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6711?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation).