🎨Computation backend: refactor director-v2 internal computational scheduler to be less resource heavy

sanderegg commented 3 weeks ago

What do these changes do?

Until now

dv-2 internal computational scheduler is run as a background task that concurrently schedules every ongoing computational job:
- every time a new pipeline is started, it re-schedules all the pipelines even if unnecessary,
- every time a pipeline is requested to stop, it re-schedules all the pipelines even if unnecessary,
- with multiple replicas this generates a load of unnecessary scheduling as all the replicas would schedule the same pipelines,

What this PR brings

exclusive decorator changes (redis_utils.py, test_redis_utils.py):
- decorator typing is now fixed
- decorator can now create the lock_key dynamically via a lock_key_builder callable (a la aiocache),
- decorator can now get the redis client dynamically via a redis_client_builder callable (a la aiocache),
dv-2 scheduling is now running 1 asyncio.Task per computational pipeline:
- each pipeline is now scheduled independently from the others every 5 seconds,
- when a pipeline is started, only that very pipeline is scheduled independently of any other pipeline,
- when a pipeline is requested to stop, only that very pipeline is scheduled independently of any other pipeline,
- when a computational node inside a pipeline is complete, only the very pipeline it lives in is re-scheduled without waiting for 5 seconds,
- when a pipeline is complete (success, cancelled or errored), the scheduling task is removed,
- with multiple replicas of the dv-2, only 1 replica schedule a pipeline at a time, the other replicas will skip it as the scheduling method is made exclusive

What this PR does not yet bring (will come in part 2 and is not part of this PR!!!)

mulitple replicas of the dv-2 shall share the load of scheduling pipelines, which will probably be done via RabbitMQ

Related issue/s

How to test

Dev-ops checklist

[x] No ENV changes or I properly updated ENV (read the instruction)

codecov[bot] commented 3 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 64.44%. Comparing base (17f486e) to head (d9df523). Report is 1 commits behind head on master.

:exclamation: There is a different number of reports uploaded between BASE (17f486e) and HEAD (d9df523). Click for more details.

HEAD has 30 uploads less than BASE
| Flag | BASE (17f486e) | HEAD (d9df523) | |------|------|------| |unittests|31|1|

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #6696 +/- ## =========================================== - Coverage 87.73% 64.44% -23.30% =========================================== Files 1571 657 -914 Lines 63271 33103 -30168 Branches 2118 265 -1853 =========================================== - Hits 55512 21333 -34179 - Misses 7433 11709 +4276 + Partials 326 61 -265 ``` | [Flag](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | *Carryforward flag | |---|---|---|---| | [integrationtests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `64.85% <ø> (+<0.01%)` | :arrow_up: | Carriedforward from [17f486e](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/commit/17f486e3848c3943bb0b5f7f2336aa19a2dbcee3?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | | [unittests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `58.38% <ø> (-27.33%)` | :arrow_down: | | *This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) to find out more. | [Components](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [api](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_aws_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_dask_task_models_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_models_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_notifications_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_postgres_database](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_service_integration](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_service_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_settings_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_simcore_sdk](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `77.44% <ø> (-7.84%)` | :arrow_down: | | [agent](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [api_server](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [autoscaling](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [catalog](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [clusters_keeper](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [dask_sidecar](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [datcore_adapter](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [director](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `58.38% <ø> (ø)` | | | [director_v2](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `76.42% <ø> (-14.38%)` | :arrow_down: | | [dynamic_scheduler](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [dynamic_sidecar](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `59.79% <ø> (-30.01%)` | :arrow_down: | | [efs_guardian](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [invitations](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [osparc_gateway_server](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `79.42% <ø> (-5.73%)` | :arrow_down: | | [payments](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [resource_usage_tracker](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [storage](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [webclient](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [webserver](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `59.79% <ø> (-28.95%)` | :arrow_down: | ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation). Last update [17f486e...d9df523](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6696?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation).