🐛🎨♻️Director-v0: improve registry caching

sanderegg commented 1 day ago

What do these changes do?

Since director-v0 was updated, the home-made caching mechanism was replaced by aiocache. This lead to a difference in behavior, as the home-made caching was running independently of Rest API calls and would pre-build the cache before any other client would list services. So the current situation is that the first call to the director /services takes a long time until it gets cached properly.

This PR brings:

migrated director-v0 client to registry from aiohttp to httpx,
added tenacity-based retrials on calls to the docker registry,
use of async generators,
modernized code,

Related issue/s

How to test

Dev-ops checklist

[x] No ENV changes or I properly updated ENV (read the instruction)

codecov[bot] commented 1 day ago

Codecov Report

Attention: Patch coverage is 68.58974% with 49 lines in your changes missing coverage. Please review.

Project coverage is 89.83%. Comparing base (ba881e9) to head (cb215ae). Report is 1 commits behind head on master.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #6799 +/- ## ========================================== + Coverage 88.40% 89.83% +1.43% ========================================== Files 1545 1102 -443 Lines 61634 47971 -13663 Branches 2155 410 -1745 ========================================== - Hits 54488 43097 -11391 + Misses 6818 4804 -2014 + Partials 328 70 -258 ``` | [Flag](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [integrationtests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `64.76% <ø> (ø)` | | | [unittests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `87.61% <68.58%> (+1.18%)` | :arrow_up: | | [Components](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [api](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_aws_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_dask_task_models_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_models_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_notifications_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_postgres_database](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_service_integration](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_service_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_settings_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `90.62% <66.66%> (-0.38%)` | :arrow_down: | | [pkg_simcore_sdk](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `85.36% <ø> (ø)` | | | [agent](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `97.00% <ø> (ø)` | | | [api_server](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `89.72% <ø> (ø)` | | | [autoscaling](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `95.21% <ø> (ø)` | | | [catalog](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `90.57% <ø> (ø)` | | | [clusters_keeper](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `98.73% <ø> (ø)` | | | [dask_sidecar](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `91.26% <ø> (ø)` | | | [datcore_adapter](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `93.17% <ø> (ø)` | | | [director](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `76.41% <68.75%> (+0.35%)` | :arrow_up: | | [director_v2](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `91.06% <ø> (-0.02%)` | :arrow_down: | | [dynamic_scheduler](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `96.59% <ø> (ø)` | | | [dynamic_sidecar](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `89.75% <ø> (ø)` | | | [efs_guardian](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `90.12% <ø> (ø)` | | | [invitations](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `93.44% <ø> (ø)` | | | [osparc_gateway_server](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `85.49% <ø> (ø)` | | | [payments](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `92.76% <ø> (ø)` | | | [resource_usage_tracker](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `90.64% <ø> (ø)` | | | [storage](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `89.66% <ø> (ø)` | | | [webclient](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [webserver](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `88.68% <ø> (ø)` | | ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation). Last update [ba881e9...cb215ae](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6799?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation).

🚨 Try these New Features: