🐛 Dy-Sidecar incident: Dy-Sidecar orphaned after long start-up time

mrnicegyu11 commented 2 years ago

f82ad5e3-f680-4e84-81b1: Oprhaned state: Sidecar running, idling, no attached dy-service-container What happened? Director-v2 fails service observation of this sidecar, with:

unexpectedly failed [OEC:140670113173664]
raise DynamicSidecarError( simcore_service_director_v2.modules.dynamic_sidecar.errors.DynamicSidecarError: Timed out while searching for an assigned NodeID for service_id=vp7nt7o8x4tiqcmwikoyfehcx

Approx. 23 minutes after this log-message, the dynamic sidecar starts, but stays idling forever. No caddy is found.

Actionable follow-ups:

Investigate long start-up time of dynamic sidecar
Make director-v2 handle long start-up times of dynamic sidecars
Tentative: Make GC clean-up orphaned sidecars just idling

@GitHK @pcrespov @sanderegg

dy-sidecar_f82ad5e3.txt oprhaned_dy_sidecar_aws_prod_1.csv

pcrespov commented 2 years ago

@GitHK please review these and organize a fix as agreed

GitHK commented 2 years ago

The behaviour is correct here.

Director-v2 could not determine a node where the dy-sidecar was started, because no containers was created in 60 seconds (DYNAMIC_SIDECAR_TIMEOUT_FETCH_DYNAMIC_SIDECAR_NODE_ID) . It is ok to fail.

The following PR https://github.com/ITISFoundation/osparc-simcore/pull/3272 has addressed the above issue. Services will get cleaned up after error and will no longer linger.

ITISFoundation / osparc-simcore

🐛 Dy-Sidecar incident: Dy-Sidecar orphaned after long start-up time #3193