Risk of data loss when restarting dynamic sidecar on autoscaled instance

[sanderegg]

Is there an existing issue for this?

• [x] I have searched the existing issues

Which deploy/s?

production aws (e.g. osparc.io), staging aws (e.g. staging.osparc.io), production on-premise (dalco), stating on-premise (dalco), development (master), other (e.g. local)

Current Behavior

Folow-up from https://github.com/ITISFoundation/osparc-simcore/issues/5840

pre-requisite

• running dynamic-sidecar with its containers on an autoscaled instance

how to reproduce

• restart the dynamic-sidecar
• have services starting at the same time

issue

• while the dynamic-sidecar restarts, there is a risk that another dynamic-sidecar "steals" the node,
• when that happens the node is then fully used by the new sidecar,
• the original sidecar cannot start and remains pending until the new sidecar is eventually switched off, --> the disk on the machine is partially used by the original sidecar, --> the user of the original sidecar is blocked --> if no care is taken the data of the original sidecar might be lost

Notes

• this will not happen if no other service is started at the same time (as the autoscaling service waits a bit before draining the node: https://github.com/ITISFoundation/osparc-simcore/issues/5842
• sometimes one can have luck

Expected Behavior

• The sidecar should restart on the same instance and no other sidecar should steal its position,
• Or, the sidecar should move to another instance keeping all the data together.

### Tasks
- [ ] working group to find a way out