Restarting a dynamic sidecar on an autoscaled node triggers node drain and termination

[sanderegg]

If the dynamic-sidecar of a service is restarted while running on an autoscaled node there are high chances that the node will be drained and terminated during that process.

Here is the explanation why this happens:

0. dynamic-sidecar running with its containers on autoscaled node
1. restart dynamic-sidecar via portainer
2. autoscaling finds an empty node, and sets the osparc-ready label to false
3. the restarted sidecar is rejected,
4. autoscaling then skips sidecars with a node.id constraint
5. autoscaling will then terminate the node
6. service is gone, with probable data loss

It is probably also possible that another dynamic sidecar would steal that node.

### Tasks
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5842

[sanderegg]

what happens when the sidecar is restarted:

(docker default behavior)

1. the sidecar is stopped,
2. some time passes <-- if there are other dynamic sidecars starting, they could steal the node. if there are none and autoscaling runs during that time it might set the node to "not-ready"
3. the replacing sidecar is started

would changing the start phase to start-first fix it?

1. the replacing sidecar is started (but would not start due to missing resources)
2. this would fail the restart due to missing resources.... so not an option

add some delay in autoscaling

• adding a delay before a node is set to "not-ready" --> would probably fix the issue where the sidecar is rejected --> will NOT fix the possibility for another sidecar to steal the node

[sanderegg]

remaining issue:

• https://github.com/ITISFoundation/osparc-simcore/issues/5862