Make director-v2 scalable/restartable

ITISFoundation / osparc-simcore

🐼 osparc-simcore simulation framework

https://osparc.io

MIT License

43 stars 27 forks source link

Make director-v2 scalable/restartable #4524

Open sanderegg opened 1 year ago

sanderegg commented 1 year ago

Currently the dv-2 has 2 main jobs:

start/stop/schedule dynamic services
start/stop/schedule computational services

Due to internal limitations, it is not able to be scaled up, which is a big problem for robustness of the oSparc platform.

### Tasks
- [ ] computational scheduler: remove in process data and rely only on database before scheduling pipelines
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5631
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5632

mrnicegyu11 commented 1 year ago

Quick comment: In my mind it has much higher priority that the guy is restartable, scaling comes afterwards ;)

sanderegg commented 1 year ago

@mrnicegyu11 please define restartable because:

restart phase: there are 2 instances running in parallel for some time (even small), which means it needs to be scalable first or unintended stuff might happen I think what you want is a kind of stateless service (which is also what I want), and that it can restart while osparc is running. But while we have several instances up they need to play nice with each other, then you can restart 1 instance without breaking computation pipelines or running dynamic services.