S-D25.4 Simulation framework SCHEDULER Y4M05

KZzizzle commented 4 years ago

[x] Intelligent scheduler and queueing;
[ ] including for HPC

keep in mind

[ ] running MPI parallelized codes across multiple containers #346
[x] backend for guided mode #327 - done
[ ] scalability #350

Definition of Done:

It is possible for a service to provide information about resources that it requires (estimated RAM, need for a certain kind of GPU, number of cores, perhaps even estimates of runtime - think s4l EM simulations). It gets dispatched accordingly.
Dependencies are automatically converted into optimized execution orders and parallel execution where feasible
Services that declare (s. above) that they support MPI can be executed across multiple docker instances with a shared MPI framework. For that, they are dispatched to suitably ‘close’ machines
Progress of queued/completed jobs can be inspected (queue manager) and tasks can be removed from the execution queue, or their relative priority adapted (less urgent).

User story:

Beyond done:

The necessary HW requirement information is part of a service’s meta-data, but can also be inspected by users (e.g., in the info section of the service)
Users with the necessary permissions can edit some of these settings (e.g., number of parallel instances in an MPI-parallelized service)

MVP for closing (25.4 and 25.5)

[x] finish computational and dynamic sidecars
[ ] dask gateways (incl. for AWS)
[x] ability of adding additional HW (incl. AWS; incl. frontend)

sanderegg commented 3 years ago

status as of 03.01.2021

Done:

Computational states are now retrieved per node (modified, waiting for dependencies) backend + frontend
Ongoing:
Bugfixing on concurrent updates of nodes after running in a pipeline (when several sidecars are used on one pipeline)
Moving scheduling, currently done by sidecar itself to director-v2 --> simplifying architecture, reducing possible issues to allow for open enhancements/issues
Creation of dynamic service sidecar, to allow for improved security, better handling of dynamic services
Open:
ensure that sidecars do not access DB directly to remove the DB max connection problem
investigate using Airflow/Prefect.io for finer grained dispatching of computational pipelines (would also bring UI to better follow individual tasks)
create a Computational Resource Manager micro-service (CRMs) that would monitor the computational queues (load) to start/stop machines and sidecars to cope with variable computational load.

sanderegg commented 3 years ago

Update on sprint Red Panda

Done:

Bugfixing on concurrent updates of nodes after running in a pipeline (when several sidecars are used on one pipeline) #2174 --> will be deployed to production next sprint
Planning of scheduler improvements/requirements
Moving scheduling, currently done by sidecar itself to director-v2 --> simplifying architecture, reducing possible issues to allow for open enhancements/issues #2164 --> will be deployed to master next sprint (was waiting for maintenance cases to go in first)

Ongoing:

Evaluation of Apache Airflow / Prefect.io as workflow management tools #1992
Creation of dynamic service sidecar, to allow for improved security, better handling of dynamic services #1887
Open:
ensure that sidecars do not access DB directly to remove the DB max connection problem #2239
create a Computational Resource Manager micro-service (CRMs) that would monitor the computational queues (load) to start/stop machines and sidecars to cope with variable computational load. #2240

pcrespov commented 3 years ago

Update on sprint Marmoset

Done:

ITISFoundation/osparc-simcore#1992: Completed review of workflow system frameworks with general overview, features comparison table and DAR table
Evaluation of dask integration with SLURM
Added dask scheduler and sidecar worker in osparc-simcore stack (main PR: ITISFoundation/osparc-simcore#2366)

Ongoing:

ITISFoundation/osparc-simcore#1887: Partially integrated in osparc-simcore:
- merged ITISFoundation/osparc-simcore#2406
- merged ITISFoundation/osparc-simcore#2408
- raising coverage before merging ITISFoundation/osparc-simcore#2411
Integration of dask in director2 (last step to allow running osparc pipelines )

Open:

ensure that sidecars do not access DB directly to remove the DB max connection problem #2239
create a Computational Resource Manager micro-service (CRMs) that would monitor the computational queues (load) to start/stop machines and sidecars to cope with variable computational load. #2240

sanderegg commented 3 years ago

Update on sprint Chevrotain

Done

Connect director-v2 to dask-based backend #2418
Allow dask-sidecar to run multiple services in separate threads #2486, #2487
Added facilities to create, modify, remove clusters and share them among groups of users #2517, #2502, #2499
Dask-sidecar to advertise clusterID if available on start #2505
Improving resiliency of Dask-based backend #2514
Bugfixes on dask-based backend #2512, #2508, #2503
Tested connection with AWS machines through docker swarm
Remove celery sidecars #2528

Ongoing
Refactor computational sidecar to remove dependencies to RabbitMQ/PostgresDB #2530
Testing with Dask gateway on AWS cluster
Adding nodeports to dynamic-sidecar #2509, #2516; extending existing validation image for nodeports support #138

sanderegg commented 3 years ago

Update on sprint Capra delle nevi

Done

Adding nodeports to dynamic-sidecar #2509, #2516; extending existing validation image for nodeports support #138
Running sidecar through Dask gateway POC

Ongoing

Refactor computational sidecar to remove dependencies to RabbitMQ/PostgresDB #2530
Connect director-v2 with dask-gateway(s) #2576

Open

Setup Dask gateway on AWS cluster (easy setup, autoscale)
Reliability, UI, register clusters

sanderegg commented 3 years ago

Update on sprint Anti-PER

Done

Replacing dynamic sidecar proxy from Traefik to Caddy for faster start #2597
Allow node_ports package to create and get presigned links in storage for delayed upload and downloads by the dask-sidecar micro-service #2605

Ongoing

Refactor computational sidecar to remove dependencies to RabbitMQ/PostgresDB #2530 99% completed
Connect director-v2 with dask-gateway(s) #2576
Setup Dask gateway on AWS cluster (easy setup, autoscale)
Reliability, UI, register clusters

Open

sanderegg commented 2 years ago

Update on sprint Meerkat

Done

new repository osparc-dask-gateway, Dask-gateway with oSparc backend, create a Dask-sidecar for each worker (a user owns a cluster, a user deploy the osparc-dask-gateway, a user registers the gateway in oSparc)
oSparc:
- a user can register a cluster providing an entrypoint and user/password
- a user may share the registered cluster among other users/groups
- keeps a pool of connections to the internal cluster and any number of external osparc-dask-gateways
- sends pipeline of computational tasks to any cluster
- receives logs, results from any cluster
scaling:
- user can add/remove machines manually to already running cluster
- dask-gateway will use the machines accordingly
  Ongoing
osparc-dask-gateway: SSL entrypoint, authentications, simplify deployment
oSparc:
- improve UI
- test connection with cluster (provides feedback in case of issue/loss)
- show cluster capabilities, running tasks
- handle closed-source services
- scaling:
- provide auto-scaling on AWS

sanderegg commented 2 years ago

Update on sprint Rudolph

Done

ITISFoundation/osparc-dask-gateway#16: oSparc-dask-gateway now available for x86, ARMv7 and ARM64, based on dask-gateway Database Backend, migrated to python 3.9
ITISFoundation/osparc-simcore#2747
ITISFoundation/osparc-dask-gateway#18
ITISFoundation/osparc-dask-gateway#19: auto-start dask-worker(s) based on available nodes
ITISFoundation/osparc-dask-gateway#21: simplify configuration and startup
Bug fixes
ITISFoundation/osparc-dask-gateway#20
Changed
ITISFoundation/osparc-dask-gateway#17

Open issue / ongoing

ITISFoundation/osparc-simcore#2477

sanderegg commented 2 years ago

Update on sprint R. Schumann

Done

investigating persistency in dask-scheduler, reactions with restarts in director-v2 or dask-scheduler or dask-sidecar
Bug fixes

Changed

Open issue / ongoing
ITISFoundation/osparc-simcore#2477:
- ITISFoundation/osparc-simcore#2798
- ITISFoundation/osparc-simcore#2781
- add HTTPS to connections with dask-gateway
- run public services available on dockerhub on external gateways
- auto-scaling in AWS

sanderegg commented 2 years ago

Update on sprint E. Shackleton

Done

ITISFoundation/osparc-simcore#2937
ITISFoundation/osparc-simcore#2925
ITISFoundation/osparc-simcore#2920
ITISFoundation/osparc-simcore#2918
ITISFoundation/osparc-simcore#2914
ITISFoundation/osparc-simcore#2909
ITISFoundation/osparc-simcore#2798
ITISFoundation/osparc-simcore#2786
Bug fixes

Changed

Open issue / ongoing
ITISFoundation/osparc-simcore#2477:
- ITISFoundation/osparc-simcore#2922
- ITISFoundation/osparc-simcore#2940
- add HTTPS to connections with dask-gateway
- run public services available on dockerhub on external gateways
- auto-scaling in AWS

sanderegg commented 2 years ago

Update on sprint Macarons

Done

Cluster used for running project is persisted and retrieved:

ITISFoundation/osparc-simcore#2922
https://github.com/ITISFoundation/osparc-simcore/pull/2978

Fixed on cluster preferences:

Retrieve cluster used vs available resources:

https://github.com/ITISFoundation/osparc-simcore/pull/2972
ITISFoundation/osparc-simcore#2940

Open issue / ongoing

ITISFoundation/osparc-simcore#2477:
- add HTTPS to connections with dask-gateway
- run public services available on dockerhub on external gateways
- auto-scaling in AWS
- https://github.com/ITISFoundation/osparc-simcore/pull/2945

mrnicegyu11 commented 2 years ago

Update on sprint Vaporwave

Done

ITISFoundation/osparc-simcore#3347

Ongoing

656
- ITISFoundation/osparc-simcore#3338
- ITISFoundation/osparc-simcore#3339
  - ITISFoundation/osparc-simcore#3364
- ITISFoundation/osparc-simcore#3340
- ITISFoundation/osparc-simcore#3341
- ITISFoundation/osparc-simcore#3342

Open

657

ITISFoundation / osparc-issues