dask-scheduler open issues

sanderegg commented 3 years ago

General

[x] handle dask-worker resources (set number of CPUs, number of GPUS, RAM per workers) #2486
[x] set real needed resources per service when submitting jobs #2486
[x] handle auto-scaling of sidecar with on-premise systems #2486
[x] API to access user available clusters
[x] UI/UX for selecting clusters and/or accessing clusters based on groups
[x] show a message when there are no available sidecar with the necessary resources
[x] remove direct dependencies in dask-sidecar (RabbitMQ, PostgresDB, S3)
[x] Handling of resiliency (restart of scheduler, restart of workers, user feedback)
- [x] jobs should not disappear if director-v2 restarts (persistence in comp_runs table?) (ITISFoundation/osparc-simcore#2798)
- [x] ever running tasks after one of director-v2, dask-scheduler, dask-worker restarts or times-out (ITISFoundation/osparc-simcore#2798)
[x] Review handling of reconnection to dask-scheduler
[x] director-v2 issue: dask client loosing connection to scheduler (staging, maybe master?)
[x] director-v2 issue: after deployment no logs/progress reports until director-v2 restarted
[x] https://github.com/ITISFoundation/osparc-simcore/issues/2585#issuecomment-947423923
[x] remove cluster_id from worker list of resources ~~- [ ] Feature: investigate how to make dask-sidecar healtch-check independent of dashboard (check nanny code?)~~
[x] set default cluster with osparc-dask-gateway

[x] show metrics as in dask dashboard
[x] metrics for GPU
[x] metrics do not show RAM/CPU usage due to the spawned containers (- [ ] investigate why logs take time to be shown)

[x] connect to dask gateway (POC)
[x] UI/UX Allow the user to set-up the credentials to access the dask-gateway using different auth options (also need to modify DB tables accordingly) #2631
[x] change dask-sidecar boot script, add @mguidon modifications
[ ] HTTPS gateway (try self signed certificates analogue to how its done in github)
Handle N gateways:
- [x] connect to gateways on the fly if necessary, keep a lazy list
- [x] persist connections if there are jobs and get updates ~~- [ ] in case of services originating in github, convert image name to dockerhub if cluster is not the default one~~
[ ] setup auto-scaling of workers
[x] Monitoring of defined Gateway (is it accessible? Up? down?)
[x] Kerberos/Jupyterhub authentication to gateways? (OBE)
[x] multi-platform (adm64, ARMv7, ARM64)
[ ] if cluster has no resources available, the pipeline keeps being pending (the cluster continuously starts/stops workers)
[x] scaling the cluster always starts on the same node (even if that node cannot handle some task due to missing resources)

issues:

[x] dask-workers memory release
[x] dask-client does not reconnect when the dask-scheduler is restarted
[x] dask-workers do not reconnect when the dask-scheduler is restarted
[x] investigate usage of persist/publish_dataset to keep jobs from disappearing and accessing the jobs from more than one client

sanderegg commented 3 years ago

Use-case:

a user go to the preferences, set the AWS token to create the cluster
a user creates the AWS cluster (select hardware, number of machines) with a unique identifier and a human readable one
a user can set access rights on the created cluster (groups, users)
status of cluster creation/availability shall be available
The run button in the pipeline view shows all the clusters where it can be run
the user shall destroy the cluster

sanderegg commented 1 year ago

closing this case as all remaining entries exist separately