Access dashboard outside of Jupyter Hub

[jacobtomlinson]

Hi folks sorry to hijack this closed thread, thought this might be a useful place to ask the question if am running dask Kubernetes without using jupyterhub in a production pipeline, What would be the best way to tap or hook into and expose the ephemeral dask cluster via the dashboard UI whenever the scheduler is created at ad-hoc? @mrocklin

Originally posted by @timothy-mugayi in https://github.com/dask/dask-kubernetes/issues/6#issuecomment-884948173

[jacobtomlinson]

This totally depends on your cluster setup.

The Dask dashboard is made available as a service. You can expose this service however you like. It's common to use ingresses, we don't support that directly in dask-kubernetes right now but we could definitely talk about adding it.

You could also build some custom proxy which exposes all Dask dashboards for you.

If you could share more about your cluster setup we may be able to answer this question more specifically.

[timothy-mugayi]

@jacobtomlinson my setup is simple, a couple of ec2 instances in AWS auto-scaling group with the client initialization cluster code as illustrated below to create the ephemeral dask pods along with the scheduler. The duration of the desk cluster only lasts as long as dask is servicing the request then we gracefully gets shut down on the job is complete. So as you suggested this proxy which exposes all dask dashboards should be invoked whenever the dashboard service is started, wondering if I should be exploring dask gateway as I plan to have more than N clusters with multiple dashboards per cluster https://gateway.dask.org/, would this allow isolation of each cluster pip packages as each cluster as variations of these which may not be same as the next cluster.

cluster = KubeCluster(pod_template=dask_pod_template,
                          scheduler_pod_template=scheduler_pod_template,
                          namespace=os.environ["MY_POD_NAMESPACE"],
                          deploy_mode='remote',
                          scheduler_service_wait_timeout=300,
                          idle_timeout="90")

cluster.scale(10)  

cluster.adapt(minimum=1, maximum=100) 

log.debug(f"Dashboard url: {cluster.dashboard_link}")

k8s dask spec file

spec:
  containers:
  - args:
      - dask-scheduler
    name: dask-scheduler
    image: $DASK_IMAGE
    resources:
      limits:
        cpu: "$CPUS"
        memory: $MEMORY
      requests:
        cpu: "$CPUS"
        memory: $MEMORY

[jacobtomlinson]

I think for your setup you would probably need to go down the road of setting up an ingress service and some wildcard domain pointing to the ingress loadbalancer. However this would likely involve some development on your part.

Dask Gateway is also an excellent option for multi-tenant Dask clusters. Without knowing much about what you are using your clusters for (in terms of user workflows) all I can recommend is that you give it a go!

[timothy-mugayi]

Thanks @jacobtomlinson for the insights let me take a crack it

[jacobtomlinson]

Closing due to inactivity.

[cagantomer]

Hi, I am facing the same issue. Thought about using the gateway but have not got into that yet. Can you elaborate on kubecluster/ingress route? Or the proxy?

A little about my setup - we are running a Jenkins job. It includes a script that builds a custom docker image and then uses it to spawn it's own ad-hoc dask cluster. The execution time can be several hours so it would be nice that the users can follow it through the dashboard.

Right now I started by just printing to the log the full address of the scheduler and how to port-forward using kubectl - but having a direct link to the dashboard would be nice.

While at it - is there a way from the KubeCluster directly, or the client, the full scheduler pod name / address? All I am seeing is truncated name of the cluster and used kubernetes library to find the actual pod name. I tried to use describe_pod but either I am not running it correctly (it is an async function and I try to run it from sync code) or there's some issue with it / environment (I tried locally on my computer, I have .kube config environment.

(if need, I can open another issue for this).

I'd be happy to try and contribute necessary code...

[jacobtomlinson]

@cagantomer this sounds like a great usage conversation to have over on the community forum.