[BUG] - Version 2023.11.1 Dask Dashboard issues

[rsignell]

Describe the bug

I upgraded from Nebari 2023.10.1 to 2023.11.1 in hopes of getting the Dask Dashboard working again with environments that use bokeh>=3.

The good news is that the Dashboard now loads.
The bad news is that only the Cluster Map seems to work. (e.g. task stream and progress components don't. )

Expected behavior

Expected behaviour is that all the Dask Dashboard components like task stream and progress work.

OS and architecture in which you are running Nebari

AWS deployment

How to Reproduce the problem?

To reproduce the screenshot above, run this code using the nebari-git-nebari-git-dask environment:

import dask_gateway

cluster = dask_gateway.GatewayCluster()
client = cluster.get_client()
cluster.scale(2)
print(cluster.dashboard_link)

import dask.array as da
x = da.random.random((10000, 10000), chunks=(1000, 1000))
y = x + x.T
z = y[:, 5000:].mean(axis=1).compute()

Command output

the output is the screenshot

Versions and dependencies used.

# Nebari version: 2023.11.1
# Kubernetes 1.26

$ conda --version
conda 4.13.0

$ (nebari-git-dask) rsignell:~
$ conda list | grep -E "bokeh|nebari-dask|hvplot|geoviews|panel|datashader"
bokeh                     3.3.1              pyhd8ed1ab_0    conda-forge
nebari-dask               2023.11.1            hd8ed1ab_0    conda-forge

Compute environment

AWS

Integrations

No response

Anything else?

No response

[aktech]

Hey @rsignell, thanks for reporting this. I had the chance to debug this issue. The problem is due to the failure in loading bokeh resources.

The path to bokeh resources is absolute instead of relative in /gateway/clusters/<cluster-id>/status:

            <script type="text/javascript" src="/static/js/bokeh-gl.min.js?v=e0e44b3afb85255b9f8a8ff9fee07ebba86718c6e0ef7bc3eb426fee09cfc1dcb4cff652ae20cc4f17d5f5cb18ce43897349e61f4a52eb98e8b0dbbbcab9c700"></script>
            <script type="text/javascript" src="/static/js/bokeh-widgets.min.js?v=bea832c1113f002130fcb5b6b3de38f9170e678251b8e18678e7420cb51f7c3f70828f97b5c40ee7a853d7939bf215e1a26281553f33a1f1a6f918839dee8169"></script>
            <script type="text/javascript" src="/static/js/bokeh-tables.min.js?v=53047657de541a8d8689e826c3db15877925535acc78347751b6f76ce38b4206f046262b1c713b22086a72d4b085cb0d835875b3eff64c1edf27e61f7e50aae9"></script>
            <script type="text/javascript" src="/static/js/bokeh-mathjax.min.js?v=c9041fdc4d9d74b373376214b2e4a595d80ab49d6b348cb792501f264bcba7c8e697bc0014d51c87aa366482a541e43f230e892d3e1111a08527555691d98d23"></script>
            <script type="text/javascript">

I'll work on a fix to be incorporated in Nebari, meanwhile you can get it working by loading bokeh resources via cdn, here is an example:

from dask_gateway import Gateway
gateway = Gateway()

# view the cluster options UI
options = gateway.cluster_options()
# Set Bokeh static resources to cdn
options.environment_vars = {"BOKEH_RESOURCES": "cdn"}
cluster = gateway.new_cluster(options)
cluster.scale(2)
print(cluster.dashboard_link)

[aktech]

Found an upstream issue for the same: https://github.com/bokeh/bokeh/issues/13521 (static resources changed from relative to absolute urls in 3.3.0)

• Relevant distributed link: https://github.com/dask/distributed/issues/8333
• Fixed in distributed recently: https://github.com/dask/distributed/pull/8347

[rsignell]

@aktech so it looks like for now we could pin bokeh=3.2 for now and then remove that pin when dask=2023.11.1 drops, right? (or I guess use dask from github/main right now if we don't care about sticking only with released versions)

[aktech]

You can set the env variable to load static files from cdn, with any bokeh version.

from dask_gateway import Gateway
gateway = Gateway()

# view the cluster options UI
options = gateway.cluster_options()
# Set Bokeh static resources to cdn
options.environment_vars = {"BOKEH_RESOURCES": "cdn"}
cluster = gateway.new_cluster(options)
cluster.scale(2)
print(cluster.dashboard_link)

[rsignell]

@aktech , just confirming that setting the env var:

options.environment_vars = {"BOKEH_RESOURCES": "cdn"}

indeed works!

Hopefully we don't have to do this in the near future because the next release of Nebari should pick up the latest dask version (2023.12.0+) that contains a fix, right?

[jacobtomlinson]

Using the CDN resources is a fine workaround. We couldn't default to this in dask because we need to continue supporting systems without internet access.

[rsignell]

Got it! Okay, thanks @jacobtomlinson , I'll close this issue then and just continue with the env var solution.

[pavithraes]

Re-opening - We need to add this workaround in the default Nebari config, to make sure Nebari users can use the labextension.