Datashader does not update in 'Panel App' preview started from Jupyter

[rfenouil]

Hi, As asked by @Hoxbro, here is a new issue revealed after the resolution of #6095

It seems that datashader figures in a Panel App don't update their content properly (zoom/unzoom) when the App is started from the blue button 'Preview Panel App' in Jupyterlab.

ALL software version info

Python              :  3.10.11 | packaged by conda-forge | (main, May 10 2023, 18:58:44) [GCC 11.3.0]
Operating system    :  Linux-6.2.0-39-generic-x86_64-with-glibc2.35
Panel comms         :  default

holoviews           :  1.18.2

bokeh               :  3.3.4
colorcet            :  3.0.1
cudf                :  23.12.1
dask                :  2023.11.0
datashader          :  0.16.0
geoviews            :  -
hvplot              :  -
ibis-framework      :  -
IPython             :  8.20.0
jupyter_bokeh       :  -
jupyterlab          :  4.0.1
matplotlib          :  3.8.0
networkx            :  3.2.1
notebook            :  6.5.4
numba               :  0.57.1
numpy               :  1.23.4
pandas              :  1.5.3
panel               :  1.3.8
param               :  2.0.2
pillow              :  10.0.1
plotly              :  5.18.0
pyarrow             :  14.0.2
pyviz_comms         :  3.0.0
scikit-image        :  0.20.0
scipy               :  1.12.0
spatialpandas       :  0.4.10
streamz             :  0.6.4
tsdownsample        :  -
xarray              :  2023.12.0

Description of expected behavior and the observed behavior

When launching the 'Panel Application Preview' from Jupyter (click on the blue button), datashader figures should update according to the zoom level, as it does in notebook cells or when starting app with command line.

Complete, minimal, self-contained example code that reproduces the issue

import holoviews as hv
import datashader as ds
from holoviews.operation.datashader import datashade, rasterize, shade, dynspread, spread
import numpy as np
import pandas as pd
import panel as pn

hv.extension('bokeh','matplotlib', width=100)

### Helper functions to create points dataset
# from: https://holoviews.org/user_guide/Large_Data.html#multidimensional-plots
np.random.seed(3)

def random_cov():
    """Random covariance for use in generating 2D Gaussian distributions"""
    A = np.random.randn(2,2)
    return np.dot(A, A.T)

def rand_gauss2d(value=0, n=10000):
    """Return a randomly shaped 2D Gaussian distribution with an associated numeric value"""
    g = 100*np.random.multivariate_normal(np.random.randn(2), random_cov(), (n,))
    return np.hstack((g,value*np.ones((g.shape[0],1))))

### Generate a dataset (5 gaussians with different values)
dataset = pd.concat([pd.DataFrame(rand_gauss2d(i), columns = ["x","y","k"]) for i in range(5)])
dataset["k"] = dataset["k"].astype(str) # Convert the category to string (just to be sure)

### Create HV data representations
hv_data = dynspread(datashade(hv.Points(dataset),
                              aggregator=ds.by("k",ds.count())))

pn.Row(hv_data).servable()

Stack traceback and/or browser JavaScript console output

Appears once after kernel is launched, then seems to be repeated on zoom/unzoom attempts:

2024-02-07 16:14:47,554 ERROR: panel.io.jupyter_server_extension - Task exception was never retrieved
future: <Task finished name='Task-97' coro=<Callback.process_on_event() done, defined at /opt/conda/lib/python3.10/site-packages/holoviews/plotting/bokeh/callbacks.py:328> exception=AbbreviatedException(<class 'RuntimeError'>, RuntimeError('_pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes'), <traceback object at 0x7fb73544db80>)>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/holoviews/plotting/bokeh/element.py", line 1954, in _update_glyphs
    self._update_glyph(renderer, properties, mapping, glyph, source, data)
  File "/opt/conda/lib/python3.10/site-packages/holoviews/plotting/bokeh/element.py", line 1737, in _update_glyph
    self._update_datasource(source, data)
  File "/opt/conda/lib/python3.10/site-packages/holoviews/plotting/bokeh/plot.py", line 192, in _update_datasource
    source.data.update(data)
  File "/opt/conda/lib/python3.10/site-packages/bokeh/core/property/wrappers.py", line 433, in update
    descriptor._notify_mutated(owner, old, hint=hint)
  File "/opt/conda/lib/python3.10/site-packages/bokeh/core/property/descriptors.py", line 658, in _notify_mutated
    self._set(obj, old, value, hint=hint)
  File "/opt/conda/lib/python3.10/site-packages/bokeh/core/property/descriptors.py", line 621, in _set
    self._trigger(obj, old, value, hint=hint, setter=setter)
  File "/opt/conda/lib/python3.10/site-packages/bokeh/core/property/descriptors.py", line 699, in _trigger
    obj.trigger(self.name, old, value, hint, setter)
  File "/opt/conda/lib/python3.10/site-packages/bokeh/model/model.py", line 569, in trigger
    super().trigger(descriptor.name, old, new, hint=hint, setter=setter)
  File "/opt/conda/lib/python3.10/site-packages/bokeh/util/callback_manager.py", line 188, in trigger
    self.document.callbacks.notify_change(cast(Model, self), attr, old, new, hint, setter, invoke)
  File "/opt/conda/lib/python3.10/site-packages/bokeh/document/callbacks.py", line 249, in notify_change
    self.trigger_on_change(event)
  File "/opt/conda/lib/python3.10/site-packages/bokeh/document/callbacks.py", line 413, in trigger_on_change
    invoke_with_curdoc(doc, invoke_callbacks)
  File "/opt/conda/lib/python3.10/site-packages/bokeh/document/callbacks.py", line 443, in invoke_with_curdoc
    return f()
  File "/opt/conda/lib/python3.10/site-packages/bokeh/document/callbacks.py", line 412, in invoke_callbacks
    cb(event)
  File "/opt/conda/lib/python3.10/site-packages/bokeh/document/callbacks.py", line 276, in <lambda>
    self._change_callbacks[receiver] = lambda event: event.dispatch(receiver)
  File "/opt/conda/lib/python3.10/site-packages/bokeh/document/events.py", line 427, in dispatch
    super().dispatch(receiver)
  File "/opt/conda/lib/python3.10/site-packages/bokeh/document/events.py", line 219, in dispatch
    cast(DocumentPatchedMixin, receiver)._document_patched(self)
  File "/opt/conda/lib/python3.10/site-packages/bokeh/server/session.py", line 244, in _document_patched
    raise RuntimeError("_pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes")
RuntimeError: _pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/holoviews/plotting/bokeh/callbacks.py", line 351, in process_on_event
    self.on_msg(msg)
  File "/opt/conda/lib/python3.10/site-packages/holoviews/plotting/bokeh/callbacks.py", line 221, in on_msg
    raise e
  File "/opt/conda/lib/python3.10/site-packages/holoviews/plotting/bokeh/callbacks.py", line 213, in on_msg
    Stream.trigger(streams)
  File "/opt/conda/lib/python3.10/site-packages/holoviews/streams.py", line 189, in trigger
    subscriber(**dict(union))
  File "/opt/conda/lib/python3.10/site-packages/holoviews/plotting/plot.py", line 243, in refresh
    raise e
  File "/opt/conda/lib/python3.10/site-packages/holoviews/plotting/plot.py", line 239, in refresh
    self._trigger_refresh(stream_key)
  File "/opt/conda/lib/python3.10/site-packages/holoviews/plotting/plot.py", line 257, in _trigger_refresh
    self.update(key)
  File "/opt/conda/lib/python3.10/site-packages/holoviews/plotting/plot.py", line 956, in update
    item = self.__getitem__(key)
  File "/opt/conda/lib/python3.10/site-packages/holoviews/plotting/plot.py", line 443, in __getitem__
    self.update_frame(frame)
  File "/opt/conda/lib/python3.10/site-packages/holoviews/plotting/bokeh/element.py", line 2034, in update_frame
    self._update_glyphs(element, ranges, self.style[self.cyclic_index])
  File "/opt/conda/lib/python3.10/site-packages/holoviews/plotting/bokeh/element.py", line 1953, in _update_glyphs
    with abbreviated_exception():
  File "/opt/conda/lib/python3.10/site-packages/holoviews/core/options.py", line 222, in __exit__
    raise AbbreviatedException(etype, value, traceback)
holoviews.core.options.AbbreviatedException: RuntimeError: _pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes

To view the original traceback, catch this exception and call print_traceback() method.

Screenshots or screencasts of the bug in action

[hoxbro]

Thank you for opening the issue. I have reduced the example down to the following:

import holoviews as hv
import numpy as np
import panel as pn
from holoviews.operation.downsample import downsample1d

hv.extension('bokeh')

a = downsample1d(hv.Points(np.random.randn(1000, 2)), algorithm='nth')
pn.panel(a).servable()

I'm not sure if the problem needs fixing in HoloViews or Panel. Do you have an idea, @philippjfr?

[philippjfr]

Probably in the stream callbacks code in HoloViews.

[jsilbergGH]

Not sure if this is applicable, but when I had a similar issue (not in jupyter, in a bokeh server application) adding explicit ann mode = "server" to creating my hv object solved the problem