Unable to to serve datashade plot to multiple users

[danjjl]

I am using the Deploying with bokeh serve example from the documentation to create a minimal holoviews/bokeh server.

However, instead of giving a holoviews.DynamicMap to hv.renderer('bokeh').app(); I am giving it a holoviews.operation.datashader.datashade.

This causes a Tornado lock to be opened and never closed making it impossible to serve multiple users.

Here is a minimal example that works fine for one user but fails for multiple users:

import numpy as np
import holoviews as hv
import holoviews.plotting.bokeh
from holoviews.operation.datashader import datashade

renderer = hv.renderer('bokeh')

def sine(frequency, phase, amplitude):
    xs = np.linspace(0, np.pi*4)
    return hv.Curve((xs, np.sin(frequency*xs+phase)*amplitude)).options(width=800)

dmap = datashade(sine(1, 0, 1))
# Replacing by next line works fine for multiple users.
# dmap = hv.DynamicMap(sine, kdims=['frequency', 'phase', 'amplitude']).redim.range(**ranges)

app = renderer.app(dmap)
print(app)

from bokeh.server.server import Server
server = Server({'/': app}, port=0)
server.start()
server.show('/')

from tornado.ioloop import IOLoop
loop = IOLoop.current()
loop.start()

Is it possible to create a minimal datashader/holoviews/bokeh server that can serve multiple users?

[danjjl]

Here is the error returned when a second user tries to open the same URL:

ERROR:tornado.application:Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x1c1d4ae8c8>, <Future finished exception=RuntimeError('_pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes',)>)
Traceback (most recent call last):
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/tornado/ioloop.py", line 759, in _run_callback
    ret = callback()
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/tornado/stack_context.py", line 276, in null_wrapper
    return fn(*args, **kwargs)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/tornado/ioloop.py", line 780, in _discard_future_result
    future.result()
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/server/session.py", line 51, in _needs_document_lock_wrapper
    result = yield yield_for_all_futures(func(self, *args, **kwargs))
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/server/session.py", line 157, in with_document_locked
    return func(*args, **kwargs)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/document/document.py", line 1076, in wrapper
    return doc._with_self_as_curdoc(invoke)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/document/document.py", line 1062, in _with_self_as_curdoc
    return f()
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/document/document.py", line 1075, in invoke
    return f(*args, **kwargs)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/document/document.py", line 852, in remove_then_invoke
    return callback(*args, **kwargs)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/holoviews/plotting/bokeh/callbacks.py", line 355, in process_on_change
    self.on_msg(msg)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/holoviews/plotting/bokeh/callbacks.py", line 104, in on_msg
    Stream.trigger(streams)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/holoviews/streams.py", line 154, in trigger
    subscriber(**dict(union))
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/holoviews/plotting/plot.py", line 535, in refresh
    self.update(stream_key)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/holoviews/plotting/plot.py", line 514, in update
    item = self.__getitem__(key)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/holoviews/plotting/plot.py", line 252, in __getitem__
    self.update_frame(frame)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/holoviews/plotting/bokeh/element.py", line 820, in update_frame
    self._update_glyphs(element, ranges)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/holoviews/plotting/bokeh/element.py", line 769, in _update_glyphs
    self._update_datasource(source, data)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/holoviews/plotting/bokeh/plot.py", line 213, in _update_datasource
    source.data.update(data)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/core/property/containers.py", line 346, in update
    descriptor._notify_mutated(owner, old, hint=hint)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/core/property/descriptors.py", line 833, in _notify_mutated
    self._real_set(obj, old, value, hint=hint)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/core/property/descriptors.py", line 796, in _real_set
    self._trigger(obj, old, value, hint=hint, setter=setter)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/core/property/descriptors.py", line 873, in _trigger
    obj.trigger(self.name, old, value, hint, setter)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/model.py", line 549, in trigger
    super(Model, self).trigger(attr, old, new, hint=hint, setter=setter)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/util/callback_manager.py", line 120, in trigger
    self._document._notify_change(self, attr, old, new, hint, setter, invoke)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/document/document.py", line 941, in _notify_change
    self._trigger_on_change(event)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/document/document.py", line 1049, in _trigger_on_change
    self._with_self_as_curdoc(invoke_callbacks)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/document/document.py", line 1062, in _with_self_as_curdoc
    return f()
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/document/document.py", line 1048, in invoke_callbacks
    cb(event)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/document/document.py", line 614, in <lambda>
    self._callbacks[receiver] = lambda event: event.dispatch(receiver)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/document/events.py", line 195, in dispatch
    super(ModelChangedEvent, self).dispatch(receiver)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/document/events.py", line 77, in dispatch
    receiver._document_patched(self)
  File "/Users/jonathan.dan/anaconda3/envs/flask/lib/python3.6/site-packages/bokeh/server/session.py", line 180, in _document_patched
    raise RuntimeError("_pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes")
RuntimeError: _pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes

[danjjl]

Any ideas on how I could get this small example to work?

[philippjfr]

Sorry for the delay, I'll have a look. Seems like an important issue.

[philippjfr]

@danjjl I'd recommend creating the DynamicMap inside the callback, otherwise all your users will share the same stream instances which means that whenever one user zooms all sessions will receive the updates. Since this causes bad interactions between threads you end up with that error. Here's how you can rewrite it to make it multi-user friendly:

import numpy as np
import holoviews as hv
import holoviews.plotting.bokeh
from holoviews.operation.datashader import datashade

from bokeh.application.handlers import FunctionHandler
from bokeh.application import Application

renderer = hv.renderer('bokeh')

def sine(frequency, phase, amplitude):
    xs = np.linspace(0, np.pi*4)
    return hv.Curve((xs, np.sin(frequency*xs+phase)*amplitude)).options(width=800)

# Replacing by next line works fine for multiple users.
# dmap = hv.DynamicMap(sine, kdims=['frequency', 'phase', 'amplitude']).redim.range(**ranges)

def modify_doc(doc):
    dmap = datashade(sine(1, 0, 1))
    renderer.server_doc(dmap, doc=doc)

handler = FunctionHandler(modify_doc)
app = Application(handler)

from bokeh.server.server import Server
server = Server({'/': app}, port=0)
server.start()
server.show('/')

from tornado.ioloop import IOLoop
loop = IOLoop.current()
loop.start()

[philippjfr]

I'll leave this issue open since this needs to be documented somewhere.

[danjjl]

Thank you very much!

This is just what I needed.

(not always easy to find my way through the docs and available workflows when starting with holoviews/datashader :) )

[dubovikmaster]

I am facing the same problem, only I am using panel. serve to create a server.

def callback():
      return some_hv_object

dmap = hv.DynamicMap(callback)
rastered = rasterize(dmap)
pn.serve(rastered, port=5000, show=False)

I actually get an interactive graph by tapping on localhost:5000. But only one user can view and change it. How can I fix this behavior?