search

holoviz / panel

Panel: The powerful data exploration & web app framework for Python
https://panel.holoviz.org
BSD 3-Clause "New" or "Revised" License
4.75k stars 517 forks source link

pyodide-worker raising error with pyodide 0.26.1 #7015

Closed MarcSkovMadsen closed 3 months ago

MarcSkovMadsen commented 3 months ago

I was trying to upgrade the pyodide and wasm code + documentation to latest version. I found that --pyodide-worker will not work with latest version of pyodide 0.26.1 and panel 1.5.0b2

Reproduce

See the error in the browser console

image

PythonError: Traceback (most recent call last):
  File "/lib/python312.zip/_pyodide/_base.py", line 574, in eval_code_async
    await CodeRunner(
  File "/lib/python312.zip/_pyodide/_base.py", line 394, in run_async
    coroutine = eval(self.code, globals, locals)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python312.zip/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/lib/python3.12/site-packages/panel/util/parameters.py", line 51, in edit_readonly
    yield
  File "<exec>", line 8, in <module>
  File "/lib/python3.12/site-packages/param/parameterized.py", line 2319, in update
    restore = dict(self_._update(arg, **kwargs))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/param/parameterized.py", line 2352, in _update
    self_._batch_call_watchers()
  File "/lib/python312.zip/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/lib/python3.12/site-packages/param/parameterized.py", line 284, in _batch_call_watchers
    yield
  File "/lib/python3.12/site-packages/param/parameterized.py", line 2546, in _batch_call_watchers
    self_._execute_watcher(watcher, events)
  File "/lib/python3.12/site-packages/param/parameterized.py", line 2506, in _execute_watcher
    watcher.fn(*args, **kwargs)
  File "/lib/python3.12/site-packages/panel/reactive.py", line 412, in _param_change
    self._apply_update(named_events, properties, model, ref)
  File "/lib/python312.zip/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/lib/python3.12/site-packages/panel/io/document.py", line 375, in unlocked
    yield
  File "/lib/python3.12/site-packages/panel/reactive.py", line 316, in _apply_update
    self._update_model(events, msg, root, model, doc, comm)
  File "/lib/python3.12/site-packages/panel/reactive.py", line 344, in _update_model
    model.update(**msg)
  File "/lib/python3.12/site-packages/bokeh/core/has_props.py", line 483, in update
    setattr(self, k, v)
  File "/lib/python3.12/site-packages/bokeh/core/has_props.py", line 336, in __setattr__
    return super().__setattr__(name, value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/bokeh/core/property/descriptors.py", line 332, in __set__
    self._set(obj, old, value, setter=setter)
  File "/lib/python3.12/site-packages/bokeh/core/property/descriptors.py", line 620, in _set
    self._trigger(obj, old, value, hint=hint, setter=setter)
  File "/lib/python3.12/site-packages/bokeh/core/property/descriptors.py", line 698, in _trigger
    obj.trigger(self.name, old, value, hint, setter)
  File "/lib/python3.12/site-packages/bokeh/model/model.py", line 571, in trigger
    super().trigger(descriptor.name, old, new, hint=hint, setter=setter)
  File "/lib/python3.12/site-packages/bokeh/util/callback_manager.py", line 186, in trigger
    self.document.callbacks.notify_change(cast(Model, self), attr, old, new, hint, setter, invoke)
  File "/lib/python3.12/site-packages/bokeh/document/callbacks.py", line 251, in notify_change
    self.trigger_on_change(event)
  File "/lib/python3.12/site-packages/bokeh/document/callbacks.py", line 423, in trigger_on_change
    invoke_with_curdoc(doc, invoke_callbacks)
  File "/lib/python312.zip/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/lib/python3.12/site-packages/bokeh/io/doc.py", line 82, in patch_curdoc
    yield
  File "/lib/python3.12/site-packages/bokeh/document/callbacks.py", line 453, in invoke_with_curdoc
    return f()
           ^^^
  File "/lib/python3.12/site-packages/bokeh/document/callbacks.py", line 422, in invoke_callbacks
    cb(event)
  File "/lib/python3.12/site-packages/panel/io/pyodide.py", line 346, in pysync
    dispatch_fn(json_patch, pyodide.ffi.to_js(buffer_map), msg_id)
  File "https://mnr-jupyterhub.de-prod.dk/mt-uk/user/masma/vscode/proxy/8000/pyodide/script.js", line 4, in sendPatch
pyodide.ffi.JsException: DataCloneError: Failed to execute 'postMessage' on 'DedicatedWorkerGlobalScope': [object Map] could not be cloned.

    at new_error (pyodide.asm.js:10:9965)
    at pyodide.asm.wasm:0x16dbfd
    at pyodide.asm.wasm:0x177431
    at _PyEM_TrampolineCall_JS (pyodide.asm.js:10:125894)
    at pyodide.asm.wasm:0x1c2eaf
    at pyodide.asm.wasm:0x2c7c0f
    at pyodide.asm.wasm:0x20a884
    at pyodide.asm.wasm:0x1c359c
    at pyodide.asm.wasm:0x1c38ab
    at pyodide.asm.wasm:0x1c3929
    at pyodide.asm.wasm:0x29e95d
    at pyodide.asm.wasm:0x2a4f54
    at pyodide.asm.wasm:0x1c3a69
    at pyodide.asm.wasm:0x1c36d2
    at pyodide.asm.wasm:0x176a95
    at callPyObjectKwargs (pyodide.asm.js:10:64068)
    at Module.callPyObjectMaybePromising (pyodide.asm.js:10:65316)
    at wrapper (pyodide.asm.js:10:27006)
    at Cn.e.port1.onmessage (pyodide.asm.js:10:102154)
MarcSkovMadsen commented 3 months ago

I see the same issue when trying to run the latest version of pyscript with the py-editor. I.e. #6995 has not been solved completely.

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width,initial-scale=1.0">
    <script src="./mini-coi.js" scope="./"></script>
    <script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-3.5.0.js"></script>
    <script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.5.0.min.js"></script>
    <script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-tables-3.5.0.min.js"></script>
    <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/@holoviz/panel@1.5.0-b.2/dist/panel.min.js"></script>

    <link rel="stylesheet" href="https://pyscript.net/releases/2024.7.1/core.css">
    <script type="module" src="https://pyscript.net/releases/2024.7.1/core.js"></script>
  </head>
  <body>
    <div id="simple_app"></div>
    <script type="py-editor" config='{"packages": ["https://cdn.holoviz.org/panel/1.5.0-b.2/dist/wheels/bokeh-3.5.0-py3-none-any.whl", "https://cdn.holoviz.org/panel/1.5.0-b.2/dist/wheels/panel-1.5.0b2-py3-none-any.whl"]}'>
      import panel as pn

      pn.extension(sizing_mode="stretch_width")

      slider = pn.widgets.FloatSlider(start=0, end=10, name='Amplitude')

      def callback(new):
         return f'Amplitude is: {new}'

      pn.Row(slider, pn.bind(callback, slider)).servable(target='simple_app');
    </script>
  </body>
</html>

image

Error raised while processing Document events: DataCloneError: Failed to execute 'postMessage' on 'DedicatedWorkerGlobalScope': [object Map] could not be cloned.

Please also note the css files that cannot be loaded.

MarcSkovMadsen commented 3 months ago

Even if I set the pyodide interpreter to 0.25.0 it does not work. The app renders but nothing happens when I drag the slider.

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width,initial-scale=1.0">
    <script src="./mini-coi.js" scope="./"></script>
    <script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-3.5.0.js"></script>
    <script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.5.0.min.js"></script>
    <script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-tables-3.5.0.min.js"></script>
    <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/@holoviz/panel@1.5.0-b.2/dist/panel.min.js"></script>

    <link rel="stylesheet" href="https://pyscript.net/releases/2024.7.1/core.css">
    <script type="module" src="https://pyscript.net/releases/2024.7.1/core.js"></script>
  </head>
  <body>
    <script type="py-editor" config='{ "interpreter": "https://cdn.jsdelivr.net/pyodide/v0.25.0/full/pyodide.mjs", "packages": ["https://cdn.holoviz.org/panel/wheels/bokeh-3.5.0-py3-none-any.whl", "https://cdn.holoviz.org/panel/1.5.0-b.2/dist/wheels/panel-1.5.0b2-py3-none-any.whl"]}'>
      import panel as pn

      pn.extension(sizing_mode="stretch_width")

      slider = pn.widgets.FloatSlider(start=0, end=10, name='Amplitude')

      def callback(new):
         return f'Amplitude is: {new}'

      pn.Row(slider, pn.bind(callback, slider)).servable(target='simple_app');
    </script>
    <div id="simple_app"></div>
  </body>
</html>
philippjfr commented 3 months ago

I suspect this is issue with the coincident implementation that PyScript uses to sync data between worker and frontend, or, equally likely, an issue with the way we use it.

@WebReflection would be very grateful for your input. The way I understand the error is that _link_docs_worker calls the dispatch_fn which is a JS function which is executed on the frontend. So behind the scenes coincident uses message passing to send the arguments from worker -> frontend, which in turn triggers the error:

pyodide.ffi.JsException: DataCloneError: Failed to execute 'postMessage' on 'DedicatedWorkerGlobalScope': [object Map] could not be cloned.

WebReflection commented 3 months ago

@philippjfr this issue starts with --pyodide-worker which we don't use at all ... then it shows error caused but network but it claims the error is some error that can't be dispatched ... I am a bit confused to be honest ... and Maps, even recursive one, can be cloned ... what kind of data needs to travel around? if that includes Blobs or Files or not supported data then I am afraid you need your own conversion there and the issue should be gone.

Quoting that module used to postMessage via buffers:

not supported yet: Blob, File, FileList, ImageBitmap, ImageData, and (raw) ArrayBuffer, but typed arrays are supported without major issues, but u/int8, u/int16, and u/int32 are the only safely suppored (right now).

WebReflection commented 3 months ago

@MarcSkovMadsen out of curiosity, does this work if you add just worker attribute and use <script type="py" ...> instead of py-editor as type? it could be that PyEditor is lacking behind some needed check/fix although I am sure this has been previously tested already too.

MarcSkovMadsen commented 3 months ago

Here is a summary @WebReflection

WebReflection commented 3 months ago

The TL;DR of the issue is that latest pyodide introduced LiteralMap to make it possible to deal with Pyodide converted values (the default .to_js()) as if these were plain objects so that these can be used out of the box with all APIs in the JS world that pretty rarely expect a Map instead of an object literal.

These conversions might be done somewhere else (either panle or bokeh in this case) and the transformer was not accounting for these kind of proxied maps so it was failing.

It was also not recursively check for these kind of proxied maps or other PyProxy values within arrays so now polyscript takes care of that and once it lands in PyScript and we release this issue whould be eventually gone.

WebReflection commented 3 months ago

FYI you should be able to test latest via https://cdn.jsdelivr.net/npm/@pyscript/core@0.5.1/dist/core.js for JS and https://cdn.jsdelivr.net/npm/@pyscript/core@0.5.1/dist/core.css for the CSS and all those demos should "just work".

MarcSkovMadsen commented 3 months ago

When using the versions mentioned above things work

Thanks @WebReflection.

Should a similar fix be applied to our own pyodide worker implementation @philippjfr ?

MarcSkovMadsen commented 3 months ago

One thing I notice is that the text is lagging behind the widget. I've never see that with a pure pyodide (worker) app. Is that a bug or a feature @WebReflection ?

https://github.com/user-attachments/assets/473e77b8-7e61-4321-8afe-542bdbb7cbdf

WebReflection commented 3 months ago

I would say "it's a feature" but the reality is this one:

I have observed a huge slow down with panel when running from a worker myself, which is why I have relaxed the deadlock warning when things are not super responsive, I would need to spend time to investigate panel internals to be sure where the bottleneck is, but I am afraid August won't be that best time to do so ... still you have a never blocking main thread responsive UI which is enough of a goal for our non worker optimized use cases (meaning, no special pyodide or module things needed) but I start wondering if, because pyodide has a worker specific build out there, if we should investigate what are the trades off or advantages of using such specific target for our needs.

I know I wrote a lot of blah-blah that maybe doesn't make much sense, but the long story short is that:

The sad story to tell though is that most WASM targeting PLs use ComLink instead of coincident behind the scene so it's hard for us to optimize for that implementation, but hear me when I say that we tried ComLink before in PyScript and it was a disaster for both unnecessary bloat and features + maintainability of the code, so I hope more WASM targeting PLs will have a look at all the things coincident enables with ease, and maybe target that implementation too.

philippjfr commented 3 months ago

Panel's worker implementation has explicit handling to avoid events from queuing up. Specifically it does the following:

Implementing this for the PyScript Worker paradigm is probably possible but I'm not quite sure how best to go about it.

WebReflection commented 3 months ago

@philippjfr that makes sense but then again, sliders a part, usually UI driven from workers with events forwarded down the pipe is fast enough but it cannot probably reach 60fps because as soon as the worker does some job it delays responses. I find it quite cool that despite that time spent in worker the ui keeps being responsive and the details get updated at the pace they can but I wonder if when a worker is meant one should use the panel-worker specific implementation instead of just panel ... has anyone tried that? I could try but I need a link to the wheel that provides such package.

MarcSkovMadsen commented 3 months ago

I will reopen this open as the there is no link to pull requests fixing the original issue which was for --pyodide-worker. The PRs linked are for pyscript and will not fix the original issue.

philippjfr commented 3 months ago

Fixed in https://github.com/holoviz/panel/pull/7016