Serverside + Backgroundmanager + Windows == Error

[noxthot]

With recent versions, the example from https://www.dash-extensions.com/transforms/serverside_output_transform#a-serversideoutputtransform works nicely on both, Windows and Ubuntu.

A slight modification introducing a background-manager with Diskcache plus Progress bar breaks the app on Windows, while it is still running on Ubuntu.

The code which is not working (also check diff before):

import time
import plotly.express as px
import diskcache

from dash import DiskcacheManager
from dash_extensions.enrich import DashProxy, Output, Input, State, Serverside, html, dcc, \
    ServersideOutputTransform

cache = diskcache.Cache("./cache")

# Background callbacks require a cache manager
background_callback_manager = DiskcacheManager(cache)

app = DashProxy(transforms=[ServersideOutputTransform()], background_callback_manager=background_callback_manager)
app.layout = html.Div(
    [
        html.Button("Query data", id="btn"),
        dcc.Dropdown(id="dd"),
        html.P("0%", id="progress-component"),
        dcc.Graph(id="graph"),
        dcc.Store(id="store"),
    ]
)

@app.callback(
    Output("store", "data"),
    Input("btn", "n_clicks"),
    background=True,
    progress=[
        Output("progress-component", "children"),
    ],
    prevent_initial_call=True
)
def query_data(set_progress, _):
    total = 3

    for i in range(total):
        # Simulating a process step
        time.sleep(1)  # emulate slow database operation

        set_progress([(i + 1) / total])  # Update progress using set_progress
    return Serverside(px.data.gapminder())  # no JSON serialization here

@app.callback(Output("dd", "options"),  Output("dd", "value"), Input("store", "data"), prevent_initial_call=True)
def update_dd(df):
    options = [{"label": column, "value": column} for column in df["year"]]   # no JSON de-serialization here
    return options, options[0]['value']

@app.callback(Output("graph", "figure"), [Input("dd", "value"), State("store", "data")], prevent_initial_call=True)
def update_graph(value, df):
    df = df.query("year == {}".format(value))  # no JSON de-serialization here
    return px.sunburst(df, path=["continent", "country"], values="pop", color="lifeExp", hover_data=["iso_alpha"])

if __name__ == "__main__":
    app.run_server()

Installed and updated packages (for completeness; note, that this is from a "larger" project): https://pastebin.com/56Nitkmk

Stacktrace (Microsoft Windows; Version 23H2 (Build 22631.3737); this is a Windows 11 version)

PS C:\Users\anonymized\anonymizedproject> pdm run .\drafts\tmp\test_serverside.py
Dash is running on http://127.0.0.1:8050/

 * Serving Flask app 'test_serverside'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:8050
Press CTRL+C to quit
[2024-06-21 12:17:48,303] ERROR in app: Exception on /_dash-update-component [POST]
Traceback (most recent call last):
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\flask\app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\flask\app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\flask\app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\flask\app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dash\dash.py", line 1373, in dispatch
    ctx.run(
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dash\_callback.py", line 389, in add_context
    job = callback_manager.call_job_fn(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dash\long_callback\managers\diskcache_manager.py", line 126, in call_job_fn
    proc.start()
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\multiprocess\process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\multiprocess\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\multiprocess\context.py", line 337, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\multiprocess\popen_spawn_win32.py", line 95, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\multiprocess\reduction.py", line 63, in dump
    ForkingPickler(file, protocol, *args, **kwds).dump(obj)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 420, in dump
    StockPickler.dump(self, obj)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 481, in dump
    self.save(obj)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 597, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 711, in save_reduce
    save(state)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 554, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 1217, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 966, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 990, in _batch_setitems
    save(v)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 554, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 1985, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 1117, in _save_with_postproc
    pickler.save_reduce(*reduction)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 686, in save_reduce
    save(args)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 554, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 881, in save_tuple
    save(element)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 554, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 1985, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 1112, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 990, in _batch_setitems
    save(v)
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "C:\Users\anonymized\AppData\Local\Programs\Python\Python312\Lib\pickle.py", line 572, in save
    rv = reduce(self.proto)
         ^^^^^^^^^^^^^^^^^^
TypeError: cannot pickle '_contextvars.ContextVar' object
127.0.0.1 - - [21/Jun/2024 12:17:48] "POST /_dash-update-component HTTP/1.1" 500 -
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\multiprocess\spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\multiprocess\spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 289, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\anonymized\anonymizedproject\.venv\Lib\site-packages\dill\_dill.py", line 444, in load
    obj = StockUnpickler.load(self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
EOFError: Ran out of input

Please note that using DiskCache Manager, background manager and progress bar without ServerSide works without problem.

Any idea what is going on? Thanks!

[noxthot]

Update: 1.0.17 shows the same behaviour.

[emilhe]

I would suggest to use WSL, if you are on Windows. I believe that would solve the issue?

[noxthot]

I suppose this should work and this is actually our plan B, but I hoped to see the same behaviour with both OS.

I am personally using Ubuntu; this error occurs on another dev's Windows machine. Will keep you updated.

[noxthot]

fyi: works when using WSL