search

jupyter-widgets / ipywidgets

Interactive Widgets for the Jupyter Notebook
https://ipywidgets.readthedocs.io
BSD 3-Clause "New" or "Revised" License
3.16k stars 950 forks source link

CONTROL_COMM_TIMEOUT too low to work for some notebooks #3633

Open hpvb opened 2 years ago

hpvb commented 2 years ago

Description

We are running some rather complicated notebooks using voila. They work fine when testing locally but when deployed to a remote server they stop working, and we see the following message in the browser log:

Failed to fetch ipywidgets through the "jupyter.widget.control" comm channel, fallback to fetching individual model state. Reason: Control comm did not respond in time

I eventually traced this down to the value of CONTROL_COMM_TIMEOUT, by manually changing this to a much higher value the notebooks work as expected. There doesn't appear to be a way to do this from configuration however.

Reproduce

Start a complicated notebook over an internet connection.

I set a breakpoint on the error being printed and worked my way backwards. Manually changed the 4e3 to 4e8 in the compiled javascript and the problem went away

Expected behavior

I expect the control channel to be retried, perhaps an exponential backoff to a larger value, or ideally a way to just increase the value as a site specific setting.

Context

Troubleshoot Output 
# jupyter troubleshoot
$PATH:
    /root/.local/bin
    /root/bin
    /usr/local/sbin
    /usr/local/bin
    /usr/sbin
    /usr/bin
    /sbin
    /bin

sys.path:
    /usr/local/bin
    /usr/lib64/python310.zip
    /usr/lib64/python3.10
    /usr/lib64/python3.10/lib-dynload
    /usr/local/lib64/python3.10/site-packages
    /usr/local/lib/python3.10/site-packages
    /usr/lib64/python3.10/site-packages
    /usr/lib/python3.10/site-packages

sys.executable:
    /usr/bin/python3

sys.version:
    3.10.8 (main, Oct 12 2022, 00:00:00) [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)]

platform.platform():
    Linux-5.17.11-300.fc36.x86_64-x86_64-with-glibc2.35

pip list:
    Package               Version
    --------------------- ---------
    anyio                 3.6.2
    appdirs               1.4.4
    argon2-cffi           21.3.0
    argon2-cffi-bindings  21.2.0
    asttokens             2.1.0
    attrs                 21.4.0
    Babel                 2.11.0
    backcall              0.2.0
    beautifulsoup4        4.11.0
    bleach                5.0.1
    boto3                 1.21.33
    botocore              1.24.33
    Bottleneck            1.3.2
    branca                0.5.0
    Brotli                1.0.9
    cachetools            5.0.0
    certifi               2021.10.8
    cffi                  1.15.0
    charset-normalizer    2.0.11
    click                 8.0.4
    click-plugins         1.1.1
    cligj                 0.7.2
    cryptography          36.0.0
    cycler                0.11.0
    dbus-python           1.2.18
    debugpy               1.6.3
    decorator             5.1.1
    defusedxml            0.7.1
    entrypoints           0.4
    et-xmlfile            1.0.1
    executing             1.2.0
    fastjsonschema        2.16.2
    Fiona                 1.8.21
    folium                0.13.0
    fonttools             4.29.1
    fs                    2.4.11
    fsspec                2022.10.0
    gcsfs                 0.6.2
    geopandas             0.11.1
    google-auth           2.14.0
    google-auth-oauthlib  0.4.1
    gpg                   1.17.0
    greenlet              1.1.2
    html5lib              1.1
    htmlmin               0.1.12
    idna                  3.3
    ImageHash             4.3.1
    ipykernel             6.17.1
    ipyleaflet            0.17.2
    ipython               8.6.0
    ipython-genutils      0.2.0
    ipywidgets            8.0.2
    jdcal                 1.4.1
    jedi                  0.18.1
    Jinja2                3.0.3
    jmespath              1.0.0
    joblib                1.2.0
    json5                 0.9.10
    jsonschema            4.17.0
    jupyter_client        7.4.1
    jupyter_core          5.0.0
    jupyter-server        1.23.2
    jupyterlab-pygments   0.2.2
    jupyterlab_server     2.16.3
    jupyterlab-widgets    3.0.3
    kiwisolver            1.4.1
    libcomps              0.1.18
    lxml                  4.7.1
    MarkupSafe            2.1.1
    matplotlib            3.5.3
    matplotlib-inline     0.1.6
    missingno             0.5.1
    mistune               2.0.4
    multimethod           1.9
    munch                 2.5.0
    munkres               1.1.2
    nbclassic             0.4.8
    nbclient              0.7.0
    nbconvert             7.2.5
    nbformat              5.7.0
    nest-asyncio          1.5.6
    networkx              2.8.8
    notebook              6.5.2
    notebook_shim         0.2.2
    numexpr               2.8.1
    numpy                 1.22.0
    oauthlib              3.0.2
    odfpy                 1.4.1
    olefile               0.46
    openpyxl              3.0.3
    packaging             21.3
    pandas                1.3.5
    pandas-datareader     0.10.0
    pandas-profiling      3.4.0
    pandocfilters         1.5.0
    parso                 0.8.3
    patsy                 0.5.3
    pexpect               4.8.0
    phik                  0.12.2
    pickleshare           0.7.5
    Pillow                9.1.0
    pip                   21.3.1
    platformdirs          2.5.4
    ply                   3.11
    prometheus-client     0.15.0
    prompt-toolkit        3.0.32
    psutil                5.9.4
    psycopg2              2.9.1
    ptyprocess            0.7.0
    pure-eval             0.2.2
    pyasn1                0.4.8
    pyasn1-modules        0.2.8
    pycairo               1.21.0
    pycparser             2.20
    pydantic              1.10.2
    Pygments              2.13.0
    PyJWT                 2.4.0
    PyMySQL               0.10.1
    pyparsing             2.4.7
    pyproj                3.4.0
    PyQt5                 5.15.6
    PyQt5-sip             12.9.0
    pyrsistent            0.19.2
    PySocks               1.7.1
    python-dateutil       2.8.2
    pytz                  2022.6
    PyWavelets            1.4.1
    PyYAML                6.0
    pyzmq                 24.0.1
    QtPy                  2.0.0
    requests              2.27.1
    requests-oauthlib     1.3.0
    rpm                   4.17.1
    rsa                   4.8
    s3transfer            0.5.1
    scipy                 1.8.1
    seaborn               0.12.1
    Send2Trash            1.8.0
    setuptools            59.6.0
    Shapely               1.8.5
    six                   1.16.0
    sniffio               1.3.0
    soupsieve             2.3.1
    SQLAlchemy            1.4.42
    stack-data            0.6.1
    statsmodels           0.13.5
    tables                3.7.0
    tabulate              0.8.10
    tangled-up-in-unicode 0.2.0
    terminado             0.17.0
    tinycss2              1.2.1
    tornado               6.2
    tqdm                  4.64.1
    traitlets             5.5.0
    traittypes            0.2.1
    typing_extensions     4.4.0
    urllib3               1.26.12
    visions               0.7.5
    voila                 0.4.0
    wcwidth               0.2.5
    webencodings          0.5.1
    websocket-client      1.4.2
    websockets            10.4
    widgetsnbextension    4.0.3
    xarray                2022.6.0
    xeus-python           0.14.3
    xeus-python-shell     0.5.0
    xlrd                  2.0.1
    XlsxWriter            3.0.3
    xlwt                  1.3.0
    xyzservices           2022.9.0
Command Line Output 
N/A
Browser Output 
Failed to fetch ipywidgets through the "jupyter.widget.control" comm channel, fallback to fetching individual model state. Reason: Control comm did not respond in time

If using JupyterLab

Installed Labextensions 
N/A
jasongrout commented 2 years ago

This is related to https://github.com/jupyter-widgets/ipywidgets/issues/3580

jasongrout commented 2 years ago

From our dev discussion in our dev meeting today:

  1. We should increase the timeout.
  2. We can look into an early reply from the control comm.
  3. We should make sure that when we abort this control comm, that we really do abort the codepath.
hpvb commented 2 years ago

Thank you! For now my local workaround works (a wonderful sed hack in the Dockerfile) but a proper fix would be very nice!

ollyhensby commented 1 year ago

Thank you! For now my local workaround works (a wonderful sed hack in the Dockerfile) but a proper fix would be very nice!

@hpvb, just wondering how you've hacked this for now as I am having the exact issue. How exactly are you editing the CONTROL_COMM_TIMEOUT value?

jgunstone commented 1 year ago

@jasongrout - sounds encouraging.

For some extra context, @ollyhensby and I are working on quite a large / complex application. It works fine in JupyterLab but we get this issue in Voila. To make matters worse - it only happens sometimes! Most of the time the app works fine, but on occasion it is blank on loading (i guess depending on usage). Would be great to see a fix.

maartenbreddels commented 1 year ago

Do you also see "Comm is already created" as I mention in https://github.com/jupyter-widgets/ipywidgets/issues/3580#issue-1357330645 ?

ollyhensby commented 1 year ago

Hi @maartenbreddels, I am getting this in the console log.

image

ollyhensby commented 1 year ago

Just to add @maartenbreddels, we've got around this issue for now by removing initial loading of some widgets and instead have them load when observed e.g. we have tabs where when a tab is clicked on, the widgets within it load upon the tab being selected.

Doing this we therefore load a lot less widgets upon initialising and avoid the error shown above.

maartenbreddels commented 1 year ago

This is voila 0.4.0, so that means from https://github.com/voila-dashboards/voila/blob/0.4.x/packages/voila/package.json

We are using https://github.com/jupyter-widgets/ipywidgets/blame/511663a56324cea5324f49a65ebe25e2f1b04d87/packages/base-manager/src/manager-base.ts#L382 which hasn't change for 2 years, so that is likely the code that you are running.

I think there is an issue in the protocol as it now is (I discussed this with @mariobuikhuizen a while ago when analyzing this because of a different bug). IIRC, we suggested back then (private communication) that we should change the protocol such that when the comm is opened, the kernel sends back an 'acknowledge' message first, after which we should ignore the timeout. Now we do not know if the kernel supports the control comm or not, and let the timeout logic assume that. After the acknowledge, we should just wait for the data.

@martinRenou IIRC there is a proposal somewhere to put this in the Comm protocol, is that right?

In Solara server we take a different approach, where we only start running the Python code when the client is connected, avoiding this issue.

maartenbreddels commented 1 year ago

Yes, that is a good workaround and idea in general. You might be interested in Reacton, which will automatically create and destroy widgets for you in such a situation. It does require investing a bit into learning to think like React (in Python)

itcarroll commented 10 months ago

The exact console warning that OP reports shows up for me when creating a new notebook in a new environment with only jupyterlab, jupyterlab-widgets, and their dependencies installed. No idea if the same cause, but it doesn't take a complicated notebook to get the warning.

Versions 
Chrome 120.0.6099.234
anyio                     4.2.0
argon2-cffi               23.1.0
argon2-cffi-bindings      21.2.0
arrow                     1.3.0
asttokens                 2.4.1
async-lru                 2.0.4
attrs                     23.2.0
Babel                     2.14.0
backcall                  0.2.0
beautifulsoup4            4.12.3
bleach                    6.1.0
certifi                   2023.11.17
cffi                      1.16.0
charset-normalizer        3.3.2
comm                      0.2.1
debugpy                   1.8.0
decorator                 5.1.1
defusedxml                0.7.1
exceptiongroup            1.2.0
executing                 2.0.1
fastjsonschema            2.19.1
fqdn                      1.5.1
idna                      3.6
importlib-metadata        7.0.1
importlib-resources       6.1.1
ipykernel                 6.29.0
ipython                   8.12.3
isoduration               20.11.0
jedi                      0.19.1
Jinja2                    3.1.3
json5                     0.9.14
jsonpointer               2.4
jsonschema                4.21.1
jsonschema-specifications 2023.12.1
jupyter_client            8.6.0
jupyter_core              5.7.1
jupyter-events            0.9.0
jupyter-lsp               2.2.2
jupyter_server            2.12.5
jupyter_server_terminals  0.5.1
jupyterlab                4.0.11
jupyterlab_pygments       0.3.0
jupyterlab_server         2.25.2
jupyterlab-widgets        3.0.9
MarkupSafe                2.1.3
matplotlib-inline         0.1.6
mistune                   3.0.2
nbclient                  0.9.0
nbconvert                 7.14.2
nbformat                  5.9.2
nest-asyncio              1.5.9
notebook_shim             0.2.3
overrides                 7.4.0
packaging                 23.2
pandocfilters             1.5.1
parso                     0.8.3
pexpect                   4.9.0
pickleshare               0.7.5
pip                       23.3.2
pkgutil_resolve_name      1.3.10
platformdirs              4.1.0
prometheus-client         0.19.0
prompt-toolkit            3.0.43
psutil                    5.9.8
ptyprocess                0.7.0
pure-eval                 0.2.2
pycparser                 2.21
Pygments                  2.17.2
python-dateutil           2.8.2
python-json-logger        2.0.7
pytz                      2023.3.post1
PyYAML                    6.0.1
pyzmq                     25.1.2
referencing               0.32.1
requests                  2.31.0
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rpds-py                   0.17.1
Send2Trash                1.8.2
setuptools                41.6.0
six                       1.16.0
sniffio                   1.3.0
soupsieve                 2.5
stack-data                0.6.3
terminado                 0.18.0
tinycss2                  1.2.1
tomli                     2.0.1
tornado                   6.4
traitlets                 5.14.1
types-python-dateutil     2.8.19.20240106
typing_extensions         4.9.0
uri-template              1.3.0
urllib3                   2.1.0
wcwidth                   0.2.13
webcolors                 1.13
webencodings              0.5.1
websocket-client          1.7.0
zipp                      3.17.0