ProgressBar issues warning message for long runs

[jeanbraun]

Hello @benbovy. When a long run is performed, use of the progress bar often results in the following warning statement being sent to output repeatedly:

IOPub message rate exceeded. The notebook server will temporarily stop sending output to the client in order to avoid crashing it. To change this limit, set the config variable --NotebookApp.iopub_msg_rate_limit.

Current values: NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec) NotebookApp.rate_limit_window=3.0 (secs)

[benbovy]

This is not an issue in xarray-simlab but an intentional limitation in jupyter notebook to prevent crashes if communication is too heavy between the notebook application backend and front-end.

I'm a bit surprised that it happens for long runs only. As this is a message rate limit, I would rather expect that it's because run steps (and thus progress bar updates) are too fast. I'm also surprised not seeing any similar issues reported either in tqdm or ipywidgets (although maybe https://github.com/voila-dashboards/voila/issues/534 and https://github.com/jupyter-widgets/ipywidgets/pull/2765 might be related). Do you have other things possibly sending messages to the front-end during a simulation (e.g., some prints in custom hook functions)?

Some possible workarounds:

• Try a different front-end for the progress bar, e.g., ProgressBar(frontend="console")

• Set a higher message rate limit, either when starting jupyter lab (or notebook) or in a config file:

jupyter lab --LabApp.iopub_msg_rate_limit=10000

jupyter notebook --NotebookApp.iopub_msg_rate_limit=10000

jupyter lab --generate-config (then edit the generated file to change the corresponding option, then restart jupyterlab).

[benbovy]

Other possible causes (not sure as I didn't see your notebook): you have a very big notebook with too many progress bars shown, or you run many simulations (with a progress bar) in a nested loop. In those cases, I think you are pushing to the limits what we can do with notebooks, and I would suggest performing the simulations using a Python script instead.

For the case of many simulation runs in a nested loop, you might want to use tqdm directly instead of xsimlab.monitoring.ProgressBar, e.g.,

from tqdm.auto import tqdm

for i in tqdm(range(10), desc='1st loop'):
    for j in tqdm(range(4), desc='2nd loop', leave=False):
        # ds.xsimlab.run(...)