search

jupyter / nbconvert

Jupyter Notebook Conversion
https://nbconvert.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
1.7k stars 562 forks source link

Deadlock combining ipywidgets.interact and IPython.display.Audio #2029

Open YannickJadoul opened 11 months ago

YannickJadoul commented 11 months ago

jupyter nbconvert --execute minimal_reproducer.ipynb --to html sometimes unpredictably hangs when executing the notebook.

I've managed to reduce the reproducing notebook to just a single cell, which combines ipywidgets.interact with a function which returns an IPython.display.Audio object as result of the function:

import glob
from IPython.display import Audio
import ipywidgets

def do_something(value):
    return Audio(value)

ipywidgets.interact(do_something, value=ipywidgets.Dropdown(options=sorted(glob.glob("audio/*.wav"))))

Some extra details:

[NbConvertApp] Searching ['/home/yannick/.jupyter', '/home/yannick/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[NbConvertApp] Looking for jupyter_config in /etc/jupyter
[NbConvertApp] Looking for jupyter_config in /usr/local/etc/jupyter
[NbConvertApp] Looking for jupyter_config in /usr/etc/jupyter
[NbConvertApp] Looking for jupyter_config in /home/yannick/.local/etc/jupyter
[NbConvertApp] Looking for jupyter_config in /home/yannick/.jupyter
[NbConvertApp] Looking for jupyter_nbconvert_config in /etc/jupyter
[NbConvertApp] Looking for jupyter_nbconvert_config in /usr/local/etc/jupyter
[NbConvertApp] Looking for jupyter_nbconvert_config in /usr/etc/jupyter
[NbConvertApp] Looking for jupyter_nbconvert_config in /home/yannick/.local/etc/jupyter
[NbConvertApp] Looking for jupyter_nbconvert_config in /home/yannick/.jupyter
[NbConvertApp] Converting notebook minimal_reproducer.ipynb to html
[NbConvertApp] Notebook name is 'minimal_reproducer'
[NbConvertApp] Template paths:
    /home/yannick/.local/share/jupyter/nbconvert/templates/lab
    /home/yannick/.local/share/jupyter/nbconvert/templates/base
    /home/yannick/.local/share/jupyter
    /home/yannick/.local/share/jupyter/nbconvert/templates
    /home/yannick/.local/share/jupyter/nbconvert/templates/compatibility
    /usr/local/share/jupyter
    /usr/local/share/jupyter/nbconvert/templates
    /usr/local/share/jupyter/nbconvert/templates/compatibility
    /usr/share/jupyter
    /usr/share/jupyter/nbconvert/templates
    /usr/share/jupyter/nbconvert/templates/compatibility
    /home/yannick/.local/share/jupyter/nbconvert/templates
[NbConvertApp] Applying preprocessor: TagRemovePreprocessor
[NbConvertApp] Applying preprocessor: RegexRemovePreprocessor
[NbConvertApp] Applying preprocessor: ExecutePreprocessor
[NbConvertApp] Instantiating kernel 'Python 3 (ipykernel)' with kernel provisioner: local-provisioner
[NbConvertApp] Starting kernel: ['/usr/bin/python3', '-m', 'ipykernel_launcher', '-f', '/tmp/tmp2cfc0f6d.json', '--HistoryManager.hist_file=:memory:']
[NbConvertApp] Connecting to: tcp://127.0.0.1:49369
[NbConvertApp] connecting iopub channel to tcp://127.0.0.1:51491
[NbConvertApp] Connecting to: tcp://127.0.0.1:51491
[NbConvertApp] connecting shell channel to tcp://127.0.0.1:41635
[NbConvertApp] Connecting to: tcp://127.0.0.1:41635
[NbConvertApp] connecting stdin channel to tcp://127.0.0.1:51223
[NbConvertApp] Connecting to: tcp://127.0.0.1:51223
[NbConvertApp] connecting heartbeat channel to tcp://127.0.0.1:34265
[NbConvertApp] connecting control channel to tcp://127.0.0.1:49369
[NbConvertApp] Connecting to: tcp://127.0.0.1:49369
[NbConvertApp] Executing cell:
import glob
from IPython.display import Audio
import ipywidgets

def do_something(value):
    return Audio(value)

ipywidgets.interact(do_something, value=ipywidgets.Dropdown(options=sorted(glob.glob("audio/*.wav"))))
[NbConvertApp] msg_type: status
[NbConvertApp] content: {'execution_state': 'busy'}
[NbConvertApp] msg_type: execute_input
[NbConvertApp] content: {'code': 'import glob\nfrom IPython.display import Audio\nimport ipywidgets\n\ndef do_something(value):\n    return Audio(value)\n\nipywidgets.interact(do_something, value=ipywidgets.Dropdown(options=sorted(glob.glob("audio/*.wav"))))', 'execution_count': 1}
[NbConvertApp] msg_type: comm_open
[NbConvertApp] content: {'data': {'state': {'_model_module': '@jupyter-widgets/base', '_model_module_version': '2.0.0', '_model_name': 'LayoutModel', '_view_count': None, '_view_module': '@jupyter-widgets/base', '_view_module_version': '2.0.0', '_view_name': 'LayoutView', 'align_content': None, 'align_items': None, 'align_self': None, 'border_bottom': None, 'border_left': None, 'border_right': None, 'border_top': None, 'bottom': None, 'display': None, 'flex': None, 'flex_flow': None, 'grid_area': None, 'grid_auto_columns': None, 'grid_auto_flow': None, 'grid_auto_rows': None, 'grid_column': None, 'grid_gap': None, 'grid_row': None, 'grid_template_areas': None, 'grid_template_columns': None, 'grid_template_rows': None, 'height': None, 'justify_content': None, 'justify_items': None, 'left': None, 'margin': None, 'max_height': None, 'max_width': None, 'min_height': None, 'min_width': None, 'object_fit': None, 'object_position': None, 'order': None, 'overflow': None, 'padding': None, 'right': None, 'top': None, 'visibility': None, 'width': None}, 'buffer_paths': []}, 'comm_id': 'd00e3f900d174410b119e9a62bf1f3d3', 'target_name': 'jupyter.widget', 'target_module': None}
[NbConvertApp] msg_type: comm_open
[NbConvertApp] content: {'data': {'state': {'_model_module': '@jupyter-widgets/controls', '_model_module_version': '2.0.0', '_model_name': 'DescriptionStyleModel', '_view_count': None, '_view_module': '@jupyter-widgets/base', '_view_module_version': '2.0.0', '_view_name': 'StyleView', 'description_width': ''}, 'buffer_paths': []}, 'comm_id': '1058e0c87a294d56af529cf2ca1b0ab2', 'target_name': 'jupyter.widget', 'target_module': None}
[NbConvertApp] msg_type: comm_open
[NbConvertApp] content: {'data': {'state': {'_dom_classes': [], '_model_module': '@jupyter-widgets/controls', '_model_module_version': '2.0.0', '_model_name': 'DropdownModel', '_options_labels': ['audio/1_b.wav', 'audio/1_y.wav', 'audio/2_b.wav', 'audio/2_y.wav', 'audio/3_b.wav', 'audio/3_y.wav', 'audio/4_b.wav', 'audio/4_y.wav', 'audio/5_b.wav', 'audio/5_y.wav', 'audio/bat.wav', 'audio/bet.wav', 'audio/the_north_wind_and_the_sun.wav'], '_view_count': None, '_view_module': '@jupyter-widgets/controls', '_view_module_version': '2.0.0', '_view_name': 'DropdownView', 'description': '', 'description_allow_html': False, 'disabled': False, 'index': 0, 'layout': 'IPY_MODEL_d00e3f900d174410b119e9a62bf1f3d3', 'style': 'IPY_MODEL_1058e0c87a294d56af529cf2ca1b0ab2', 'tabbable': None, 'tooltip': None}, 'buffer_paths': []}, 'comm_id': '1bb52825004a414aafc3100657a46e80', 'target_name': 'jupyter.widget', 'target_module': None}
[NbConvertApp] msg_type: comm_open
[NbConvertApp] content: {'data': {'state': {'_model_module': '@jupyter-widgets/base', '_model_module_version': '2.0.0', '_model_name': 'LayoutModel', '_view_count': None, '_view_module': '@jupyter-widgets/base', '_view_module_version': '2.0.0', '_view_name': 'LayoutView', 'align_content': None, 'align_items': None, 'align_self': None, 'border_bottom': None, 'border_left': None, 'border_right': None, 'border_top': None, 'bottom': None, 'display': None, 'flex': None, 'flex_flow': None, 'grid_area': None, 'grid_auto_columns': None, 'grid_auto_flow': None, 'grid_auto_rows': None, 'grid_column': None, 'grid_gap': None, 'grid_row': None, 'grid_template_areas': None, 'grid_template_columns': None, 'grid_template_rows': None, 'height': None, 'justify_content': None, 'justify_items': None, 'left': None, 'margin': None, 'max_height': None, 'max_width': None, 'min_height': None, 'min_width': None, 'object_fit': None, 'object_position': None, 'order': None, 'overflow': None, 'padding': None, 'right': None, 'top': None, 'visibility': None, 'width': None}, 'buffer_paths': []}, 'comm_id': '680f378b839a466f97ed0988e6037d26', 'target_name': 'jupyter.widget', 'target_module': None}
[NbConvertApp] msg_type: comm_open
[NbConvertApp] content: {'data': {'state': {'_dom_classes': ['widget-interact'], '_model_module': '@jupyter-widgets/controls', '_model_module_version': '2.0.0', '_model_name': 'VBoxModel', '_view_count': None, '_view_module': '@jupyter-widgets/controls', '_view_module_version': '2.0.0', '_view_name': 'VBoxView', 'box_style': '', 'children': [], 'layout': 'IPY_MODEL_680f378b839a466f97ed0988e6037d26', 'tabbable': None, 'tooltip': None}, 'buffer_paths': []}, 'comm_id': 'c01bc0355e13449a8550dc6979da258d', 'target_name': 'jupyter.widget', 'target_module': None}
[NbConvertApp] msg_type: comm_msg
[NbConvertApp] content: {'data': {'method': 'update', 'state': {'description': 'value'}, 'buffer_paths': []}, 'comm_id': '1bb52825004a414aafc3100657a46e80'}
[NbConvertApp] msg_type: comm_open
[NbConvertApp] content: {'data': {'state': {'_model_module': '@jupyter-widgets/base', '_model_module_version': '2.0.0', '_model_name': 'LayoutModel', '_view_count': None, '_view_module': '@jupyter-widgets/base', '_view_module_version': '2.0.0', '_view_name': 'LayoutView', 'align_content': None, 'align_items': None, 'align_self': None, 'border_bottom': None, 'border_left': None, 'border_right': None, 'border_top': None, 'bottom': None, 'display': None, 'flex': None, 'flex_flow': None, 'grid_area': None, 'grid_auto_columns': None, 'grid_auto_flow': None, 'grid_auto_rows': None, 'grid_column': None, 'grid_gap': None, 'grid_row': None, 'grid_template_areas': None, 'grid_template_columns': None, 'grid_template_rows': None, 'height': None, 'justify_content': None, 'justify_items': None, 'left': None, 'margin': None, 'max_height': None, 'max_width': None, 'min_height': None, 'min_width': None, 'object_fit': None, 'object_position': None, 'order': None, 'overflow': None, 'padding': None, 'right': None, 'top': None, 'visibility': None, 'width': None}, 'buffer_paths': []}, 'comm_id': 'e7f16b501e99438e9281b32bd3fd307a', 'target_name': 'jupyter.widget', 'target_module': None}
[NbConvertApp] msg_type: comm_open
[NbConvertApp] content: {'data': {'state': {'_dom_classes': [], '_model_module': '@jupyter-widgets/output', '_model_module_version': '1.0.0', '_model_name': 'OutputModel', '_view_count': None, '_view_module': '@jupyter-widgets/output', '_view_module_version': '1.0.0', '_view_name': 'OutputView', 'layout': 'IPY_MODEL_e7f16b501e99438e9281b32bd3fd307a', 'msg_id': '', 'outputs': [], 'tabbable': None, 'tooltip': None}, 'buffer_paths': []}, 'comm_id': '34df9a2e8fe5474cb1bf5256f2524a37', 'target_name': 'jupyter.widget', 'target_module': None}
[NbConvertApp] msg_type: comm_msg
[NbConvertApp] content: {'data': {'method': 'update', 'state': {'children': ['IPY_MODEL_1bb52825004a414aafc3100657a46e80', 'IPY_MODEL_34df9a2e8fe5474cb1bf5256f2524a37']}, 'buffer_paths': []}, 'comm_id': 'c01bc0355e13449a8550dc6979da258d'}
[NbConvertApp] msg_type: comm_msg
[NbConvertApp] content: {'data': {'method': 'update', 'state': {'msg_id': '1a83a999-dfbe126f90f6b626080d6ee7_108239_3'}, 'buffer_paths': []}, 'comm_id': '34df9a2e8fe5474cb1bf5256f2524a37'}
[NbConvertApp] msg_type: clear_output
[NbConvertApp] content: {'wait': True}
[NbConvertApp] msg_type: display_data
[NbConvertApp] content: {'data': {'text/plain': '<IPython.lib.display.Audio object>', 'text/html': '\n                <audio  controls="controls" >\n                    <source src="data:audio/x-wav;base64,..." type="audio/x-wav" />\n                    Your browser does not support the audio element.\n                </audio>\n              '}, 'metadata': {}, 'transient': {}}
[NbConvertApp] msg_type: comm_msg
[NbConvertApp] content: {'data': {'method': 'update', 'state': {'msg_id': ''}, 'buffer_paths': []}, 'comm_id': '34df9a2e8fe5474cb1bf5256f2524a37'}
[NbConvertApp] msg_type: display_data
[NbConvertApp] content: {'data': {'text/plain': "interactive(children=(Dropdown(description='value', options=('audio/1_b.wav', 'audio/1_y.wav', 'audio/2_b.wav'…", 'application/vnd.jupyter.widget-view+json': {'version_major': 2, 'version_minor': 0, 'model_id': 'c01bc0355e13449a8550dc6979da258d'}}, 'metadata': {}, 'transient': {}}
[NbConvertApp] msg_type: execute_result
[NbConvertApp] content: {'data': {'text/plain': '<function __main__.do_something(value)>'}, 'metadata': {}, 'execution_count': 1}
[NbConvertApp] msg_type: status
[NbConvertApp] content: {'execution_state': 'idle'}

TermeHansen commented 8 months ago

I often get similiar problems with plotly with nbconvert ending with the same content: {'execution_state': 'idle'} when using the --debug flag

and for me as well putting in some time.sleep(1) around seems to solve the issue. Clearly some race conditions taking place...

YannickJadoul commented 8 months ago

@TermeHansen I agree about the smell of race conditions, but I didn't manage to figure out a consistent solution of where to put sleep.

hjliu0206 commented 2 months ago

i have raise an merge request about the issue https://github.com/jupyter/nbclient/issues/312