Core dumped Operation not permitted (src/thread.cpp:122)

jcwchen commented 3 years ago

Description

This core dump error only happens in certain environment: using a docker image from binfmt to simulate aarch64 environment (manylinux2014_aarch64) on general Ubuntu. Two kinds of crash:

jupyter nbconvert --to notebook --inplace --execute xxx.ipynb The crash log:

2021-09-28T15:17:33.3669336Z [NbConvertApp] Converting notebook onnx/examples/xxx.ipynb to notebook
2021-09-28T15:17:37.0294397Z Operation not permitted (src/thread.cpp:122)
2021-09-28T15:17:37.4379426Z /usr/bin/bash: line 9:  2586 Aborted                 (core dumped) jupyter nbconvert --to notebook --inplace --execute onnx/examples/xxx.ipynb
2021-09-28T15:17:38.2560671Z ##[error]Process completed with exit code 134.

pytest xxx.ipynb The crash log:

2021-09-28T15:19:22.4754205Z Fatal Python error: Aborted
2021-09-28T15:19:22.4754611Z 
2021-09-28T15:19:22.4765058Z Current thread 0x00000055012e9410 (most recent call first):
2021-09-28T15:19:22.4883802Z   File "/ws/.env/lib/python3.6/site-packages/zmq/sugar/socket.py", line 83 in __init__
2021-09-28T15:19:22.4884993Z   File "/ws/.env/lib/python3.6/site-packages/zmq/sugar/context.py", line 264 in socket
2021-09-28T15:19:22.4886297Z   File "/ws/.env/lib/python3.6/site-packages/jupyter_client/connect.py", line 617 in _create_connected_socket
2021-09-28T15:19:22.4887687Z   File "/ws/.env/lib/python3.6/site-packages/jupyter_client/manager.py", line 268 in _connect_control_socket
2021-09-28T15:19:22.4889289Z   File "/ws/.env/lib/python3.6/site-packages/jupyter_client/manager.py", line 314 in _async_post_start_kernel
2021-09-28T15:19:22.4890498Z   File "/opt/python/cp36-cp36m/lib/python3.6/asyncio/tasks.py", line 180 in _step
2021-09-28T15:19:22.4891592Z   File "/ws/.env/lib/python3.6/site-packages/nest_asyncio.py", line 169 in step
2021-09-28T15:19:22.4892722Z   File "/opt/python/cp36-cp36m/lib/python3.6/asyncio/events.py", line 145 in _run
2021-09-28T15:19:22.4893744Z   File "/ws/.env/lib/python3.6/site-packages/nest_asyncio.py", line 100 in _run_once
2021-09-28T15:19:22.4894815Z   File "/ws/.env/lib/python3.6/site-packages/nest_asyncio.py", line 64 in run_until_complete
2021-09-28T15:19:22.4895930Z   File "/ws/.env/lib/python3.6/site-packages/jupyter_client/utils.py", line 23 in wrapped
2021-09-28T15:19:22.4897106Z   File "/ws/.env/lib/python3.6/site-packages/jupyter_client/manager.py", line 337 in _async_start_kernel
2021-09-28T15:19:22.4898380Z   File "/opt/python/cp36-cp36m/lib/python3.6/asyncio/tasks.py", line 180 in _step
2021-09-28T15:19:22.4899462Z   File "/ws/.env/lib/python3.6/site-packages/nest_asyncio.py", line 169 in step
2021-09-28T15:19:22.4900543Z   File "/opt/python/cp36-cp36m/lib/python3.6/asyncio/events.py", line 145 in _run
2021-09-28T15:19:22.4901638Z   File "/ws/.env/lib/python3.6/site-packages/nest_asyncio.py", line 100 in _run_once
2021-09-28T15:19:22.4902812Z   File "/ws/.env/lib/python3.6/site-packages/nest_asyncio.py", line 64 in run_until_complete
2021-09-28T15:19:22.4903993Z   File "/ws/.env/lib/python3.6/site-packages/jupyter_client/utils.py", line 23 in wrapped
2021-09-28T15:19:22.4905186Z   File "/ws/.env/lib/python3.6/site-packages/nbval/kernel.py", line 53 in start_new_kernel
2021-09-28T15:19:22.4906310Z   File "/ws/.env/lib/python3.6/site-packages/nbval/kernel.py", line 88 in __init__
2021-09-28T15:19:22.4907419Z   File "/ws/.env/lib/python3.6/site-packages/nbval/plugin.py", line 237 in setup
2021-09-28T15:19:22.4908560Z   File "/ws/.env/lib/python3.6/site-packages/_pytest/runner.py", line 449 in prepare
2021-09-28T15:19:22.4909901Z   File "/ws/.env/lib/python3.6/site-packages/_pytest/runner.py", line 150 in pytest_runtest_setup
2021-09-28T15:19:22.4911191Z   File "/ws/.env/lib/python3.6/site-packages/pluggy/_callers.py", line 39 in _multicall
2021-09-28T15:19:22.4912355Z   File "/ws/.env/lib/python3.6/site-packages/pluggy/_manager.py", line 80 in _hookexec
2021-09-28T15:19:22.4913496Z   File "/ws/.env/lib/python3.6/site-packages/pluggy/_hooks.py", line 265 in __call__
2021-09-28T15:19:22.4914618Z   File "/ws/.env/lib/python3.6/site-packages/_pytest/runner.py", line 255 in <lambda>
2021-09-28T15:19:22.4915988Z   File "/ws/.env/lib/python3.6/site-packages/_pytest/runner.py", line 311 in from_call
2021-09-28T15:19:22.4917196Z   File "/ws/.env/lib/python3.6/site-packages/_pytest/runner.py", line 255 in call_runtest_hook
2021-09-28T15:19:22.4918407Z   File "/ws/.env/lib/python3.6/site-packages/_pytest/runner.py", line 215 in call_and_report
2021-09-28T15:19:22.4919633Z   File "/ws/.env/lib/python3.6/site-packages/_pytest/runner.py", line 120 in runtestprotocol
2021-09-28T15:19:22.4920928Z   File "/ws/.env/lib/python3.6/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
2021-09-28T15:19:22.4922140Z   File "/ws/.env/lib/python3.6/site-packages/pluggy/_callers.py", line 39 in _multicall
2021-09-28T15:19:22.4923314Z   File "/ws/.env/lib/python3.6/site-packages/pluggy/_manager.py", line 80 in _hookexec
2021-09-28T15:19:22.4924455Z   File "/ws/.env/lib/python3.6/site-packages/pluggy/_hooks.py", line 265 in __call__
2021-09-28T15:19:22.4925609Z   File "/ws/.env/lib/python3.6/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
2021-09-28T15:19:22.4926814Z   File "/ws/.env/lib/python3.6/site-packages/pluggy/_callers.py", line 39 in _multicall
2021-09-28T15:19:22.4928000Z   File "/ws/.env/lib/python3.6/site-packages/pluggy/_manager.py", line 80 in _hookexec
2021-09-28T15:19:22.4988190Z   File "/ws/.env/lib/python3.6/site-packages/pluggy/_hooks.py", line 265 in __call__
2021-09-28T15:19:22.5042380Z   File "/ws/.env/lib/python3.6/site-packages/_pytest/main.py", line 323 in _main
2021-09-28T15:19:22.5043564Z   File "/ws/.env/lib/python3.6/site-packages/_pytest/main.py", line 269 in wrap_session
2021-09-28T15:19:22.5044762Z   File "/ws/.env/lib/python3.6/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
2021-09-28T15:19:22.5045961Z   File "/ws/.env/lib/python3.6/site-packages/pluggy/_callers.py", line 39 in _multicall
2021-09-28T15:19:22.5047119Z   File "/ws/.env/lib/python3.6/site-packages/pluggy/_manager.py", line 80 in _hookexec
2021-09-28T15:19:22.5048255Z   File "/ws/.env/lib/python3.6/site-packages/pluggy/_hooks.py", line 265 in __call__
2021-09-28T15:19:22.5049362Z   File "/ws/.env/lib/python3.6/site-packages/_pytest/config/__init__.py", line 163 in main
2021-09-28T15:19:22.5050530Z   File "/ws/.env/lib/python3.6/site-packages/_pytest/config/__init__.py", line 185 in console_main
2021-09-28T15:19:22.5051334Z   File "/ws/.env/bin/pytest", line 8 in <module>
2021-09-28T15:19:22.8454387Z /usr/bin/bash: line 5:    60 Aborted                 (core dumped) pytest xxx.ipynb

I have tried different version combinations of jupyter-related tools (including nbconvert) and pytest, but none of them helped. This crash only happens in the latest docker image from https://github.com/tonistiigi/binfmt. Previous version of docker image still works fine. I guess some system dependencies might be missing in the latest environment, but I have no clue from the crash log...

I have also reached out the docker image's maintainer to see whether he senses this crash. Meanwhile, I have a question to jupyter/nbconvert: how does this core dump imply? How can resolve it if it happens in certain environment?

Thank you for the assistance!

Reproduce

Running jupyter notebook commands in a docker environment binfmt on Ubuntu.

Expected behavior

pytest with .ipynb and jupyter nbconvert should work well without crash.

Context

Jupyter nbconvert version: 6.2.0 (old versions do not work either)

SylvainCorlay commented 3 years ago

Hey @jcwchen I think this is due to an issue in your ZeroMQ build, and not with nbconvert.

Are you able to open the notebook and run code in this environment?

jcwchen commented 3 years ago

Are you able to open the notebook and run code in this environment?

I bumped into this issue in a CI pipeline so actually I don't know the answer, but I guess the answer is probably not since all functions related to notebook seem broken (nbconvert, pytest .ipynb).

Or, perhaps naive question, do you know how I can run/test a notebook without opening a browser and server? (like only command lines to achieve that) Thank you for your help @SylvainCorlay

jupyter / nbconvert