Kernel crashes due to upstream problems with zmq package (seen with usage of bokeh and similar packages)

[uwezi]

Environment data

• VS Code version: 1.67.2
• Jupyter Extension version (available under the Extensions sidebar): v2022.4.1021342353
• Python Extension version (available under the Extensions sidebar): v2022.6.3
• OS (Windows | Mac | Linux distro) and version: Windows 10 64 bit
• Python and/or Anaconda version: winpython 3.9.8
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
• Jupyter server running: Local

Expected behaviour

flawless execution of example notebook from bqplot

Actual behaviour

crashing of the IDE

Steps to reproduce:

download "interaction layer.ipynb" from https://github.com/bqplot/bqplot/tree/master/examples/Interactions

Logs

Issue Type: Bug

Opening an example script file for the bqplot library, such as "Interaction Layer.ipynb" from https://github.com/bqplot/bqplot/tree/master/examples/Interactions worked fine in the previous release of VSCode. Since upgrading on both my computer to the latest VSCode-release (April 2022) the script somehow crashes the IDE. When running the script, operation suddenly ceases and a prompt asks to save changes in the notebook file. Afterwards cell contents has been disappearing.

No change in the script files, nor in the python installation has been done simultaneously. Only change was the update of VSCode.

I observed this first yesterday on my stationary computer, today also my laptop updated itself to the new version of VSCode and now this behavior is persistent on both systems. Reinstallation of the jupyter-extension didn't help.

Extension version: 2022.4.1021342353 VS Code version: Code 1.67.2 (c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5, 2022-05-17T18:15:52.058Z) OS version: Windows_NT x64 10.0.19044 Restricted Mode: No

System Info

|Item|Value| |---|---| |CPUs|AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx (8 x 2096)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: disabled_off
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
oop_rasterization: enabled
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_renderer: enabled_on
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled| |Load (avg)|undefined| |Memory (System)|17.94GB (7.71GB free)| |Process Argv|--crash-reporter-id dfd4ad75-9b10-405c-836f-ded4c33df111| |Screen Reader|no| |VM|0%|

A/B Experiments

``` vsliv368:30146709 vsreu685:30147344 python383cf:30185419 vspor879:30202332 vspor708:30202333 vspor363:30204092 pythonvspyl392:30443607 pythontb:30283811 pythonvspyt551cf:30345471 pythonptprofiler:30281270 vshan820:30294714 vstes263:30335439 vscoreces:30445986 pythondataviewer:30285071 vscod805cf:30301675 binariesv615:30325510 bridge0708:30335490 bridge0723:30353136 vsaa593:30376534 vsc1dst:30438360 pythonvs932:30410667 wslgetstarted:30449410 vscscmwlcmt:30465135 cppdebug:30492333 vscaac:30438847 vsclangdf:30486550 ```

[IanMatthewHuff]

@uwezi Thanks for the report here. I actually repro the same thing. Checking into it now. I see this as the possible first error in the developer tools.

FATAL ERROR: Error::New napi_get_last_error_info 1: 0x118d09f85 node::FatalException(v8::Isolate*, v8::TryCatch const&) [/Applications/Visual Studio Code - Insiders.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework] 2: 0x118d0a12f node::OnFatalError(char const*, char const*) [/Applications/Visual Studio Code - Insiders.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework] 3: 0x118d09fa9 node::FatalException(v8::Isolate*, v8::TryCatch const&) [/Applications/Visual Studio Code - Insiders.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework] 4: 0x118cbb2ea napi_fatal_error [/Applications/Visual Studio Code - Insiders.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework] 5: 0x10d14f46d zmq::stream_listener_base_t::create_engine(int) [/Users/ianhuff/.vscode-insiders/extensions/ms-toolsai.jupyter-2022.6.1001611018/out/node_modules/zeromq/prebuilds/darwin-x64/electron.napi.node] 6: 0x10d14f20e zmq::stream_listener_base_t::create_engine(int) [/Users/ianhuff/.vscode-insiders/extensions/ms-toolsai.jupyter-2022.6.1001611018/out/node_modules/zeromq/prebuilds/darwin-x64/electron.napi.node] 7: 0x10d15fd9e napi_register_module_v1 [/Users/ianhuff/.vscode-insiders/extensions/ms-toolsai.jupyter-2022.6.1001611018/out/node_modules/zeromq/prebuilds/darwin-x64/electron.napi.node] 8: 0x10d168fb4 napi_register_module_v1 [/Users/ianhuff/.vscode-insiders/extensions/ms-toolsai.jupyter-2022.6.1001611018/out/node_modules/zeromq/prebuilds/darwin-x64/electron.napi.node] 9: 0x118cba4bb napi_is_detached_arraybuffer [/Applications/Visual Studio Code - Insiders.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]10: 0x11433ea2b v8::internal::Accessors::MakeAccessor(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Name>, void (*)(v8::Local<v8::Name>, v8::PropertyCallbackInfo<v8::Value> const&), void (*)(v8::Local<v8::Name>, v8::Local<v8::Value>, v8::PropertyCallbackInfo<v8::Boolean> const&)) [/Applications/Visual Studio Code - Insiders.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]11: 0x1207f0aa38 12: 0x12003bb998 

Not sure on the cause yet, but does seem to be coming from the Jupyter extension.

[rchiodo]

Looks like an out of memory error. I wonder if we're keeping messages for ipywidgets in memory by mistake. Might this be related to the work that Don was doing?

[DonJayamanne]

@IanMatthewHuff that looks like something that'll crash the node runtime (native node modules can bring down the extension host, in this case we're bringing down the entire VS Code UI) - think somehow our native modules are getting loaded in a context that it shouldn't.

[DonJayamanne]

Should test an older version of VS Code, could be due to a change in vscode (version of node shipped with VS Code).

[DonJayamanne]

@rchiodo

Might this be related to the work that Don was doing?

Which changes, I don't think I'm caching anything, can't remember, would like to look into it.

[rchiodo]

The code were we were not disposing of widgets correctly. Maybe that was in the renderer though.

[rchiodo]

We should try this again with Don's new changes.

[rchiodo]

This problem only occurs in the desktop ZMQ version. Works fine in web and remote and with ZMQ disabled.

Workaround then is to set this hidden setting (or use remote):

"jupyter.disableZMQSupport": true

[uwezi]

This problem only occurs in the desktop ZMQ version. Works fine in web and remote and with ZMQ disabled.

Workaround then is to set this hidden setting (or use remote):

"jupyter.disableZMQSupport": true

As an un-initiated user: where would I find this hidden setting?

[rchiodo]

As an un-initiated user: where would I find this hidden setting?

You can just type it into your global settings.json. The schema validation won't recognize it, but it will still allow you to type it.

[rchiodo]

Did some more investigation. This is a bug in ZMQ 6. If I switch our code to ZMQ 5, the problem goes away.

I think we have two choices:

• Fix the bug in ZMQ 6. This probably requires us building new node modules for ZMQ 6 and maybe shipping from a fork if the owners of ZMQ don't take our change.
• Move back to ZMQ 5. This has the problem of the node files in ZMQ 5 don't work with the latest version of electron. I had to run electron-rebuild to get them to work. We'd have to ship new node modules here too.

[rchiodo]

ZMQ 5 of our source is here: https://github.com/microsoft/vscode-jupyter/tree/rchiodo/try_zmq5

[IanMatthewHuff]

Ack, that's sticky...

NAPI support was our big reason for moving to ZMQ6. Electron-rebuild works for dev work, but I'm not even sure how we'd ship ZMQ5. VS Code versions update node as needed, extension can't just support a specific single version. I should look and see how others are dealing with this.

[rchiodo]

Perhaps we can rebuild the ZMQ5 code to use NAPI. Although seems easier to just fix the bug in 6 version.

[IanMatthewHuff]

Per the versions it honestly looks like ZMQ6 might be abandoned? Looks like it stalled out a years ago. https://www.npmjs.com/package/zeromq

Readme and guide pushed 6.0 for new projects, but it last released in 2019.

[rchiodo]

I think we might become the owners or next to update it then. @DonJayamanne is actually going to be building new bits for M1 support for ZMQ 6 based on the latest feedback on this:

5206

[IanMatthewHuff]

Yeah, I have zero clue on the work involved, but doesn't seem unreasonable for us to help with the next 6.0 release push if that's an option.

[rchiodo]

Here's the callstack of the exit process:

kernel32.dll!ExitProcessImplementation()    Unknown
    Code - Insiders.exe!exit_or_terminate_process(const unsigned int return_code) Line 143  C++
    Code - Insiders.exe!common_exit(const int return_code, const _crt_exit_cleanup_mode cleanup_mode, const _crt_exit_return_mode return_mode) Line 280 C++
    Code - Insiders.exe!node::Abort() Line 260  C++
    Code - Insiders.exe!node::OnFatalError(const char * location, const char * message) Line 450    C++
    Code - Insiders.exe!node::FatalError(const char * location, const char * message) Line 421  C++
    Code - Insiders.exe!napi_fatal_error(const char * location, unsigned __int64 location_len, const char * message, unsigned __int64 message_len) Line 763 C++
    electron.napi.node!00007ff94b9d2b09()   Unknown
    electron.napi.node!00007ff94b9e80f4()   Unknown
    electron.napi.node!00007ff94b9e8725()   Unknown
    electron.napi.node!00007ff94b9fd963()   Unknown
    electron.napi.node!00007ff94b9f48f3()   Unknown
    [Inline Frame] Code - Insiders.exe!v8impl::`anonymous namespace'::CallbackWrapperBase::InvokeCallback::<lambda_1>::operator()(napi_env__ * env) Line 312    C++

[rchiodo]

Ipyvolume and fileupload also crash this way.

[uwezi]

The problem persists in VS Code 1.68.1

[Bartel-C8]

Related to this (closed) issue, which was not reproducible:

https://github.com/microsoft/vscode-jupyter/issues/4822

[corbt]

Problem still exists in VS Code 1.72.2. Trying to upload a file using ipywidgets.FileUpload crashes VS Code.

[corbt]

This problem only occurs in the desktop ZMQ version. Works fine in web and remote and with ZMQ disabled.

Workaround then is to set this hidden setting (or use remote):

"jupyter.disableZMQSupport": true

This workaround appears to no longer work? When I enable this setting and run my notebook, I get the following error:

originalException: a [Error]: '_xsrf' argument missing from POST
      at Function.create (/home/ubuntu/.vscode-server/extensions/ms-toolsai.jupyter-2022.9.1202862440/out/node_modules/@jupyterlab/services.js:2:53387)

[aminya]

Could you try the latest beta version of zeromq to see if the issue is resolved?

https://github.com/zeromq/zeromq.js/releases

[brentkendrick]

Disabling ZMQ support fix works for me with VS Code version 1.77.3 using Windows.

[DonJayamanne]

Closing this issue as this hasn't been reported for a while.