Remote servers: zmq-subprocess-error void-function

When trying to connect to kernels on remote notebook servers, I always get zmq-subprocess-error void-function. The server is in a docker container, on an EC2 instance.

emacs-jupyter works fantastic with a local kernel!

When running a notebook server via jupyter notebook --ip 0.0.0.0 --allow-root --no-browser:

I can connect fine in the browser
I can start a new remote kernels via emacs-jupyter
I can list kernels via emacs-jupyter
All tests pass in the zmq source dir (after downloading the test file)

However, I always get the same error when trying to connect to a kernel, for example if I press RET on a kernel listed in the buffer produced by jupyter-server-list-kernels. The full output with toggle-debug-on-error is under 'Error 1', below.

When killing the call to jupyter notebook ... and instead using jupyter kernel, and trying to connect via TRAMP syntax to a JSON file, I get Error 2.

Any ideas for me?

Package versions: zmq-20230214.36, jupyter-20230214.215 emacs-version: GNU Emacs 28.2 (build 1, x86_64-apple-darwin21.6.0, NS appkit-2113.60 Version 12.6.1 (Build 21G217)) of 2022-12-28. IIRC I didn't use native compilation.

Jupyter server info:

Selected Jupyter core packages...
IPython          : 7.34.0
ipykernel        : 6.12.1
ipywidgets       : 8.0.4
jupyter_client   : 7.4.9
jupyter_core     : 4.12.0
jupyter_server   : 1.23.5
jupyterlab       : 3.4.7
nbclient         : 0.7.2
nbconvert        : 7.2.9
nbformat         : 5.7.3
notebook         : 6.5.2
qtconsole        : 5.4.0
traitlets        : 5.9.0

Error 1:

Debugger entered--Lisp error: (zmq-subprocess-error void-function (nil))
  signal(zmq-subprocess-error (void-function (nil)))
  zmq--subprocess-filter(#<process zmq> "(error void-function (nil))")
  accept-process-output(nil 0.25)
  jupyter-ioloop-wait-until(#<jupyter-server-ioloop jupyter-server-ioloop-1fec9f67f430> start identity)
  #f(compiled-function (arg1 arg2 &rest rest) #<bytecode -0x1abb25ded472fdf9>)(#<jupyter-server-ioloop jupyter-server-ioloop-1fec9f67f430> #f(compiled-function (event) #<bytecode 0x1ada72949b509b07>))
  apply(#f(compiled-function (arg1 arg2 &rest rest) #<bytecode -0x1abb25ded472fdf9>) #<jupyter-server-ioloop jupyter-server-ioloop-1fec9f67f430> #f(compiled-function (event) #<bytecode 0x1ada72949b509b07>))
  jupyter-ioloop-start(#<jupyter-server-ioloop jupyter-server-ioloop-1fec9f67f430> #f(compiled-function (event) #<bytecode 0x1ada72949b509b07>))
  #f(compiled-function (comm) #<bytecode -0xdbdb7fd0264a18e>)(#<jupyter-server jupyter-server-1fec9f60346a>)
  apply(#f(compiled-function (comm) #<bytecode -0xdbdb7fd0264a18e>) #<jupyter-server jupyter-server-1fec9f60346a>)
  #f(compiled-function (&rest cnm-args) #<bytecode -0x84dc6af3e3cfea0>)()
  #f(compiled-function (cl--cnm comm) #<bytecode 0x10eec37be315cf46>)(#f(compiled-function (&rest cnm-args) #<bytecode -0x84dc6af3e3cfea0>) #<jupyter-server jupyter-server-1fec9f60346a>)
  apply(#f(compiled-function (cl--cnm comm) #<bytecode 0x10eec37be315cf46>) #f(compiled-function (&rest cnm-args) #<bytecode -0x84dc6af3e3cfea0>) #<jupyter-server jupyter-server-1fec9f60346a>)
  #f(compiled-function (&rest args) #<bytecode -0x159b7f0916a6a5fa>)(#<jupyter-server jupyter-server-1fec9f60346a>)
  apply(#f(compiled-function (&rest args) #<bytecode -0x159b7f0916a6a5fa>) #<jupyter-server jupyter-server-1fec9f60346a> nil)
  jupyter-comm-start(#<jupyter-server jupyter-server-1fec9f60346a>)
  #f(compiled-function (comm &rest ignore) "Register COMM to receive server events.\nIf SERVER receives events that have the same kernel ID as the\nkernel associated with COMM, then COMM's `jupyter-event-handler'\nwill receive those events." #<bytecode -0x18362b2c5cbdacda>)(#<jupyter-server-kernel-comm jupyter-server-kernel-comm-1fec9f644fca>)
  apply(#f(compiled-function (comm &rest ignore) "Register COMM to receive server events.\nIf SERVER receives events that have the same kernel ID as the\nkernel associated with COMM, then COMM's `jupyter-event-handler'\nwill receive those events." #<bytecode -0x18362b2c5cbdacda>) #<jupyter-server-kernel-comm jupyter-server-kernel-comm-1fec9f644fca> nil)
  jupyter-comm-start(#<jupyter-server-kernel-comm jupyter-server-kernel-comm-1fec9f644fca>)
  #f(compiled-function (manager) "Start a websocket connection to MANAGER's kernel.\nMANAGER's COMM slot will be set to the `jupyter-comm-layer'\nreceiving events on the websocket when this method returns." #<bytecode -0x134ffb6f8a935e4f>)(#<jupyter-server-kernel-manager jupyter-server-kernel-manager-1feca6fe755c>)
  apply(#f(compiled-function (manager) "Start a websocket connection to MANAGER's kernel.\nMANAGER's COMM slot will be set to the `jupyter-comm-layer'\nreceiving events on the websocket when this method returns." #<bytecode -0x134ffb6f8a935e4f>) #<jupyter-server-kernel-manager jupyter-server-kernel-manager-1feca6fe755c> nil)
  jupyter-comm-start(#<jupyter-server-kernel-manager jupyter-server-kernel-manager-1feca6fe755c>)
  #f(compiled-function (manager class &rest slots) #<bytecode -0xf8b54f488f84712>)(#f(compiled-function (&rest cnm-args) #<bytecode 0x6396485e6cbaefb>) #<jupyter-server-kernel-manager jupyter-server-kernel-manager-1feca6fe755c> jupyter-repl-client)
  apply(#f(compiled-function (manager class &rest slots) #<bytecode -0xf8b54f488f84712>) #f(compiled-function (&rest cnm-args) #<bytecode 0x6396485e6cbaefb>) (#<jupyter-server-kernel-manager jupyter-server-kernel-manager-1feca6fe755c> jupyter-repl-client))
  #f(compiled-function (&rest args) #<bytecode 0x1ed8f48b6663c234>)(#<jupyter-server-kernel-manager jupyter-server-kernel-manager-1feca6fe755c> jupyter-repl-client)
  apply(#f(compiled-function (&rest args) #<bytecode 0x1ed8f48b6663c234>) (#<jupyter-server-kernel-manager jupyter-server-kernel-manager-1feca6fe755c> jupyter-repl-client))
  #f(compiled-function (&rest args) #<bytecode -0x158969df9a4882aa>)(#<jupyter-server-kernel-manager jupyter-server-kernel-manager-1feca6fe755c> jupyter-repl-client)
  apply(#f(compiled-function (&rest args) #<bytecode -0x158969df9a4882aa>) #<jupyter-server-kernel-manager jupyter-server-kernel-manager-1feca6fe755c> jupyter-repl-client)
  jupyter-make-client(#<jupyter-server-kernel-manager jupyter-server-kernel-manager-1feca6fe755c> jupyter-repl-client)
  jupyter-connect-server-repl(#<jupyter-server jupyter-server-1fec9f60346a> "eab186f1-ae37-4a06-a21b-ec97c191ba73")
  jupyter-server-kernel-list-new-repl()
  funcall-interactively(jupyter-server-kernel-list-new-repl)
  call-interactively(jupyter-server-kernel-list-new-repl nil nil)
  command-execute(jupyter-server-kernel-list-new-repl)

Error 2:

Debugger entered--Lisp error: (error "Kernel did not respond to kernel-info request")
  signal(error ("Kernel did not respond to kernel-info request"))
  error("Kernel did not respond to kernel-info request")
  jupyter-kernel-info(#<jupyter-org-client jupyter-org-client-1fec9f641c18>)
  #f(compiled-function (client &optional repl-name associate-buffer display) #<bytecode 0x3897f914c3a6735>)(#<jupyter-org-client jupyter-org-client-1fec9f641c18> nil nil nil)
  apply(#f(compiled-function (client &optional repl-name associate-buffer display) #<bytecode 0x3897f914c3a6735>) (#<jupyter-org-client jupyter-org-client-1fec9f641c18> nil nil nil))
  #f(compiled-function (&rest args) #<bytecode 0x15d718fd21442c55>)(#<jupyter-org-client jupyter-org-client-1fec9f641c18> nil nil nil)
  apply(#f(compiled-function (&rest args) #<bytecode 0x15d718fd21442c55>) #<jupyter-org-client jupyter-org-client-1fec9f641c18> (nil nil nil))
  jupyter-bootstrap-repl(#<jupyter-org-client jupyter-org-client-1fec9f641c18> nil nil nil)
  jupyter-connect-repl("/ssh:REDACTED@host|docker:root@015d03a08ee6:/root/..." nil nil jupyter-org-client)
  #f(compiled-function (session kernel) "Initiate a client connected to a remote kernel process." #<bytecode -0x8aa945e55ec9b70>)(#s(org-babel-jupyter-remote-session :name "/ssh:REDACTED@host|docker:root@015d03a08ee6:/root/..." :connect-repl-p t) "python3")
  apply(#f(compiled-function (session kernel) "Initiate a client connected to a remote kernel process." #<bytecode -0x8aa945e55ec9b70>) (#s(org-babel-jupyter-remote-session :name "/ssh:redacted@host|docker:root@015d03a08ee6:/root/..." :connect-repl-p t) "python3"))
  #f(compiled-function (&rest args) #<bytecode -0x1355e2a81368f7ab>)(#s(org-babel-jupyter-remote-session :name "/ssh:REDACTED@host|docker:root@015d03a08ee6:/root/..." :connect-repl-p t) "python3")
  apply(#f(compiled-function (&rest args) #<bytecode -0x1355e2a81368f7ab>) (#s(org-babel-jupyter-remote-session :name "/ssh:redacted@host|docker:root@015d03a08ee6:/root/..." :connect-repl-p t) "python3"))
  #f(compiled-function (&rest cnm-args) #<bytecode 0x1f010df4c37e68>)()
  #f(compiled-function (cl--cnm session kernel) "Rename the returned client's REPL buffer to include SESSION's name.\nAlso set `jupyter-include-other-output' to nil for the session so\nthat output produced by other clients do not get handled by the\nclient." #<bytecode 0x2a0adbe19a0cb42>)(#f(compiled-function (&rest cnm-args) #<bytecode 0x1f010df4c37e68>) #s(org-babel-jupyter-remote-session :name "/ssh:redacted@host|docker:root@015d03a08ee6:/root/..." :connect-repl-p t) "python3")
  apply(#f(compiled-function (cl--cnm session kernel) "Rename the returned client's REPL buffer to include SESSION's name.\nAlso set `jupyter-include-other-output' to nil for the session so\nthat output produced by other clients do not get handled by the\nclient." #<bytecode 0x2a0adbe19a0cb42>) #f(compiled-function (&rest cnm-args) #<bytecode 0x1f010df4c37e68>) (#s(org-babel-jupyter-remote-session :name "/ssh:redacted@host|docker:root@015d03a08ee6:/root/..." :connect-repl-p t) "python3"))
  #f(compiled-function (&rest args) #<bytecode 0x573fe3dfc033526>)(#s(org-babel-jupyter-remote-session :name "/ssh:redacted@host|docker:root@015d03a08ee6:/root/..." :connect-repl-p t) "python3")
  apply(#f(compiled-function (&rest args) #<bytecode 0x573fe3dfc033526>) #s(org-babel-jupyter-remote-session :name "/ssh:redacted@host|docker:root@015d03a08ee6:/root/..." :connect-repl-p t) "python3")
  org-babel-jupyter-initiate-client(#s(org-babel-jupyter-remote-session :name "/ssh:redacted@host|docker:root@015d03a08ee6:/root/..." :connect-repl-p t) "python3")
  org-babel-jupyter-initiate-session-by-key("/ssh:redacted@host|docker:root@015d03a08ee6:/root/..." ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type . value) (:results . "replace") (:exports . "code") (:cache . "no") (:noweb . "no") (:hlines . "no") (:tangle . "no") (:kernel . "python3") (:async . "yes") (:session . "/ssh:redacted@host|docker:root@015d03a08ee6:/root/...")))
  org-babel-jupyter-initiate-session("/ssh:redacted@host|docker:root@015d03a08ee6:/root/..." ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type . value) (:results . "replace") (:exports . "code") (:cache . "no") (:noweb . "no") (:hlines . "no") (:tangle . "no") (:kernel . "python3") (:async . "yes") (:session . "/ssh:redacted@host|docker:root@015d03a08ee6:/root/...")))
  org-babel-execute:jupyter-python("4 + 4" ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type . value) (:results . "replace") (:exports . "code") (:cache . "no") (:noweb . "no") (:hlines . "no") (:tangle . "no") (:kernel . "python3") (:async . "yes") (:session . "/ssh:redacted@host|docker:root@015d03a08ee6:/root/...")))
  org-babel-execute-src-block(nil ("jupyter-python" "4 + 4" ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type . value) (:results . "replace") (:exports . "code") (:session . "/ssh:redacted@host|docker:root@015d03a08ee6:/root/...") (:async . "yes") (:kernel . "python3") (:tangle . "no") (:hlines . "no") (:noweb . "no") (:cache . "no")) "" nil 74 "(ref:%s)"))
  org-ctrl-c-ctrl-c(nil)
  funcall-interactively(org-ctrl-c-ctrl-c nil)
  call-interactively(org-ctrl-c-ctrl-c nil nil)
  command-execute(org-ctrl-c-ctrl-c)

emacs-jupyter / jupyter

Remote servers: zmq-subprocess-error void-function #445