Error executing org-mode code block using remote server session

[ricardog]

When I try to execute an org-mode code block using a remote server session I get the following error

jupyter-ioloop-alive-p: Wrong type argument: jupyter-ioloop, unbound, ioloop

The kernel starts on the server and I can connect a REPL to it. Every time I try to execute the source block a new kernel starts. I can see them by running jupyter-server-list-kernels.

Emacs is running on macOS. The server runs in a container on Linux. I can successfully execute org-mode code blocks when I run Emacs on a different host.

I enabled jupyter--debug but it didn't print anything insightful (or maybe anything at all). Below is the emacs-lisp stack trace at the time of the error.

Emacs version GNU Emacs 26.3 (build 1, x86_64-apple-darwin17.7.0, Carbon Version 158 AppKit 1561.61) of 2020-05-04

Backtrace

Debugger entered--Lisp error: (wrong-type-argument jupyter-ioloop unbound ioloop)
  signal(wrong-type-argument (jupyter-ioloop unbound ioloop))
  jupyter-ioloop-alive-p(unbound)
  #f(compiled-function (comm) #<bytecode 0x450d1541>)(#<jupyter-server jupyter-server>)
  apply(#f(compiled-function (comm) #<bytecode 0x450d1541>) #<jupyter-server jupyter-server> nil)
  jupyter-comm-alive-p(#<jupyter-server jupyter-server>)
  #f(compiled-function (comm id) "Return non-nil if COMM has a WebSocket connection to a kernel with ID." #<bytecode 0x44f699ed>)(#<jupyter-server jupyter-server> "94a0e500-b7b9-4a7d-ab75-cc2ed4298949")
  apply(#f(compiled-function (comm id) "Return non-nil if COMM has a WebSocket connection to a kernel with ID." #<bytecode 0x44f699ed>) #<jupyter-server jupyter-server> "94a0e500-b7b9-4a7d-ab75-cc2ed4298949")
  jupyter-server-kernel-connected-p(#<jupyter-server jupyter-server> "94a0e500-b7b9-4a7d-ab75-cc2ed4298949")
  #f(compiled-function (comm) "Return non-nil if COMM can receive server events for its associated kernel." #<bytecode 0x44f6b521>)(#<jupyter-server-kernel-comm jupyter-server-kernel-comm>)
  apply(#f(compiled-function (comm) "Return non-nil if COMM can receive server events for its associated kernel." #<bytecode 0x44f6b521>) #<jupyter-server-kernel-comm jupyter-server-kernel-comm> nil)
  jupyter-comm-alive-p(#<jupyter-server-kernel-comm jupyter-server-kernel-comm>)
  #f(compiled-function (manager) "Start a websocket connection to MANAGER's kernel.\nMANAGER's COMM slot will be set to the `jupyter-comm-layer'\nreceiving events on the websocket when this method returns." #<bytecode 0x44f6d2c5>)(#<jupyter-server-kernel-manager jupyter-server-kernel-manager>)
  apply(#f(compiled-function (manager) "Start a websocket connection to MANAGER's kernel.\nMANAGER's COMM slot will be set to the `jupyter-comm-layer'\nreceiving events on the websocket when this method returns." #<bytecode 0x44f6d2c5>) #<jupyter-server-kernel-manager jupyter-server-kernel-manager> nil)
  jupyter-comm-start(#<jupyter-server-kernel-manager jupyter-server-kernel-manager>)
  #f(compiled-function (manager &rest ignore) "Ensure that the gateway can receive events from its kernel." #<bytecode 0x44f6d309>)(#<jupyter-server-kernel-manager jupyter-server-kernel-manager>)
  apply(#f(compiled-function (manager &rest ignore) "Ensure that the gateway can receive events from its kernel." #<bytecode 0x44f6d309>) #<jupyter-server-kernel-manager jupyter-server-kernel-manager>)
  #f(compiled-function (&rest cnm-args) #<bytecode 0x5158476d>)()
  #f(compiled-function (kernel &rest args) "Error when KERNEL is already alive, otherwise call the next method." #<bytecode 0x4423dc49>)(#f(compiled-function (&rest cnm-args) #<bytecode 0x5158476d>) #<jupyter-server-kernel-manager jupyter-server-kernel-manager>)
  apply(#f(compiled-function (kernel &rest args) "Error when KERNEL is already alive, otherwise call the next method." #<bytecode 0x4423dc49>) #f(compiled-function (&rest cnm-args) #<bytecode 0x5158476d>) #<jupyter-server-kernel-manager jupyter-server-kernel-manager>)
  #f(compiled-function (&rest args) #<bytecode 0x4527b7a1>)(#<jupyter-server-kernel-manager jupyter-server-kernel-manager>)
  apply(#f(compiled-function (&rest args) #<bytecode 0x4527b7a1>) #<jupyter-server-kernel-manager jupyter-server-kernel-manager> nil)
  jupyter-start-kernel(#<jupyter-server-kernel-manager jupyter-server-kernel-manager>)
  jupyter-server-start-new-kernel(#<jupyter-server jupyter-server> "python3" jupyter-org-client)
  jupyter-run-server-repl(#<jupyter-server jupyter-server> "python3" nil nil jupyter-org-client)
  #f(compiled-function (session kernel) #<bytecode 0x43f1c5a1>)(#s(org-babel-jupyter-server-session :name "/jpy:192.168.0.46#8888:/r2r2" :connect-repl-p nil) "python3")
  apply(#f(compiled-function (session kernel) #<bytecode 0x43f1c5a1>) (#s(org-babel-jupyter-server-session :name "/jpy:192.168.0.46#8888:/r2r2" :connect-repl-p nil) "python3"))
  #f(compiled-function (&rest args) #<bytecode 0x450d9d89>)(#s(org-babel-jupyter-server-session :name "/jpy:192.168.0.46#8888:/r2r2" :connect-repl-p nil) "python3")
  apply(#f(compiled-function (&rest args) #<bytecode 0x450d9d89>) (#s(org-babel-jupyter-server-session :name "/jpy:192.168.0.46#8888:/r2r2" :connect-repl-p nil) "python3"))
  #f(compiled-function (&rest cnm-args) #<bytecode 0x51835f35>)()
  #f(compiled-function (cl--cnm session kernel) "Rename the returned client's REPL buffer to include SESSION's name.\nAlso set `jupyter-include-other-output' to nil for the session so\nthat output produced by other clients do not get handled by the\nclient." #<bytecode 0x43f1c52d>)(#f(compiled-function (&rest cnm-args) #<bytecode 0x51835f35>) #s(org-babel-jupyter-server-session :name "/jpy:192.168.0.46#8888:/r2r2" :connect-repl-p nil) "python3")
  apply(#f(compiled-function (cl--cnm session kernel) "Rename the returned client's REPL buffer to include SESSION's name.\nAlso set `jupyter-include-other-output' to nil for the session so\nthat output produced by other clients do not get handled by the\nclient." #<bytecode 0x43f1c52d>) #f(compiled-function (&rest cnm-args) #<bytecode 0x51835f35>) (#s(org-babel-jupyter-server-session :name "/jpy:192.168.0.46#8888:/r2r2" :connect-repl-p nil) "python3"))
  #f(compiled-function (&rest args) #<bytecode 0x450d9db5>)(#s(org-babel-jupyter-server-session :name "/jpy:192.168.0.46#8888:/r2r2" :connect-repl-p nil) "python3")
  apply(#f(compiled-function (&rest args) #<bytecode 0x450d9db5>) #s(org-babel-jupyter-server-session :name "/jpy:192.168.0.46#8888:/r2r2" :connect-repl-p nil) "python3")
  org-babel-jupyter-initiate-client(#s(org-babel-jupyter-server-session :name "/jpy:192.168.0.46#8888:/r2r2" :connect-repl-p nil) "python3")
  org-babel-jupyter-initiate-session-by-key("/jpy:192.168.0.46#8888:/r2r2" ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type . value) (:results . "replace") (:exports . "code") (:cache . "no") (:noweb . "no") (:hlines . "no") (:tangle . "no") (:kernel . "python3") (:async . "no") (:session . "/jpy:192.168.0.46#8888:/r2r2")))
  org-babel-jupyter-initiate-session("/jpy:192.168.0.46#8888:/r2r2" ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type . value) (:results . "replace") (:exports . "code") (:cache . "no") (:noweb . "no") (:hlines . "no") (:tangle . "no") (:kernel . "python3") (:async . "no") (:session . "/jpy:192.168.0.46#8888:/r2r2")))
  org-babel-execute:jupyter-python("print('Hello world')" ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type . value) (:results . "replace") (:exports . "code") (:cache . "no") (:noweb . "no") (:hlines . "no") (:tangle . "no") (:kernel . "python3") (:async . "no") (:session . "/jpy:192.168.0.46#8888:/r2r2")))
  org-babel-execute-src-block(nil ("jupyter-python" "print('Hello world')" ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type . value) (:results . "replace") (:exports . "code") (:session . "/jpy:192.168.0.46#8888:/r2r2") (:async . "no") (:kernel . "python3") (:tangle . "no") (:hlines . "no") (:noweb . "no") (:cache . "no")) "" nil 926 "(ref:%s)"))
  org-ctrl-c-ctrl-c(nil)
  funcall-interactively(org-ctrl-c-ctrl-c nil)
  call-interactively(org-ctrl-c-ctrl-c nil nil)
  command-execute(org-ctrl-c-ctrl-c)

[ricardog]

This seems to be caused by using use-package for configuration. When I start emacs -q and evaluate the following

(require 'package)
(setq package-user-dir reg-package-dir)
(package-initialize)
(require 'use-package)
(use-package jupyter
  :ensure t
  :defer nil
  :requires org
  :init
  (setq jupyter-api-authentication-method 'token
        jupyter-eval-use-overlays t
        org-babel-default-header-args:jupyter-python '((:async . "yes")
                                                       (:kernel . "python3")))
  )

(org-babel-do-load-languages
 'org-babel-load-languages
 '((emacs-lisp . t)
   (python . t)
   (jupyter . t)
   ))

Execution of source code blocks fails with the error above. I only have the following packages in package-user-dir

  total used in directory 16 available 41466000
  drwxr-xr-x  31 ricardog  staff   992B Jul 22 09:38 .
  drwxr-xr-x   4 ricardog  staff   128B Jul 22 09:29 ..
  drwxr-xr-x   4 ricardog  staff   128B Jul 15 16:11 archives
  drwxr-xr-x   6 ricardog  staff   192B Jul 15 16:11 bind-key-20191110.416
  drwxr-xr-x  69 ricardog  staff   2.2K Jul 15 16:18 jupyter-20200417.1907
  drwxr-xr-x   6 ricardog  staff   192B Jul 15 16:15 simple-httpd-20191103.1446
  -rw-r--r--   1 ricardog  staff   154B Jul 15 16:14 spinner-1.7.3.signed
  -rw-r--r--   1 ricardog  staff   154B Jul 15 16:14 undo-tree-0.7.4.signed
  drwxr-xr-x  22 ricardog  staff   704B Jul 15 16:11 use-package-20200629.1856
  drwxr-xr-x   6 ricardog  staff   192B Jul 15 16:15 websocket-20200419.2124
  drwxr-xr-x  10 ricardog  staff   320B Jul 15 16:18 zmq-20200305.2345

However, when I start emacs -q and eval this bit of code

(add-to-list 'load-path "~/_emacs.d/.packages/26.3/elpa/jupyter-20200417.1907")
(add-to-list 'load-path "~/_emacs.d/.packages/26.3/elpa/simple-httpd-20191103.1446")
(add-to-list 'load-path "~/_emacs.d/.packages/26.3/elpa/websocket-20200419.2124")
(add-to-list 'load-path "~/_emacs.d/.packages/26.3/elpa/zmq-20200305.2345")
(load-library "jupyter")
(load-library "jupyter-tramp")
(load-library "org")
(setq jupyter-api-authentication-method 'token
      jupyter-eval-use-overlays t
      org-babel-default-header-args:jupyter-python '((:async . "yes")
                                                     (:kernel . "python3")))

(org-babel-do-load-languages
   'org-babel-load-languages
   '((emacs-lisp . t)
     (python . t)
     (jupyter . t)
     ))

Source code block execution works correctly.

Any thoughts on why use-package breaks jupyter-emacs?

Thanks.

[fritz-k]

I had the same problem with my running config, both of your emacs -q-versions worked fine though. Your second example lead me to add

(use-package jupyter-tramp)

to my config which solved this problem for me. Maybe that helps in your case too.

[ricardog]

Thanks for the pointer @fritz-k . Unfortunately that didn't solve the problem for me. I'll keep looking.

[fritz-k]

Too bad. Two other things I could imagine are the load order and the :requires org.

In your plain example the load order is jupyer -> jupter-tramp -> org in the use-package case it's (implicitly due to :requires org) org -> jupyter.

:requires simply doesn't load the package if the dependency isn't available, but doesn't change the load order.

Here's the config I use , maybe that helps:

(use-package jupyter
  :after (ob-jupyter ob-python)
  :config
  (setq jupyter-api-authentication-method 'token)
  (setq org-babel-default-header-args:jupyter-python '((:session . "/jpy::py")
                                                       (:kernel . "python3")
                                                       (:async . "yes")))
  (add-to-list 'org-structure-template-alist '("jp" . "src jupyter-python")))

(use-package jupyter-tramp)

(use-package ob
  :after (org)
  :config
  (setq org-confirm-babel-evaluate nil
      org-src-tab-acts-natively t
      org-src-preserve-indentation t)
  :custom
  (org-babel-load-languages '((emacs-lisp . t)
                              (latex . t)
                              (shell . t))))

(use-package ob-jupyter
  :after (ob))

(use-package ob-python
  :after (ob)
  :config
  (add-to-list 'org-structure-template-alist '("p" . "src python"))
  (setq org-babel-default-header-args:python '((:session . "py")
                                               (:kernel . "python3")
                                               (:async . "yes"))))

(I'm not 100% sure both these ob-* are required, but IRC they add the relevant languages to org-babel-load-languages)

[SebastianCallh]

I am also experiencing this error, but I'm using straight and use-package so the setup is somewhat different.

[littlehome-eugene]

I just had this error, and fixed by changing server port, it seems token is remembered by jupyter and I'm looking for ways to change it

[gmoutso]

I just tried using a different port and it did resolve it. I haven't tried changing my init config yet.