ZMQ subprocess error

[e-eight]

I recently re-installed emacs-jupyter, and now whenever I am trying to run a jupyter-python source block in Org-mode I am getting the following error:

error in process filter: zmq--subprocess-filter: Error in ZMQ subprocess: void-variable, (jupyter-channel)
error in process filter: Error in ZMQ subprocess: void-variable, (jupyter-channel)
cl--assertion-failed: Assertion failed: (process-live-p process)

This is with natively compiled emacs 28. With emacs 27, not natively compiled, it appears to be working properly.

[smsegal]

I'm experiencing the same problem with native-comp on emacs28. If anyone is experiencing the same error and using straight.el, you can disable native compilation for the jupyter package by specifying :no-native-compile t in your straight recipe. E.g

(use-package jupyter
  :straight (:no-native-compile t))

[mbarton98]

I was able to get this to work since I'm not using straight. (setq comp-deferred-compilation-deny-list (list "jupyter"))

[veprbl]

Should be fixed by #248

[Jonghyun-Yun]

I am unable to use emacs-jupyter in Emacs 28.0.91 with native-comp. Evaluating a jupyter-python block leads to the error below. I am using Doom Emacs.

Check for compatible module binary to download? (y or n) y
Downloading https://github.com/nnicandro/emacs-zmq/releases/download/v0.10.10/emacs-zmq-x86_64-apple-darwin17.4.0.tar.gz
Verifying sha256 signature of emacs-zmq-x86_64-apple-darwin17.4.0.tar.gz
Wrote /Users/yunj/doom-emacs/.local/straight/build-28.0.91/zmq/emacs-zmq-x86_64-apple-darwin17.4.0.tar.gz
Jupyter: (cl-assertion-failed (not enable-multibyte-characters))
org-babel-execute-src-block: No org-babel-execute function for jupyter-python!

[Panadestein]

Pardon the necrobumping. I think this bug is alive again:

Executing Jupyter-Python code block at position 114...
Starting python3 kernel process...done
Warning: Unknown type: zmq-socket
error in process filter: zmq--subprocess-filter: Error in ZMQ subprocess: (void-variable )
error in process filter: Error in ZMQ subprocess: (void-variable )
Warning: Unknown type: zmq-socket
Assertion failed: (process-live-p process)
finalizer failed: (cl-no-primary-method jupyter-kill-kernel #s(jupyter-kernel-process-manager (#<finalizer used>) jupyter--kernel-managers #s(jupyter-spec-kernel (#<finalizer> #<finalizer used>) ("python3" "/nix/store/pwr103hd66pfl4il84ldmljxc055wn9w-python3-3.10.9-env/share/jupyter/kernels/python3" :argv ["python" "-m" "ipykernel_launcher" "-f" "{connection_file}"] :env nil :display_name "Python 3 (ipykernel)" :language "python" :interrupt_mode "signal" :metadata (:debugger t)) #s(jupyter-session (:kernel_name "python3" :transport "tcp" :ip "127.0.0.1" :signature_scheme "hmac-sha256" :key "42fc60d6-c899-4771-b5cb-fc4c5082b4ae" :hb_port 33749 :stdin_port 45959 :control_port 40559 :shell_port 41345 :iopub_port 45179) "fa4e6e14-2019-4f8c-8fbb-53f1fcf94184" "42fc60d6-c899-4771-b5cb-fc4c5082b4ae") #<process jupyter-kernel-python3<1>>) #s(jupyter-zmq-channel :control #s(jupyter-session (:kernel_name "python3" :transport "tcp" :ip "127.0.0.1" :signature_scheme "hmac-sha256" :key "42fc60d6-c899-4771-b5cb-fc4c5082b4ae" :hb_port 33749 :stdin_port 45959 :control_port 40559 :shell_port 41345 :iopub_port 45179) "fa4e6e14-2019-4f8c-8fbb-53f1fcf94184" "42fc60d6-c899-4771-b5cb-fc4c5082b4ae") "tcp://127.0.0.1:40559" #<user-ptr ptr=0x9cda3e0 finalizer=0x7fa620e7fa10>)))

Full backtrace:

Debugger entered--Lisp error: (zmq-subprocess-error (void-variable ))
  signal(zmq-subprocess-error ((void-variable )))
  zmq--subprocess-filter(#<process zmq> "\nError: void-variable ()\n  debug-early-backtrace(...")
  accept-process-output(nil 0.25)
  jupyter-ioloop-wait-until(#<jupyter-zmq-channel-ioloop jupyter-zmq-channel-ioloop-27c1756> start identity)
  #f(compiled-function (arg1 arg2 &rest rest) #<bytecode 0x186427b2e0e2e74c>)(#<jupyter-zmq-channel-ioloop jupyter-zmq-channel-ioloop-27c1756> #f(compiled-function (event) #<bytecode 0x70a192d7e5ae66e>))
  apply(#f(compiled-function (arg1 arg2 &rest rest) #<bytecode 0x186427b2e0e2e74c>) #<jupyter-zmq-channel-ioloop jupyter-zmq-channel-ioloop-27c1756> #f(compiled-function (event) #<bytecode 0x70a192d7e5ae66e>))
  jupyter-ioloop-start(#<jupyter-zmq-channel-ioloop jupyter-zmq-channel-ioloop-27c1756> #f(compiled-function (event) #<bytecode 0x70a192d7e5ae66e>))
  #f(compiled-function (comm) #<bytecode -0xb25cd1cc32964dd>)(#<jupyter-channel-ioloop-comm jupyter-channel-ioloop-comm-229daee>)
  apply(#f(compiled-function (comm) #<bytecode -0xb25cd1cc32964dd>) #<jupyter-channel-ioloop-comm jupyter-channel-ioloop-comm-229daee> nil)
  jupyter-comm-start(#<jupyter-channel-ioloop-comm jupyter-channel-ioloop-comm-229daee>)
  #f(compiled-function (comm obj) #<bytecode 0x9adde364f45440a>)(#<jupyter-channel-ioloop-comm jupyter-channel-ioloop-comm-229daee> #<jupyter-org-client jupyter-org-client-229da26>)
  apply(#f(compiled-function (comm obj) #<bytecode 0x9adde364f45440a>) #<jupyter-channel-ioloop-comm jupyter-channel-ioloop-comm-229daee> #<jupyter-org-client jupyter-org-client-229da26>)
  #f(compiled-function (arg &rest args) #<bytecode -0xdee67ab66deb581>)(#<jupyter-channel-ioloop-comm jupyter-channel-ioloop-comm-229daee> #<jupyter-org-client jupyter-org-client-229da26>)
  apply(#f(compiled-function (arg &rest args) #<bytecode -0xdee67ab66deb581>) #<jupyter-channel-ioloop-comm jupyter-channel-ioloop-comm-229daee> #<jupyter-org-client jupyter-org-client-229da26> nil)
  jupyter-comm-add-handler(#<jupyter-channel-ioloop-comm jupyter-channel-ioloop-comm-229daee> #<jupyter-org-client jupyter-org-client-229da26>)
  #f(compiled-function (client) #<bytecode -0x1b75d991ee90683c>)(#<jupyter-org-client jupyter-org-client-229da26>)
  apply(#f(compiled-function (client) #<bytecode -0x1b75d991ee90683c>) #<jupyter-org-client jupyter-org-client-229da26> nil)
  jupyter-start-channels(#<jupyter-org-client jupyter-org-client-229da26>)
  jupyter-start-new-kernel("python3" jupyter-org-client)
  jupyter-run-repl("python3" nil nil jupyter-org-client)
  #f(compiled-function (session kernel) #<bytecode -0x43deee6f2e03c41>)(#s(org-babel-jupyter-session :name "cdwac") "python3")
  apply(#f(compiled-function (session kernel) #<bytecode -0x43deee6f2e03c41>) (#s(org-babel-jupyter-session :name "cdwac") "python3"))
  #f(compiled-function (&rest args) #<bytecode 0x19717a1d2f972332>)()
  #f(compiled-function (cl--cnm session kernel) #<bytecode 0xa911f1009b956f5>)(#f(compiled-function (&rest args) #<bytecode 0x19717a1d2f972332>) #s(org-babel-jupyter-session :name "cdwac") "python3")
  apply(#f(compiled-function (cl--cnm session kernel) #<bytecode 0xa911f1009b956f5>) #f(compiled-function (&rest args) #<bytecode 0x19717a1d2f972332>) (#s(org-babel-jupyter-session :name "cdwac") "python3"))
  #f(compiled-function (session kernel) "Rename the returned client's REPL buffer to include SESSION's name.\nAlso set `jupyter-include-other-output' to nil for the session so\nthat output produced by other clients do not get handled by the\nclient." #<bytecode 0x59c5e4364f225d6>)(#s(org-babel-jupyter-session :name "cdwac") "python3")
  apply(#f(compiled-function (session kernel) "Rename the returned client's REPL buffer to include SESSION's name.\nAlso set `jupyter-include-other-output' to nil for the session so\nthat output produced by other clients do not get handled by the\nclient." #<bytecode 0x59c5e4364f225d6>) #s(org-babel-jupyter-session :name "cdwac") "python3")
  org-babel-jupyter-initiate-client(#s(org-babel-jupyter-session :name "cdwac") "python3")
  org-babel-jupyter-initiate-session-by-key("cdwac" ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type . value) (:results . "replace") (:exports . "code") (:cache . "no") (:noweb . "no") (:hlines . "no") (:tangle . "no") (:kernel . "python3") (:async . "no") (:session . "cdwac")))
  org-babel-jupyter-initiate-session("cdwac" ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type . value) (:results . "replace") (:exports . "code") (:cache . "no") (:noweb . "no") (:hlines . "no") (:tangle . "no") (:kernel . "python3") (:async . "no") (:session . "cdwac")))
  org-babel-execute:jupyter-python("import numpy as np\nimport matplotlib\nimport matplo..." ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type . value) (:results . "replace") (:exports . "code") (:cache . "no") (:noweb . "no") (:hlines . "no") (:tangle . "no") (:kernel . "python3") (:async . "no") (:session . "cdwac")))
  org-babel-execute-src-block(nil ("jupyter-python" "import numpy as np\nimport matplotlib\nimport matplo..." ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type . value) (:results . "replace") (:exports . "code") (:session . "cdwac") (:async . "no") (:kernel . "python3") (:tangle . "no") (:hlines . "no") (:noweb . "no") (:cache . "no")) "" nil 114 "(ref:%s)"))
  org-ctrl-c-ctrl-c(nil)
  funcall-interactively(org-ctrl-c-ctrl-c nil)
  command-execute(org-ctrl-c-ctrl-c)

[nakkaya]

Pardon the necrobumping. I think this bug is alive again:

I can also confirm bug is live. Stopping emacs from compiling the package fixes the problem.

[havarddj]

Hi! I tried (setq native-comp-deferred-compilation-deny-list '("jupyter.el" "zmq.el")) but it doesn't seem to prevent it from compiling. What do you use? Running emacs 29 on macOS

[Panadestein]

For reference, I am using Emacs from master in NixOS, and emacs-jupyter with straight.el

[dakra]

@havarddj native-comp-deferred-compilation-deny-list is a list of regexps. So you probably want something like "zmq.*\\.el" etc.

[nakkaya]

@havarddj I was able to stop it from compiling using,

(add-to-list 'native-comp-deferred-compilation-deny-list ".*jupyter.*")

I don't have zmq on the list but worked for me. (Emacs 28.2 on OS X)

[havarddj]

RIght, seems like my problem is actually with zmq failing some tests. Thanks for the help!

[benneti]

@Panadestein were you able to fix it, I think I run into the exact same problem (also on nixos)

[Panadestein]

@benneti unfortunately, it is still broken for me. I have tried to prevent native compilation with straight, as suggested above:

(use-package jupyter
  :straight (:build (:not native-compile)))

But the problem persists. I still get the error:

Starting python3 kernel process...done
Warning: Unknown type: zmq-socket
error in process filter: zmq--subprocess-filter: Error in ZMQ subprocess: (void-variable )
error in process filter: Error in ZMQ subprocess: (void-variable )
Warning: Unknown type: zmq-socket

The other ways of preventing native compilation suggested in this issue do not have any effect on my config.

[benneti]

I tried this too, even cleaning out the elm in the nix store (they seem to be generated even with the build recipe). But it didn't help either but I am not sure whether I got all the eln files either.

[Panadestein]

Oh, you can go ahead and purge the build folder of jupyter, straight won't do it for you. Still, I have checked, and I don't have any .eln file, but the issue persists. How are you managing your Python environment in Nix? I am worried it could be a NixOS exclusive problem.

[benneti]

I use python.withPackages (globally installed in my user profile). On the other hand going back to emacs 28 fixes the problem, which in my opinion makes it equally likely that the problem is emacs 29.

[iskur]

@Panadestein The problem is not limited to NiXOS. I also experience it on ArchLinux with the emacs-git package from AUR (version 30).

[Panadestein]

@iskur @benneti nice to know it is not exclusive to NixOS. Well, I do not know how to debug this further, maybe someone else has an idea.

[havarddj]

@iskur @benneti @Panadestein Switching from emacs 29 to 28 fixed the problem on macOS, so it's likely the same problem.

Running make test in the emacs-zmq directory (after copying zmq-tests.el from the github repo) gave the following output:

emacs -nw -Q -batch -L . -l ert -l zmq-tests.el \
        --eval "(ert-run-tests-batch-and-exit)"
Loading /Users/havard/.emacs.d/elpa/zmq-20220510.1820/emacs-zmq.dylib (module)...
ZMQ Version: 4.3.1
Running 11 tests (2023-01-29 13:25:44+0000, selector ‘t’)
   passed   1/11  zmq-close-socket-once (0.002947 sec)
   passed   2/11  zmq-contexts (0.000047 sec)
   passed   3/11  zmq-encryption (0.000021 sec)
   passed   4/11  zmq-globrefs (0.235227 sec)
   passed   5/11  zmq-messages (0.001528 sec)
   passed   6/11  zmq-polling (2.018907 sec)
Test zmq-send-unicode backtrace:
  zmq-recv(#<user-ptr ptr=0x600003ba4140 finalizer=0x10b72a784> nil)
  zmq-recv-decoded(#<user-ptr ptr=0x600003ba4140 finalizer=0x10b72a784
  (setq s (zmq-recv-decoded s2 'utf-16))
  (let ((u "çπ§") (s nil)) (zmq-send s1 u) (setq s (zmq-recv s2)) (let
  (progn (let ((u "çπ§") (s nil)) (zmq-send s1 u) (setq s (zmq-recv s2
  (unwind-protect (progn (let ((u "çπ§") (s nil)) (zmq-send s1 u) (set
  (let* ((--cl-rest-- (zmq-create-bound-pair ctx zmq-PAIR zmq-PAIR nil
  (let* ((ctx (zmq-context))) (let* ((--cl-rest-- (zmq-create-bound-pa
  (let ((ert--infos (cons (cons "Info: " "Sending unicode messages") e
  (let ((ert--infos (cons (cons "Info: " "Unicode messages") ert--info
  (closure (t) nil (let ((ert--infos (cons (cons "Info: " "Unicode mes
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name zmq-send-unicode :documentation nil :
  ert-run-or-rerun-test(#s(ert--stats :selector t :tests ... :test-map
  ert-run-tests(t #f(compiled-function (event-type &rest event-args) #
  ert-run-tests-batch(nil)
  ert-run-tests-batch-and-exit()
  command-line-1(("-L" "." "-l" "ert" "-l" "zmq-tests.el" "--eval" "(e
  command-line()
  normal-top-level()
Test zmq-send-unicode condition:
    Info: Unicode messages
    Info: Sending unicode messages
    (wrong-type-argument utf-8-string-p "\376\377\0\0\247use")
   FAILED   7/11  zmq-send-unicode (0.002582 sec) at zmq-tests.el:185
Test zmq-sockets backtrace:
  signal(wrong-type-argument (utf-8-string-p "\376\377\0t\0\0t\0\0\0")
  apply(signal (wrong-type-argument (utf-8-string-p "\376\377\0t\0\0t\
  (setq value-332 (apply fn-330 args-331))
  (unwind-protect (setq value-332 (apply fn-330 args-331)) (setq form-
  (if (unwind-protect (setq value-332 (apply fn-330 args-331)) (setq f
  (let (form-description-334) (if (unwind-protect (setq value-332 (app
  (let ((value-332 'ert-form-evaluation-aborted-333)) (let (form-descr
  (let* ((fn-330 #'equal) (args-331 (condition-case err (let ((signal-
  (let ((topic "tést")) (zmq-socket-set-encoded sock zmq-ROUTING-ID to
  (let ((ert--infos (cons (cons "Info: " "Unicode options") ert--infos
  (let ((sock (zmq-socket ctx zmq-SUB))) (let ((ert--infos (cons (cons
  (let ((sock (zmq-socket ctx zmq-PULL))) (let ((ert--infos (cons (con
  (let ((ert--infos (cons (cons "Info: " "Socket options") ert--infos)
  (let* ((ctx (zmq-context)) (endpoint "tcp://127.0.0.1:5345") (s (zmq
  (closure (t) nil (let* ((ctx (zmq-context)) (endpoint "tcp://127.0.0
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name zmq-sockets :documentation nil :body 
  ert-run-or-rerun-test(#s(ert--stats :selector t :tests ... :test-map
  ert-run-tests(t #f(compiled-function (event-type &rest event-args) #
  ert-run-tests-batch(nil)
  ert-run-tests-batch-and-exit()
  command-line-1(("-L" "." "-l" "ert" "-l" "zmq-tests.el" "--eval" "(e
  command-line()
  normal-top-level()
Test zmq-sockets condition:
    Info: Socket options
    Info: Unicode options
    (wrong-type-argument utf-8-string-p "\376\377\0t\0\0t\0\0\0")
   FAILED   8/11  zmq-sockets (0.002320 sec) at zmq-tests.el:208
Test zmq-subprocess backtrace:
  signal(ert-test-failed (((should-error (setq proc (zmq-start-process
  ert-fail(((should-error (setq proc (zmq-start-process (lambda (_a _b
  (if errorp401 nil (ert-fail (append (funcall form-description-fn-402
  (let ((errorp401 nil) (form-description-fn-402 #'(lambda nil form-de
  (let (form-description-400) (let ((errorp401 nil) (form-description-
  (let ((value-399 (gensym "ert-form-evaluation-aborted-"))) (let (for
  (let ((ert--infos (cons (cons "Info: " "Only functions with 0 or 1 a
  (progn (let ((value-395 (gensym "ert-form-evaluation-aborted-"))) (l
  (unwind-protect (progn (let ((value-395 (gensym "ert-form-evaluation
  (let (proc) (unwind-protect (progn (let ((value-395 (gensym "ert-for
  (let ((ert--infos (cons (cons "Info: " "Validating sexp") ert--infos
  (closure (t) nil (let ((ert--infos (cons (cons "Info: " "Validating 
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name zmq-subprocess :documentation nil :bo
  ert-run-or-rerun-test(#s(ert--stats :selector t :tests ... :test-map
  ert-run-tests(t #f(compiled-function (event-type &rest event-args) #
  ert-run-tests-batch(nil)
  ert-run-tests-batch-and-exit()
  command-line-1(("-L" "." "-l" "ert" "-l" "zmq-tests.el" "--eval" "(e
  command-line()
  normal-top-level()
Test zmq-subprocess condition:
    Info: Validating sexp
    Info: Only functions with 0 or 1 arguments
    (ert-test-failed
     ((should-error
       (setq proc
         (zmq-start-process ...)))
      :form
      (setq proc
        (zmq-start-process
         #'(lambda ...)))
      :value #<process zmq> :fail-reason "did not signal an error"))
   FAILED   9/11  zmq-subprocess (0.000619 sec) at zmq-tests.el:332
   passed  10/11  zmq-utility (0.000200 sec)
   passed  11/11  zmq-wrong-object-type (0.000041 sec)

Ran 11 tests, 8 results as expected, 3 unexpected (2023-01-29 13:25:46+0000, 2.412057 sec)

3 unexpected results:
   FAILED  zmq-send-unicode
   FAILED  zmq-sockets
   FAILED  zmq-subprocess

make: *** [test] Error 1 ```

[benneti]

On latest master (i.e. bc3b78be4333bb7a51e262018f7e6dcded621c41 for jupyter and 00dea2f9f88cd3e723d3b909aea26313dd33f9c6 for zmq) it is working for me using the pgtk branch (apart from some json warnings).

[hvzzzz]

I tried rebuilding emacs-zmq.so running make in the emacs-zmq directory created by straight, in my case .emacs.d/.local/straight/build-29.0.91/zmq, it worked for me on Emacs 29.

[Necktwi]

it worked initially and then i rebooted the pc, dang! emacs version: 29.3