Closed snunez1 closed 3 years ago
Pushed a fix. Let me know if it works.
No joy. Same error.
Did you git clone common-lisp-jupyter into ~/quicklisp/local-projects/
. It won't update in quicklisp for some time.
I did get fetch
into a directory on ASDF's load-path. It isn't ~/quicklisp/local-projects/
, but a location known to work when cloning projects. To be certain, I just checked the ~quicklisp/dists/software/
directory, and it's not there, and I'm certain this is the only common-lisp-jupyter
it could be loading.
I think I see what's going on here. In the local install, the pathname is hardcoded to local-projects
from a relative path (~/quicklisp). Perhaps this should use the system source directory as either the primary directory, or as an additional installer-path-part
. You can get the system source directory with (asdf:system-source-directory 'common-lisp-jupyter)
. That way, no matter where the source is installed you can obtain the system directory path, and it works across platforms too.
I don't think that will fix the issue with the system bundler.
Are you sure you want to use the bundler? It collects all the needed packages and deposits them into the system directory so there is no quicklisp dependency. It also depends on the version of common-lisp-jupyter in the quicklisp distribution, which could be why this is failing.
It's odd that my commit fixed it on my end. Can you verify that you got that the latest commit? If everything is there then the following should work.
(ql:quickload :common-lisp-jupyter)
(defvar a (make-instance 'j:system-bundle-installer))
(j::installer-path-part a :root)
I'm not exactly clear on what the best option is. That's part of the reason I opened up issue #64. I do want to use quicklisp and asdf from the jupyter notebook and any install option that makes that possible is fine.
Using the instructions above I get the same error. I'm relatively certain this is because my common-lisp-jupyter
is not in any of the standard ASDF nor quicklisp locations, and it seems that cl-jupyter may be looking for specific locations. All source is on a NAS drive and the paths for loading have been adjusted accordingly; nothing is on the local machine.
I am a little confused about which method you are using.
~/.config/common-lisp/source-registry.conf.d/
(see Configuring ASDF to find your systems)Ok. In this case I would do a user non-image kernel with just
(ql:quickload :common-lisp-jupyter)
(cl-jupyter:install)
That seems to have made the kernel appear in Jupyter-Lab, but it's unable to run any code.
(cl-jupyter:install)
Creating directories.
Installing kernel spec file C:/Users/nunez/AppData/Roaming/jupyter/kernels/common-lisp/kernel.json
Installing kernel resources to C:/Users/nunez/AppData/Roaming/jupyter/kernels/common-lisp/.
Installing local systems to C:/Users/nunez/AppData/Roaming/common-lisp-jupyter/local-projects/
Probably because there is no connection:
If I look into the installation directories, I see that C:/Users/nunez/AppData/Roaming/common-lisp-jupyter/local-projects/
is empty and C:\Users\nunez\AppData\Roaming\jupyter\kernels\common-lisp
contains a JSON file and PNG.
That empty directory looks suspicious, unless it is a data directory. I have confirmed that libzmq.dll is on the PATH
.
A local user install will only install stuff into C:\Users\nunez\AppData\Roaming\jupyter\kernels\common-lisp
Try testing with jupyter-console --kernel=common-lisp
versus using JupyterLab.
For anyone finding this issue with the same problems: jupyter notebook
and jupyter-console
are separate packages, installed with pip
.
Testing with jupyter-console --kernel=common-lisp
brings up SBCL, and then fails with:
PS S:\src\data-frame> jupyter-console --kernel=common-lisp
This is SBCL 2.0.10.7-1968e18ce, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.
SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
Traceback (most recent call last):
File "c:\users\nunez\appdata\local\programs\python\python38\lib\site-packages\jupyter_console\ptshell.py", line 433, in init_kernel_info
reply = self.client.get_shell_msg(timeout=1)
File "c:\users\nunez\appdata\local\programs\python\python38\lib\site-packages\jupyter_client\client.py", line 78, in get_shell_msg
return self.shell_channel.get_msg(*args, **kwargs)
File "c:\users\nunez\appdata\local\programs\python\python38\lib\site-packages\jupyter_client\blocking\channels.py", line 54, in get_msg
raise Empty
_queue.Empty
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "c:\users\nunez\appdata\local\programs\python\python38\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\nunez\appdata\local\programs\python\python38\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\nunez\AppData\Local\Programs\Python\Python38\Scripts\jupyter-console.exe\__main__.py", line 7, in <module>
File "c:\users\nunez\appdata\local\programs\python\python38\lib\site-packages\jupyter_core\application.py", line 254, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "c:\users\nunez\appdata\local\programs\python\python38\lib\site-packages\traitlets\config\application.py", line 844, in launch_instance
app.initialize(argv)
File "c:\users\nunez\appdata\local\programs\python\python38\lib\site-packages\traitlets\config\application.py", line 87, in inner
return method(app, *args, **kwargs)
File "c:\users\nunez\appdata\local\programs\python\python38\lib\site-packages\jupyter_console\app.py", line 137, in initialize
self.init_shell()
File "c:\users\nunez\appdata\local\programs\python\python38\lib\site-packages\jupyter_console\app.py", line 107, in init_shell
self.shell = ZMQTerminalInteractiveShell.instance(parent=self,
File "c:\users\nunez\appdata\local\programs\python\python38\lib\site-packages\traitlets\config\configurable.py", line 537, in instance
inst = cls(*args, **kwargs)
File "c:\users\nunez\appdata\local\programs\python\python38\lib\site-packages\jupyter_console\ptshell.py", line 351, in __init__
self.init_kernel_info()
File "c:\users\nunez\appdata\local\programs\python\python38\lib\site-packages\jupyter_console\ptshell.py", line 436, in init_kernel_info
raise RuntimeError("Kernel didn't respond to kernel_info_request") from e
RuntimeError: Kernel didn't respond to kernel_info_request
PS S:\src\data-frame> To load "common-lisp-jupyter":
Load 1 ASDF system:
common-lisp-jupyter
; Loading "common-lisp-jupyter"
.....
debugger invoked on a FILE-DOES-NOT-EXIST in thread
#<THREAD "main thread" RUNNING {1002300003}>:
The file #P"C:/Users/nunez/AppData/Roaming/jupyter/runtime/kernel-17488.json"
does not exist:
The system cannot find the file specified.
Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
restarts (invokable by number or by possibly-abbreviated name):
0: [CREATE ] Reopen with :if-does-not-exist :create
1: [CONTINUE ] Retry opening.
2: [USE-VALUE] Try opening a different file.
3: Ignore runtime option --eval "(jupyter:run-kernel 'common-lisp-jupyter:kernel #\"C:\\Users\\nunez\\AppData\\Roaming\\jupyter\\runtime\\kernel-17488.json\")".
4: [ABORT ] Skip rest of --eval and --load options.
5: Skip to toplevel READ/EVAL/PRINT loop.
6: [EXIT ] Exit SBCL (calling #'EXIT, killing the process).
(SB-IMPL::FILE-PERROR #P"C:/Users/nunez/AppData/Roaming/jupyter/runtime/kernel-17488.json" 2 FILE-DOES-NOT-EXIST)
0]
CL-USER(1):
This is usually an indicator that there is something wrong with the Jupyter setup. It can be kind of hard to track down. I'll start setting up a Windows VM and see if I can reproduce it.
FYI, the JupyterLab stuff, sans common lisp, works fine. Let me know if I can help.
I have not been able to reproduce this yet, mostly to having a heck of a time trying to get msys setup with Jupyter and zmq.
I have been able to get a working version using miniconda and the version of common-lisp-jupyter in quicklisp. After installing miniconda I did the following:
conda install -c conda-forge jupyterlab
conda install -c msys2 m2w64-gcc m2w64-zeromq
I then installed sbcl and quicklisp and did the following in sbcl
(ql:quickload :common-lisp-jupyter)
(clj:install)
Initially the latter failed possibly because my code page was not utf-8. I don't really use Windows much, so I just fixed the offending code which was in language-codes
in the quicklisp directory and added :external-format :utf-8
to https://github.com/Shinmera/language-codes/blob/e7aa0e37cb97a3d37d6bc7316b479d01bff8f42e/languages.lisp#L45
Is there anything I can do to help debug zmq? What test can be run to ensure connectivity?
I installed zmq with MSYS2 (note there is a difference with MSYS and MSYS2) and ensured it's on the PATH and it would be good to confirm it's functioning if we can.
Also, it would be helpful for me to understand the differences between miniconda and pip for installations. For example I know that the pip Lab/Notebook works, and wonder then if it might be configuration?
Can you tell what packages you installed in MSYS2? Like which python, etc.
Anaconda/Miniconda distribute binaries in additional to Python packages. More of a general package manager. MSYS2 would not be needed in this case. Also I think that installation of all dependencies can be done with conda install -c conda-forge jupyterlab m2w64-gcc m2w64-zeromq
. I'm testing this in GitHub actions right now.
PS S:\src> which python
/c/Users/nunez/AppData/Local/Programs/Python/Python38/python
MSYS2 is pretty standard for what you need for CFFI. Here's what in there:
PS S:\src> pacman -Qe
autoconf 2.69-5
automake-wrapper 11-1
base 2020.12-1
diffutils 3.7-1
git 2.30.0-1
m4 1.4.18-2
make 4.3-1
mingw-w64-i686-hunspell 1.7.0-5
mingw-w64-x86_64-clang 11.0.0-5
mingw-w64-x86_64-clang-tools-extra 11.0.0-5
mingw-w64-x86_64-cmake 3.19.2-1
mingw-w64-x86_64-emacs 27.1-2
mingw-w64-x86_64-gnuplot 5.4.0-2
mingw-w64-x86_64-hunspell 1.7.0-5
mingw-w64-x86_64-hunspell-en 2019.10.06-1
mingw-w64-x86_64-openlibm 0.7.3-1
mingw-w64-x86_64-polly 11.0.0-5
mingw-w64-x86_64-zeromq 4.3.3-1
rlwrap 0.43-0
sharutils 4.15.2-1
texinfo 6.7-3
texinfo-tex 6.7-3
So you did not use the Python from MSYS2?
No, that's from the Python download for Windows
And clang vs gcc, correct?
I didn't compile anything, it happened as part of the CLJ install. GCC is installed as part of MSYS2 base, so whatever was in the makefile was probably used. CC is configured as gcc version 10.2.0 (Rev6, Built by MSYS2 project)
I just installed miniconda and Juypter. Pretty much the same problem. Jupyter Lab runs, but the kernel returns a similar error. I notice a timeout=1 in the output and wonder if this may be related to waiting for SBCL? When running SBCL under emacs/slime, start-up time can be up to 120 seconds, possibly due to the source being on a NFS mounted drive and having a very large set of files to scan through (ASDF scans for all the ASD files in the configured directory.
(base) PS C:\Users\nunez> jupyter-console --kernel=common-lisp
This is SBCL 2.0.10.7-1968e18ce, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.
SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
Traceback (most recent call last):
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_console\ptshell.py", line 433, in init_kernel_info
reply = self.client.get_shell_msg(timeout=1)
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_client\client.py", line 78, in get_shell_msg
return self.shell_channel.get_msg(*args, **kwargs)
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_client\blocking\channels.py", line 54, in get_msg
raise Empty
_queue.Empty
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\nunez\miniconda3\Scripts\jupyter-console-script.py", line 9, in <module>
sys.exit(main())
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_core\application.py", line 254, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "C:\Users\nunez\miniconda3\lib\site-packages\traitlets\config\application.py", line 844, in launch_instance
app.initialize(argv)
File "C:\Users\nunez\miniconda3\lib\site-packages\traitlets\config\application.py", line 87, in inner
return method(app, *args, **kwargs)
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_console\app.py", line 137, in initialize
self.init_shell()
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_console\app.py", line 107, in init_shell
self.shell = ZMQTerminalInteractiveShell.instance(parent=self,
File "C:\Users\nunez\miniconda3\lib\site-packages\traitlets\config\configurable.py", line 537, in instance
inst = cls(*args, **kwargs)
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_console\ptshell.py", line 351, in __init__
self.init_kernel_info()
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_console\ptshell.py", line 436, in init_kernel_info
raise RuntimeError("Kernel didn't respond to kernel_info_request") from e
RuntimeError: Kernel didn't respond to kernel_info_request
(base) PS C:\Users\nunez> To load "common-lisp-jupyter":
Load 1 ASDF system:
common-lisp-jupyter
; Loading "common-lisp-jupyter"
.....
debugger invoked on a FILE-DOES-NOT-EXIST in thread
#<THREAD "main thread" RUNNING {1002310003}>:
The file #P"C:/Users/nunez/AppData/Roaming/jupyter/runtime/kernel-27424.json"
does not exist:
The system cannot find the file specified.
Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
restarts (invokable by number or by possibly-abbreviated name):
0: [CREATE ] Reopen with :if-does-not-exist :create
1: [CONTINUE ] Retry opening.
2: [USE-VALUE] Try opening a different file.
3: Ignore runtime option --eval "(jupyter:run-kernel 'common-lisp-jupyter:kernel #\"C:\\Users\\nunez\\AppData\\Roaming\\jupyter\\runtime\\kernel-27424.json\")".
4: [ABORT ] Skip rest of --eval and --load options.
5: Skip to toplevel READ/EVAL/PRINT loop.
6: [EXIT ] Exit SBCL (calling #'EXIT, killing the process).
(SB-IMPL::FILE-PERROR #P"C:/Users/nunez/AppData/Roaming/jupyter/runtime/kernel-27424.json" 2 FILE-DOES-NOT-EXIST)
0]
CL-USER(1):
The default timeout for the kernel to respond is 60 seconds. You can increase it with the --ZMQTerminalInteractiveShell.kernel_timeout=180
option on the jupyter console command line.
I think you can configure for the Lab/Notebook in the .jupyter/jupyter_notebook_config.py
file as c.MappingKernelManager.kernel_info_timeout
Console works with that parameter. However, it is specific to juypter-console. At least there's something working at the moment. Labs probably has something similar.
Okay, so for anyone that finds this issue in the future:
If you're getting kernel timeouts:
jupyter-lab
c.MappingKernelManager.kernel_info_timeout=180
If you only installed Jupyter Lab, the notebook file won't exist, that's why you have to create it.
Glad you got it working. I'll leave the issue open so others can see the solution and update the documentation along with the improved installation details soon.
I'll report anything else here then. First thing to note: a fresh conda install with juypter labs is not a light-weight. 2GB taken up with jupyter-lab alone.
DId you do Anaconda or Miniconda?
Miniconda
This problem has resurfaced after upgrading in early May 2021. I get the same error as reported above on 26 Feb (even the line numbers are the same), however increasing the notebook timeout has no effect this time. Increasing the ZMQ timeout results in an error about an invalid argument to ZMQ. The c.MappingKernelManager.kernel_info_timeout=600
is set in the ~/.jupyter/jupyter_notebook_config.py
PS S:\src\common-lisp-jupyter> jupyter-console --kernel=common-lisp --ZMQTerminalInteractiveShell.kernel_timeout=600
This is SBCL 2.0.10.7-1968e18ce, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.
SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
To load "common-lisp-jupyter":
Load 1 ASDF system:
common-lisp-jupyter
; Loading "common-lisp-jupyter"
......
Assertion failed: Invalid argument [10022] (../zeromq-4.3.3/src/zmq.cpp:1061)
Assertion failed: Invalid argument [10022] (../zeromq-4.3.3/src/zmq.cpp:1061)
debugger invoked on a SB-WIN32:EXCEPTION in thread
#<THREAD "main thread" RUNNING {1002320003}>:
An exception occurred in context #.(SB-SYS:INT-SAP #X007FE330): #.(SB-SYS:INT-SAP #X007FEEE0). (Exception code: 1073741845)
Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
restarts (invokable by number or by possibly-abbreviated name):
0: [CONTINUE] Return from the exception handler
1: Ignore runtime option --eval "(jupyter:run-kernel 'common-lisp-jupyter:kernel)".
2: [ABORT ] Skip rest of --eval and --load options.
3: Skip to toplevel READ/EVAL/PRINT loop.
4: [EXIT ] Exit SBCL (calling #'EXIT, killing the process).
(SB-KERNEL:HANDLE-WIN32-EXCEPTION #.(SB-SYS:INT-SAP #X007FE330) #.(SB-SYS:INT-SAP #X007FEEE0))
0]
Jupyter Lab no longer uses jupyter/jupyter_notebook_config.py
. If you run jupyter-lab --generate-config
it will generate a default config file and say something like this:
Writing default config to: /home/yitzchak/.jupyter/jupyter_lab_config.py
I've got that generated, however as far as I can tell the timeout is still controlled by jupyter/jupyter_notebook_config.py
, though it only has that single parameter. Also, I've not updated Jupyter Lab, so unless it auto-updates without telling me, the only change is CLJ and shasht.
I believe that Jupyter Lab is now using a different kernel manager so the settings may not be shared. Their kernel manager is AsyncMappingKernelManager
so the setting is probably c.AsyncMappingKernelManager.kernel_info_timeout=600
in the lab config file. You could put in both files just in case. The setting is on page 120 here
I added c.AsyncMappingKernelManager.kernel_info_timeout=600
to the notebook config file. Same error from ZMQ around 'Invalid argument'
Assertion failed: Invalid argument [10022] (../zeromq-4.3.3/src/zmq.cpp:1061)
Assertion failed: Invalid argument [10022] (../zeromq-4.3.3/src/zmq.cpp:1061)
debugger invoked on a SB-WIN32:EXCEPTION in thread
#<THREAD "main thread" RUNNING {1002310003}>:
An exception occurred in context #.(SB-SYS:INT-SAP #X007FE330): #.(SB-SYS:INT-SAP #X007FEEE0). (Exception code: 1073741845)
This looks like a CFFI error. There's more information on the screen when I Ctrl-C out of the failure:
Traceback (most recent call last):
File "C:\Users\nunez\miniconda3\Scripts\jupyter-console-script.py", line 9, in <module>
sys.exit(main())
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_core\application.py", line 254, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "C:\Users\nunez\miniconda3\lib\site-packages\traitlets\config\application.py", line 844, in launch_instance
app.initialize(argv)
File "C:\Users\nunez\miniconda3\lib\site-packages\traitlets\config\application.py", line 87, in inner
return method(app, *args, **kwargs)
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_console\app.py", line 137, in initialize
self.init_shell()
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_console\app.py", line 107, in init_shell
self.shell = ZMQTerminalInteractiveShell.instance(parent=self,
File "C:\Users\nunez\miniconda3\lib\site-packages\traitlets\config\configurable.py", line 537, in instance
inst = cls(*args, **kwargs)
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_console\ptshell.py", line 351, in __init__
self.init_kernel_info()
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_console\ptshell.py", line 433, in init_kernel_info
reply = self.client.get_shell_msg(timeout=1)
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_client\client.py", line 78, in get_shell_msg
return self.shell_channel.get_msg(*args, **kwargs)
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_client\blocking\channels.py", line 47, in get_msg
ready = self.socket.poll(timeout)
File "C:\Users\nunez\miniconda3\lib\site-packages\zmq\sugar\socket.py", line 810, in poll
evts = dict(p.poll(timeout))
File "C:\Users\nunez\miniconda3\lib\site-packages\zmq\sugar\poll.py", line 105, in poll
return zmq_poll(self.sockets, timeout=timeout)
File "zmq/backend/cython/_poll.pyx", line 137, in zmq.backend.cython._poll.zmq_poll
File "zmq/backend/cython/_poll.pyx", line 117, in zmq.backend.cython._poll.zmq_poll
File "zmq/backend/cython/checkrc.pxd", line 13, in zmq.backend.cython.checkrc._check_rc
PyErr_CheckSignals()
File "C:\Users\nunez\miniconda3\lib\site-packages\jupyter_console\app.py", line 119, in handle_sigint
if self.shell._executing:
AttributeError: 'ZMQTerminalIPythonApp' object has no attribute 'shell'
PS S:\src\IPS\notebooks>
Ok, I'll set up a Windows VM and see if I can reproduce. I may come back with questions about your setup.
Were you able to reproduce the problem?
Unfortunately, no. I used the installation method here and it works just fine. Did you do an installation different that this?
Cancel that last statement. Looks I just reproduced it. Had an old quicklisp version around.
I think that I have a solution. Looks like the issue is in pzmq with a struct that is a different size on win64. Unfortunately I don't think I can grovel the size since zeromq changed some of the names of types along the way. If you want to try out a fix until I manage to figure out a general solution you can change following line to (fd :uint64)
https://github.com/orivej/pzmq/blob/91dd3a891e639f41d44bfb5f449daf0fcd408e29/c-api.lisp#L588
I can give it a whirl if it's helpful, though of course a general solution is preferred so that others can reproduce the analysis and work on Windows. Is there anything I can do to help with that?
I submitted a PR to pzmq for this. Until it gets reviewed/merged you can clone the following branch if you want to try it out.
The maintainer of pzmq was able to find a better way to fix this and has merged that change.
It looks like it's going to be a while before this makes it into the next release. ZMQ seems to do a new release about every six months, and then a month later it gets picked up by MSYS2.
The changes to ZMQ are just documentation changes so you shouldn't need a new release there. The changes to make everything work again are just in https://github.com/orivej/pzmq/ so you should just be able to clone that or wait till the next Quicklisp updates.
Quicklisp just updated. Everything should work now. Can you try again?
It's running again now, thanks.
Awesome. Closing.
We might want to leave it open as a historical record.
To reproduce using a git clone (cloned on 4 Feb) repo, SBCL, on the latest version of MS Windows and the latest Jupyter-Lab:
In the debugger: