search

jerry40 / guile-kernel

GNU Guile Scheme kernel for Jupyter Notebook
GNU General Public License v3.0
35 stars 16 forks source link

Kernel Dies Unexpectedly #1

Closed TENSAE21 closed 6 years ago

TENSAE21 commented 6 years ago

[I 18:22:44.513 NotebookApp] Kernel started: dc6d9eab-2a05-4dc3-a2af-64ee41f66891 195a772e3db49c8f5b3c0a7584156c3837793f4d18ce1e4e3e69dd629100ae1303caefe50c04d98acb275727b8f4cdcb38e05ee73912870016be8b857327e214a529e811408412f3819bd05145af3fed1cf0b183a5482ec42ade085235be4b20[W 18:22:54.527 NotebookApp] Timeout waiting for kernel_info reply from dc6d9eab-2a05-4dc3-a2af-64ee41f66891 0dce973ab6940e8a031b84fd5c15a4e987089f38386ba4b0a37c171467c9eae8289d6861412ea71d595f46921211c338cf48f46b332744779c805f1b490a19e66fbe726789a2045363aed4aaadc26d071fce0479287a56f4132fd4a0fcc47be2574dacf41e011dffee8d0b3146776c377dae8af020f043328e4e874737a343f4 ----------------------->comm_info_request Backtrace: In ice-9/boot-9.scm: 157: 8 [catch #t #<catch-closure 1191ba0> ...] In unknown file: ?: 7 [apply-smob/1 #<catch-closure 1191ba0>] In ice-9/boot-9.scm: 63: 6 [call-with-prompt prompt0 ...] In ice-9/eval.scm: 432: 5 [eval # #] In ice-9/boot-9.scm: 2401: 4 [save-module-excursion #<procedure 11ae9c0 at ice-9/boot-9.scm:4045:3 ()>] 4050: 3 [#<procedure 11ae9c0 at ice-9/boot-9.scm:4045:3 ()>] In unknown file: ?: 2 [load-compiled/vm "/home/admin/.cache/guile/ccache/2.0-LE-8-2.0/home/admin/.local/share/jupyter/kernels/guile/main.scm.go"] In /home/admin/.local/share/jupyter/kernels/guile/main.scm: 102: 1 [general-handler #<pointer 0xf0ccc0>] In unknown file: ?: 0 [#f #<pointer 0xf0ccc0> "232a2f702a5f4def958afa5f0ba4a4a5" ...]

ERROR: In procedure #f: ERROR: Wrong type to apply: #f

jerry40 commented 6 years ago

Hi! Can you explain at what moment it happened? Was you evaluating any guile code? Or the kernel did not start?

TENSAE21 commented 6 years ago

Hey there! So let's say I create a new Guile notebook, the kernel starts and runs fine for a few seconds but then dies, it opts to restart and that runs for a few minutes but then it dies again. In the times that it's running, it evaluates scheme code perfectly and the above error is shown on the terminal when it dies.

jerry40 commented 6 years ago

Interesting, thanks for the details! Might it fail because of scheme code you've evaluated? Can you remember your scheme code and if so post it here? Or the kernel fails even if you evaluate simple code like (+ 1 2)? Also (sorry for so many questions), does it fail after code evaluates or it can fail without any code being evaluated?

TENSAE21 commented 6 years ago

This happens whenever a notebook begins, it could be a new one with no code added yet or one with just marked down text or in restarting a guile notebook that run and shutdown properly. This is an example of the later. The notebook had the following in it... (use-modules (ice-9 readline)) (activate-readline) (add-to-load-path "/usr/local/share/opencog/scm") (add-to-load-path ".") (use-modules (opencog)) (use-modules (opencog query)) (use-modules (opencog exec)) This run fine with an output of # I shut down the notebook and restarted it later to find the error... (thanks for the responses)

jerry40 commented 6 years ago

Thanks! By any chance, could anybody use the same kernel at the same time you were using it? I'm thinking about possible parallel sessions collisions... I think I did not test this part deeply enough, so problems are possible.

tesYolan commented 6 years ago

I am running into similar issues when spawning a new guile kernel. Here is the log when I spawn with jupyter notebook --debug with zeromq and pyzmq version of 4.1.4 and 15.2.0 respectively. 

[I 16:30:35.755 NotebookApp] Creating new notebook in 
[D 16:30:35.758 NotebookApp] Saving /opencog/notebooks/Untitled4.ipynb
[I 16:30:35.918 NotebookApp] Writing notebook-signing key to /root/.local/share/jupyter/notebook_secret
[D 16:30:36.014 NotebookApp] 201 POST /api/contents (172.17.0.1) 260.02ms
[D 16:30:36.022 NotebookApp] Using contents: services/contents
[D 16:30:36.119 NotebookApp] Path base/images/favicon-notebook.ico served from /usr/local/lib/python3.5/dist-packages/notebook/static/base/images/favicon-notebook.ico
[D 16:30:36.120 NotebookApp] Path components/bootstrap-tour/build/css/bootstrap-tour.min.css served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/bootstrap-tour/build/css/bootstrap-tour.min.css
[D 16:30:36.120 NotebookApp] Path components/codemirror/lib/codemirror.css served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/codemirror/lib/codemirror.css
[D 16:30:36.120 NotebookApp] Path notebook/css/override.css served from /usr/local/lib/python3.5/dist-packages/notebook/static/notebook/css/override.css
[D 16:30:36.122 NotebookApp] Path components/text-encoding/lib/encoding.js served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/text-encoding/lib/encoding.js
[D 16:30:36.122 NotebookApp] Path notebook/js/main.min.js served from /usr/local/lib/python3.5/dist-packages/notebook/static/notebook/js/main.min.js
[D 16:30:36.127 NotebookApp] 200 GET /notebooks/Untitled4.ipynb?kernel_name=guile-kernel (172.17.0.1) 106.24ms
[D 16:30:36.146 NotebookApp] Path components/MathJax/MathJax.js served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/MathJax/MathJax.js
[D 16:30:36.147 NotebookApp] 304 GET /static/components/MathJax/MathJax.js?config=TeX-AMS-MML_HTMLorMML-full,Safe&delayStartupUntil=configured (172.17.0.1) 1.77ms
[D 16:30:36.149 NotebookApp] 304 GET /custom/custom.css (172.17.0.1) 1.12ms
[D 16:30:36.288 NotebookApp] Path components/MathJax/config/TeX-AMS-MML_HTMLorMML-full.js served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/MathJax/config/TeX-AMS-MML_HTMLorMML-full.js
[D 16:30:36.288 NotebookApp] 304 GET /static/components/MathJax/config/TeX-AMS-MML_HTMLorMML-full.js?rev=2.6.0 (172.17.0.1) 0.96ms
[D 16:30:36.337 NotebookApp] Path components/MathJax/config/Safe.js served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/MathJax/config/Safe.js
[D 16:30:36.338 NotebookApp] 304 GET /static/components/MathJax/config/Safe.js?rev=2.6.0 (172.17.0.1) 1.15ms
[D 16:30:36.526 NotebookApp] Paths used for configuration of notebook: 
        /etc/jupyter/nbconfig/notebook.json
[D 16:30:36.526 NotebookApp] Paths used for configuration of notebook: 
        /usr/local/etc/jupyter/nbconfig/notebook.d/widgetsnbextension.json
        /usr/local/etc/jupyter/nbconfig/notebook.json
[D 16:30:36.527 NotebookApp] Paths used for configuration of notebook: 
        /usr/etc/jupyter/nbconfig/notebook.json
[D 16:30:36.527 NotebookApp] Paths used for configuration of notebook: 
        /root/.jupyter/nbconfig/notebook.json
[D 16:30:36.528 NotebookApp] 200 GET /api/config/notebook?_=1527093036412 (172.17.0.1) 3.12ms
[D 16:30:36.529 NotebookApp] Paths used for configuration of common: 
        /etc/jupyter/nbconfig/common.json
[D 16:30:36.529 NotebookApp] Paths used for configuration of common: 
        /usr/local/etc/jupyter/nbconfig/common.json
[D 16:30:36.530 NotebookApp] Paths used for configuration of common: 
        /usr/etc/jupyter/nbconfig/common.json
[D 16:30:36.530 NotebookApp] Paths used for configuration of common: 
        /root/.jupyter/nbconfig/common.json
[D 16:30:36.531 NotebookApp] 200 GET /api/config/common?_=1527093036413 (172.17.0.1) 2.28ms
[D 16:30:36.574 NotebookApp] Path components/MathJax/extensions/MathML/content-mathml.js served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/MathJax/extensions/MathML/content-mathml.js
[D 16:30:36.575 NotebookApp] 304 GET /static/components/MathJax/extensions/MathML/content-mathml.js?rev=2.6.0 (172.17.0.1) 1.00ms
[D 16:30:36.598 NotebookApp] Found kernel python3 in /usr/local/share/jupyter/kernels
[D 16:30:36.598 NotebookApp] Found kernel guile-kernel in /usr/local/share/jupyter/kernels
[D 16:30:36.600 NotebookApp] 304 GET /api/kernelspecs (172.17.0.1) 2.70ms
[D 16:30:36.700 NotebookApp] 200 GET /api/contents/Untitled4.ipynb?type=notebook&_=1527093036414 (172.17.0.1) 70.24ms
[D 16:30:36.701 NotebookApp] Path components/MathJax/jax/output/HTML-CSS/fonts/STIX-Web/fontdata.js served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/MathJax/jax/output/HTML-CSS/fonts/STIX-Web/fontdata.js
[D 16:30:36.702 NotebookApp] 304 GET /static/components/MathJax/jax/output/HTML-CSS/fonts/STIX-Web/fontdata.js?rev=2.6.0 (172.17.0.1) 0.90ms
[D 16:30:36.702 NotebookApp] Path jupyter-js-widgets/extension.js served from /usr/local/share/jupyter/nbextensions/jupyter-js-widgets/extension.js
[D 16:30:36.706 NotebookApp] 200 GET /nbextensions/jupyter-js-widgets/extension.js?v=20180523163024 (172.17.0.1) 4.62ms
[D 16:30:36.707 NotebookApp] Path components/MathJax/extensions/Safe.js served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/MathJax/extensions/Safe.js
[D 16:30:36.708 NotebookApp] 304 GET /static/components/MathJax/extensions/Safe.js?rev=2.6.0 (172.17.0.1) 0.83ms
[D 16:30:36.708 NotebookApp] 304 GET /custom/custom.js?v=20180523163024 (172.17.0.1) 0.63ms
[D 16:30:36.767 NotebookApp] Starting kernel: ['guile', '-s', '/usr/local/share/jupyter/kernels/guile-kernel/src/guile-jupyter-kernel.scm', '--', '/root/.local/share/jupyter/runtime/kernel-db5d5da2-cf23-406e-8982-ee35ae5a6b4c.json']
[D 16:30:36.770 NotebookApp] Connecting to: tcp://127.0.0.1:32955
[D 16:30:36.770 NotebookApp] Connecting to: tcp://127.0.0.1:36790
[I 16:30:36.772 NotebookApp] Kernel started: db5d5da2-cf23-406e-8982-ee35ae5a6b4c
[D 16:30:36.772 NotebookApp] Kernel args: {'kernel_name': 'guile-kernel', 'cwd': '/opencog/notebooks'}
[D 16:30:36.773 NotebookApp] 201 POST /api/sessions (172.17.0.1) 18.93ms
[D 16:30:36.775 NotebookApp] 200 GET /api/contents/Untitled4.ipynb/checkpoints?_=1527093036415 (172.17.0.1) 1.21ms
[D 16:30:36.793 NotebookApp] Initializing websocket connection /api/kernels/db5d5da2-cf23-406e-8982-ee35ae5a6b4c/channels
[D 16:30:36.795 NotebookApp] Requesting kernel info from db5d5da2-cf23-406e-8982-ee35ae5a6b4c
[D 16:30:36.795 NotebookApp] Connecting to: tcp://127.0.0.1:51742
b614e7f41ff2ee17e3284578076824197794cdf9f41184d00f221b444d14e19b[D 16:30:37.473 NotebookApp] activity on db5d5da2-cf23-406e-8982-ee35ae5a6b4c: status
544f5ec38530f2025bbc155b6c28e85eab8edb827ec1c9d11679fa8f86d658bf600d0c1bc56ffb4b62bdb1afe6bd2d0a4c11f967f0bfa4b64c6e3d62c8f7d78d[D 16:30:38.562 NotebookApp] activity on db5d5da2-cf23-406e-8982-ee35ae5a6b4c: status
[W 16:30:46.796 NotebookApp] Timeout waiting for kernel_info reply from db5d5da2-cf23-406e-8982-ee35ae5a6b4c
[D 16:30:46.797 NotebookApp] 101 GET /api/kernels/db5d5da2-cf23-406e-8982-ee35ae5a6b4c/channels?session_id=7a8aaa9b53bd425d8d7102a4f47a26eb (172.17.0.1) 10004.77ms
[D 16:30:46.798 NotebookApp] Opening websocket /api/kernels/db5d5da2-cf23-406e-8982-ee35ae5a6b4c/channels
[D 16:30:46.798 NotebookApp] Getting buffer for db5d5da2-cf23-406e-8982-ee35ae5a6b4c
[D 16:30:46.799 NotebookApp] Connecting to: tcp://127.0.0.1:51742
[D 16:30:46.799 NotebookApp] Connecting to: tcp://127.0.0.1:36790
[D 16:30:46.799 NotebookApp] Connecting to: tcp://127.0.0.1:47558
8c9c3a88d60fe1c0005967a32fa7912dc06ce77d4f1abba1fa6d6d820a225fae[D 16:30:47.484 NotebookApp] activity on db5d5da2-cf23-406e-8982-ee35ae5a6b4c: status
3fc8ea30cdbc5b7b922155d339a8c8eab6bae36ab0f37d60a08e2ad34e59cdcc[D 16:30:48.162 NotebookApp] Path components/codemirror/mode/scheme/scheme.js served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/codemirror/mode/scheme/scheme.js
[D 16:30:48.165 NotebookApp] 200 GET /static/components/codemirror/mode/scheme/scheme.js?v=20180523163024 (172.17.0.1) 3.98ms
400ce3fad2624b5340d8032f6f1289c001a1333805a1c27c8991cd48f9d2a301[D 16:30:48.684 NotebookApp] activity on db5d5da2-cf23-406e-8982-ee35ae5a6b4c: status
d16272fb7c28ff2cab7333ea1dc84b0d81fc92ff91c953022c95877f5665cb8e
----------------------->comm_info_request
[D 16:30:49.232 NotebookApp] activity on db5d5da2-cf23-406e-8982-ee35ae5a6b4c: status
Backtrace:
           3 (primitive-load "/usr/local/share/jupyter/kernels/guile…")
In ice-9/eval.scm:
    619:8  2 (_ #(#(#<directory (guile-user) fb8140>) #<pointer 0x…>))
    619:8  1 (_ #(#(#(#(#(#(#(#(#(#(#(…) …) …) …) …) …) …) …) …) …) …))
In unknown file:
           0 (_ #<pointer 0xe299f0> "7a8aaa9b53bd425d8d7102a4f47a26…" …)

ERROR: Wrong type to apply: #f
[I 16:30:51.771 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
WARNING:root:kernel db5d5da2-cf23-406e-8982-ee35ae5a6b4c restarted
[D 16:30:51.773 NotebookApp] Starting kernel: ['guile', '-s', '/usr/local/share/jupyter/kernels/guile-kernel/src/guile-jupyter-kernel.scm', '--', '/root/.local/share/jupyter/runtime/kernel-db5d5da2-cf23-406e-8982-ee35ae5a6b4c.json']
[D 16:30:51.777 NotebookApp] Connecting to: tcp://127.0.0.1:32955
[D 16:30:54.780 NotebookApp] KernelRestarter: restart apparently succeeded
[D 16:32:37.445 NotebookApp] 200 GET /api/contents/Untitled4.ipynb?content=0&_=1527093036416 (172.17.0.1) 2.56ms
[I 16:32:37.454 NotebookApp] Saving file at /Untitled4.ipynb
[D 16:32:37.455 NotebookApp] Saving /opencog/notebooks/Untitled4.ipynb
[D 16:32:37.574 NotebookApp] 200 PUT /api/contents/Untitled4.ipynb (172.17.0.1) 120.70ms

This comes about when I spawn the a new notebook with the guile kernel. There is no other thing is using it AFAIK. If I reconnect it seems to work just fine. But I do wonder what and why is the following message in the above appears:

[W 16:30:46.796 NotebookApp] Timeout waiting for kernel_info reply from db5d5da2-cf23-406e-8982-ee35ae5a6b4c

And also the following line seems pertinent:

----------------------->comm_info_request
[D 16:30:49.232 NotebookApp] activity on db5d5da2-cf23-406e-8982-ee35ae5a6b4c: status
Backtrace:
           3 (primitive-load "/usr/local/share/jupyter/kernels/guile…")
In ice-9/eval.scm:
    619:8  2 (_ #(#(#<directory (guile-user) fb8140>) #<pointer 0x…>))
    619:8  1 (_ #(#(#(#(#(#(#(#(#(#(#(…) …) …) …) …) …) …) …) …) …) …))
In unknown file:
           0 (_ #<pointer 0xe299f0> "7a8aaa9b53bd425d8d7102a4f47a26…" …)

ERROR: Wrong type to apply: #f

At first i thought the guile auto-compile was the reason for the lag, as such, I have setted

export GUILE_AUTO_COMPILE=0

But still the error persists when spawning a new kernel.

@jerry40 What do you think additional reasons could be for the kernel dying as such? I am thinking a timeout should be a bit longer from the following line:

[D 16:30:46.797 NotebookApp] 101 GET /api/kernels/db5d5da2-cf23-406e-8982-ee35ae5a6b4c/channels?session_id=7a8aaa9b53bd425d8d7102a4f47a26eb (172.17.0.1) 10004.77ms

or zeromq version should be bumped up to 4.2+?

jerry40 commented 6 years ago

Hi, tesYolan! Many thanks for the additional details!

I see that in both cases we have something with comm_info_request and I see that you use some widgets or extensions (*.js). guile-kernel ignores this type of a message so far (since I do not have any extension installed, my bad!). I need to start accepting this sort of messages and I'm sure that the timeout problem will disappear. Can you (if possible) say what extensions are you using? I can test them on my side.

tesYolan commented 6 years ago

Thank you for the prompt reply. Regarding the list of different versions of items I am utilizing, here is what i believe are all of env variables I am utilizing. jupyter version 4.4.0 that I installed from pip3 from jupyter serverexension list, I get no outputs, meaning I haven't utilized any server extensions.

The environmental variables I installed for pip3 obtained by pip freeze are as follows:

backcall==0.1.0
bleach==2.1.3
decorator==4.3.0
entrypoints==0.2.3
html5lib==1.0.1
ipykernel==4.8.2
ipython==6.3.1
ipython-genutils==0.2.0
ipywidgets==7.2.1
jedi==0.12.0
Jinja2==2.10
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.2.3
jupyter-console==5.2.0
jupyter-core==4.4.0
MarkupSafe==1.0
mistune==0.8.3
nbconvert==5.3.1
nbformat==4.4.0
notebook==5.4.1
pandocfilters==1.4.2
parso==0.2.0
pexpect==4.5.0
pickleshare==0.7.4
prompt-toolkit==1.0.15
ptyprocess==0.5.2
pycurl==7.43.0
Pygments==2.2.0
pygobject==3.20.0
python-apt==1.1.0b1+ubuntu0.16.4.1
python-dateutil==2.7.2
pyzmq==15.2.0
qtconsole==4.3.1
Send2Trash==1.5.0
simplegeneric==0.8.1
simplejson==3.8.1
six==1.11.0
terminado==0.8.1
testpath==0.3.1
tornado==4.5.3
traitlets==4.3.2
unattended-upgrades==0.1
wcwidth==0.1.7
webencodings==0.5.1
widgetsnbextension==3.2.1

Besides this I hadn't had utilized or installed any applications. Let me know if these information suffice. If not, let me know.

jerry40 commented 6 years ago

I think I know how to fix this issue. Hopefully, I'll commit new version soon. Thanks!

jerry40 commented 6 years ago

I've committed new version of the the jupyter kernel (guile-jupyter-kernel.scm). I think it will fix the problem with kernel death, but it will not fix the problem with timeout (which seems to be occurred only for 1st kernel info request, so we can ignore this warning)

Can you test this version on your side?

tesYolan commented 6 years ago

Yes. I think it resolves the issue on my end. And Yes the timeout issue persists, but isn't followed by death of the kernel. Great!

jerry40 commented 6 years ago

I'm happy to hear that! Let's wait for TENSAE21's answer and I'll close the issue then. Thank you for your help!

TENSAE21 commented 6 years ago

yes, it's running fine for me too. Thank you!... May I ask what the problem was?

jerry40 commented 6 years ago

Thanks! It was my mistake, I use function 'dispatch' to select which particular function should be executed for particular jupyter's message type. And unfortunately this function had no 'default' choice, so it was returning #f. So guile had tried to execute (#f ...) function and everything failed.