Closed davidpham87 closed 4 years ago
Note: I tested it using the spacemacs environment and I used it on two different computers and get the same undesired behavior.
So, I set up a jupyter notebook server on an AWS instance using a cert file and password. Believe me, no one was more surprised than I when everything worked the way it should. I was able to log in from my local windows machine and do all the normal notebook things.
This makes me think there is a configuration issue.
Are you able to log in and view the notebooklist buffer? What are the contents of the ein:log-all and, if you are able, the messages emitted by jupyter when you try to open a notebook and execute a cell?
Dear @millejoh,
Thanks a lot for your swift answer. I can log in and read any notebook. My issue is when I try to execute any cell.
Here is the content of ein:log-all
:
[debug] NOTEBOOKLIST-OPEN: https://localhost:8000/ @#<buffer *eshell*>
[info] Opened notebook list at https://localhost:8000 with path . @#<buffer *ein:notebooklist https://localhost:8000*>
[info] Opening notebook Untitled.ipynb... @#<buffer *ein:notebooklist https://localhost:8000*>
[debug] [GC-PREPARE-OPERATION] Setting cons threshold to 1000000000000. @#<buffer *ein:notebooklist https://localhost:8000*>
[debug] Opening notebook at Untitled.ipynb @#<buffer *ein:notebooklist https://localhost:8000*>
[info] Reading nbformat4 notebook. @#<buffer *ein:notebooklist https://localhost:8000*>
[info] Worksheet Untitled.ipynb is ready @#<buffer *ein: https://localhost:8000/Untitled.ipynb*>
[info] Enabling autosaves for Untitled.ipynb with frequency 60 seconds. @#<buffer *ein:notebooklist https://localhost:8000*>
[debug] [GC-COMPLETE-OPERATION] Reverting cons threshold to 100000000. @#<buffer *ein:notebooklist https://localhost:8000*>
[info] Notebook Untitled.ipynb is ready @#<buffer *ein:notebooklist https://localhost:8000*>
[info] Kernel started: 83d394e8-afcc-47f1-8d71-894ad586b237 @#<buffer *ein: https://localhost:8000/Untitled.ipynb*>
[info] Starting channels WS: "wss://localhost:8000/api/kernels/83d394e8-afcc-47f1-8d71-894ad586b237/channels?session_id=baf2cfac-a075-43c5-8373-9e90248d8487" @#<buffer *ein: https://localhost:8000/Untitled.ipynb*>
[debug] EIN:WEBSOCKET--PREPARE-COOKIES Storing cookies in prep for opening websocket ((... ...)) @#<buffer *ein: https://localhost:8000/Untitled.ipynb*>
[debug] NOTEBOOKLIST-OPEN: https://localhost:8000/ @#<buffer *ein: https://localhost:8000/Untitled.ipynb*>
[info] Opened notebook list at https://localhost:8000 with path . @#<buffer *ein:notebooklist https://localhost:8000*>
[debug] (via run-at-time) onclose-arg changed to: (:ws-url "wss://localhost:8000/api/kernels/83d394e8-afcc-47f1-8d71-894ad586b237" :already-called-onclose nil :early nil) @#<buffer *ein: https://localhost:8000/Untitled.ipynb*>
[warn] Kernel is not ready yet! (or closed already.) @#<buffer *ein: https://localhost:8000/Untitled.ipynb*>
The content of the terminal from jupyter
console
[I 18:37:08.003 NotebookApp] Adapting to protocol v5.1 for kernel 83d394e8-afcc-47f1-8d71-894ad586b237
[W 18:39:25.093 NotebookApp] Clearing invalid/expired login cookie username-localhost-8000
[W 18:39:25.093 NotebookApp] Clearing invalid/expired login cookie username-localhost-8000
[I 18:39:25.094 NotebookApp] 302 POST /login (127.0.0.1) 1.54ms
[W 18:39:25.243 NotebookApp] 405 POST / (127.0.0.1) 0.54ms referer=None
Maybe related to a similar problem, though I am not seeing any websocket errors in your output. Question for @davidpham87, the Jupyter server is running on AWS but the logs show localhost
. Are you also running Emacs/EIN on AWS rather than from a local emacs session?
@julienchastang I am not sure if I understand correctly the question, but I forwarded the port on the server jupyter notebok --port=8888
to localhost:8000
, which could be the reason why it is written localhost.
I tried to debug the elisp code and the function returning if a kernel is open or not (the function returns nil
).
The port forwarding is probably causing the problem - EIN is writing an auth cookie tagged for the localhost:8000 URL, but jupyter may be expecting a cookie tagged with the url:port of the AWS instance. If you were able to see the log of the jupyter server running in debug mode (start via jupyter notebook --debug
) it might give some more details in this respect.
It would be good to know how the web browser client reconciles all this, and maybe I could recreate it in EIN. To be honest, thinking about this gives me a headache...
So I tested without the forwarding to localhost, so connecting directly to the ip with amazon.com:8888
at the end. However, I still had the issue. Please find below the output of jupyter notebook --debug
.
[Edit]: I don't know if this is relevant, but I decided to open emacs25 in the terminal directly from the remote server and their, whenever I try to connect to the notebook, there is a *Network Security Manager*
asking me to continue connecting. If I accept, the notebook is working fine. However, this message is not sent when I use on my local computer.
Can you tell me which steps you did for connecting and running your notebook on aws? Thanks a lot for your support!
[D 12:23:06.405 NotebookApp] Searching ['/home/ubuntu/machine-learning/projects/capstone', '/home/ubuntu/.jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 12:23:06.406 NotebookApp] Looking for jupyter_config in /etc/jupyter
[D 12:23:06.406 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 12:23:06.406 NotebookApp] Looking for jupyter_config in /usr/etc/jupyter
[D 12:23:06.406 NotebookApp] Looking for jupyter_config in /home/ubuntu/.jupyter
[D 12:23:06.406 NotebookApp] Looking for jupyter_config in /home/ubuntu/machine-learning/projects/capstone
[D 12:23:06.407 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter
[D 12:23:06.407 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 12:23:06.407 NotebookApp] Looking for jupyter_notebook_config in /usr/etc/jupyter
[D 12:23:06.407 NotebookApp] Looking for jupyter_notebook_config in /home/ubuntu/.jupyter
[D 12:23:06.408 NotebookApp] Loaded config file: /home/ubuntu/.jupyter/jupyter_notebook_config.py
[D 12:23:06.408 NotebookApp] Looking for jupyter_notebook_config in /home/ubuntu/machine-learning/projects/capstone
[D 12:23:06.608 NotebookApp] Paths used for configuration of jupyter_notebook_config:
/etc/jupyter/jupyter_notebook_config.json
[D 12:23:06.608 NotebookApp] Paths used for configuration of jupyter_notebook_config:
/usr/local/etc/jupyter/jupyter_notebook_config.json
[D 12:23:06.609 NotebookApp] Paths used for configuration of jupyter_notebook_config:
/usr/etc/jupyter/jupyter_notebook_config.json
[D 12:23:06.609 NotebookApp] Paths used for configuration of jupyter_notebook_config:
/home/ubuntu/.jupyter/jupyter_notebook_config.json
[I 12:23:06.610 NotebookApp] Serving notebooks from local directory: /home/ubuntu/machine-learning/projects/capstone
[I 12:23:06.610 NotebookApp] The Jupyter Notebook is running at:
[I 12:23:06.610 NotebookApp] https://(ip-172-31-23-98 or 127.0.0.1):8888/
[I 12:23:06.610 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[D 12:23:29.669 NotebookApp] Using contents: services/contents
[D 12:23:29.697 NotebookApp] Path base/images/favicon.ico served from /usr/local/lib/python3.5/dist-packages/notebook/static/base/images/favicon.ico
[D 12:23:29.697 NotebookApp] Path components/jquery-ui/themes/smoothness/jquery-ui.min.css served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css
[D 12:23:29.697 NotebookApp] Path components/jquery-typeahead/dist/jquery.typeahead.min.css served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/jquery-typeahead/dist/jquery.typeahead.min.css
[D 12:23:29.697 NotebookApp] Path style/style.min.css served from /usr/local/lib/python3.5/dist-packages/notebook/static/style/style.min.css
[D 12:23:29.698 NotebookApp] Path auth/css/override.css served from /usr/local/lib/python3.5/dist-packages/notebook/static/auth/css/override.css
[D 12:23:29.698 NotebookApp] Path components/es6-promise/promise.min.js served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/es6-promise/promise.min.js
[D 12:23:29.699 NotebookApp] Path components/preact/index.js served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/preact/index.js
[D 12:23:29.699 NotebookApp] Path components/proptypes/index.js served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/proptypes/index.js
[D 12:23:29.699 NotebookApp] Path components/preact-compat/index.js served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/preact-compat/index.js
[D 12:23:29.699 NotebookApp] Path components/requirejs/require.js served from /usr/local/lib/python3.5/dist-packages/notebook/static/components/requirejs/require.js
[D 12:23:29.699 NotebookApp] Path base/images/logo.png served from /usr/local/lib/python3.5/dist-packages/notebook/static/base/images/logo.png
[W 12:23:29.700 NotebookApp] 401 POST /login (77.57.125.18) 31.35ms referer=None
[I 12:23:40.126 NotebookApp] 302 POST /login (77.57.125.18) 1.17ms
[W 12:23:40.273 NotebookApp] 405 POST / (77.57.125.18) 0.53ms referer=None
[D 12:23:49.192 NotebookApp] 200 GET /api/contents/ (77.57.125.18) 3.31ms
[D 12:23:50.021 NotebookApp] 200 GET /api/sessions (77.57.125.18) 1.42ms
[D 12:23:52.974 NotebookApp] 200 GET /api/contents/multivariate_time_series_error_detection (77.57.125.18) 3.00ms
[D 12:23:53.800 NotebookApp] 200 GET /api/sessions (77.57.125.18) 0.78ms
[D 12:24:00.403 NotebookApp] 200 GET /api/contents/ (77.57.125.18) 2.38ms
[D 12:24:01.247 NotebookApp] 200 GET /api/sessions (77.57.125.18) 0.85ms
[D 12:24:05.365 NotebookApp] 200 GET /api/contents/Untitled.ipynb (77.57.125.18) 102.54ms
[D 12:24:06.228 NotebookApp] Starting kernel: ['/usr/bin/python3', '-m', 'ipykernel_launcher', '-f', '/run/user/1000/jupyter/kernel-5e5e5f68-dd19-4ae2-ad9a-62c81f9dec3c.json']
[D 12:24:06.231 NotebookApp] Connecting to: tcp://127.0.0.1:36683
[D 12:24:06.232 NotebookApp] Connecting to: tcp://127.0.0.1:41752
[I 12:24:06.233 NotebookApp] Kernel started: 5e5e5f68-dd19-4ae2-ad9a-62c81f9dec3c
[D 12:24:06.233 NotebookApp] Kernel args: {'kernel_name': 'python3', 'cwd': '/home/ubuntu/machine-learning/projects/capstone'}
[D 12:24:06.234 NotebookApp] 201 POST /api/sessions (77.57.125.18) 21.93ms
[D 12:24:06.766 NotebookApp] activity on 5e5e5f68-dd19-4ae2-ad9a-62c81f9dec3c: status
[D 12:24:08.254 NotebookApp] 200 GET /api/contents/ (77.57.125.18) 2.67ms
[D 12:24:09.088 NotebookApp] 200 GET /api/sessions (77.57.125.18) 0.86ms
Below is the config file I'm running on the EC2 instance. The instance is running Ubuntu and has jupyter notebook v5.6.0 installed.
It was working, and now it isn't, but it doesn't quite feel like an ein issue so much as a configuration problem. I'll write more if I can get everything working again.
At the bottom of #163 I've tried to describe a bit what EIN is doing when it opens a websocket connection. One thing you can try doing is call M-x url-cookie-list
in the notebook buffer that is not executing and see if the cookies that appear there have the correct host name and port.
Note there also might be an issue with the certificate for the https connection, especially if the certificate is self-signed. In that case you need to ensure curl is called with the --insecure
argument.
I set the --insecure
argument from request-curl-option
already. I can connect to the remote notebooks and navigate in the tree, open a notebook, but I can't execute its content :-(
As far as the cookies, I see two lines which are really similar to what you showed in the #163, thanks for the comments by the way! I tried also with GCE and get the same problem, so probably not an issue from the provider.
I can tell that this function returns nil
and this is what makes the ein:execute-kernel
stopping:
(defun ein:websocket-open-p (websocket)
(eql (websocket-ready-state (ein:$websocket-ws websocket)) 'open))
When I tried to debug using debug-on-entry
on ein:execute-kernel, I got (long text)
ein:websocket-open-p([cl-struct-ein:$websocket [cl-struct-websocket connecting
#0 #[(ws) "\305H
>\204 \306\307\302D\"\210\310H\211\305H>\204\" \306\307\311\nD\"\210\n\312H\211\205? \313\f\n\305H>\204; \306\307\311\nD\"\210\n\314H\"*\207"
[ws cl-struct-websocket-tags websocket cl-struct-ein:$websocket-tags it 0 signal
wrong-type-argument 2 ein:$websocket 6 apply 7] 7] #[(ws frame) "\306H
>\204 \307\310\305D\"\210\311H\n\306H>\204! \307\310\312\nD\"\210\n\311H\211\306H>\2046 \307\310\313
D\"\210
\311H\211\205[ \f\205[ \314\f
\306H>\204W \307\310\313
D\"\210
\315H#+\207"
[ws cl-struct-websocket-tags frame cl-struct-websocket-frame-tags packet
websocket 0 signal wrong-type-argument 2 websocket-frame ein:$websocket apply 3
cl-struct-ein:$websocket-tags it] 8] #[(ws) "\305H
>\204 \306\307\302D\"\210\310H\211\305H>\204\" \306\307\311\nD\"\210\n\312H\211\205@ \313\f\n\211\305H>\204<
Check list
*.elc
files from source directory of EIN and its dependencies.Description of the problem you have
Hello,
I am using EIN through a AWS server and recently I can't execute any cell anymore in any notebook anymore.
Steps to reproduce the problem
Expected output
Execute the cell as in a local notebook.
Your EIN configuration (in .emacs.d/init.el or somewhere else)
Your IPython configuration
What is your IPython version? (run
ipython --version
):jupyter 4.4.0
How do you start IPython? (e.g.,
ipython notebook --port 9999
):jupyter notebook --port=8888
What is your IPython notebook port number or URL?:
8888
Additional information (if any)
It was working until a few months ago.
System info: