search

millejoh / emacs-ipython-notebook

Jupyter notebook client in Emacs
http://millejoh.github.io/emacs-ipython-notebook/
GNU General Public License v3.0
1.47k stars 122 forks source link

error 403 on new notebook #754

Closed sam-s closed 3 years ago

sam-s commented 3 years ago

Problem description

When I click on New Notebook, I get the following remote logs 

[I 16:26:45.092 NotebookApp] 302 GET /login (100.107.210.216) 1.16ms
[I 16:26:45.107 NotebookApp] 302 GET / (100.107.210.216) 0.37ms
[I 16:26:51.040 NotebookApp] Creating new notebook in 
[I 16:26:51.252 NotebookApp] Kernel started: d84ce358-14de-4252-996c-c5f5d45188ef, name: python3
[W 16:26:51.484 NotebookApp] Couldn't authenticate WebSocket connection
[W 16:26:51.485 NotebookApp] 403 GET /api/kernels/d84ce358-14de-4252-996c-c5f5d45188ef/channels?session_id=6c0afeab-b74e-4ad5-9466-a2422c9f04f6 (100.107.210.216) 2.30ms referer=None

and backtrace

Debugger entered--Lisp error: (websocket-closed #s(websocket-frame :opcode pong :payload "2e9fb87e0d8ef1ea96893aa6906147a371bb\" type=\"text/c..." :length nil :completep t))
  signal(websocket-closed (#s(websocket-frame :opcode pong :payload "2e9fb87e0d8ef1ea96893aa6906147a371bb\" type=\"text/c..." :length nil :completep t)))
  (if (websocket-openp websocket) nil (signal 'websocket-closed (list frame)))
  websocket-send(#s(websocket :ready-state open :client-data #s(ein:$websocket :ws #1 :kernel #s(ein:$kernel :url-or-port "https://data-science.k8s.region-001.p-use-1.braze...." :path "Untitled.ipynb" :kernelspec nil :events #<ein:events ein:events-1ff1cbe63972> :api-version 6 :session-id "e89e5459-1173-418f-8d45-c69a2eefd7f2" :kernel-id "bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :shell-channel nil :iopub-channel nil :websocket nil :base-url "/api/kernels" :kernel-url "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :ws-url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :stdin-activep nil :username "username" :msg-callbacks #<hash-table equal 0/65 0x1ff1cbe670ad> :oinfo-cache #<hash-table equal 0/65 0x1ff1cbe66db3> :after-start-hook nil :after-execute-hook nil) :closed-by-client t) :on-open #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>) :on-message #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>) :on-close #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>) :on-error #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>) :negotiated-protocols nil :negotiated-extensions nil :server-p nil :url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :protocols nil :extensions nil :custom-header-alist nil :conn #<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2> :server-conn nil :accept-string "wCTPssTm1pug7dZTTudiz17YgJ4=" :inflight-input "\n              exports: \"typ") #s(websocket-frame :opcode pong :payload "2e9fb87e0d8ef1ea96893aa6906147a371bb\" type=\"text/c..." :length nil :completep t))
  (closure ((opcode . ping) (frame . #s(websocket-frame :opcode ping :payload "2e9fb87e0d8ef1ea96893aa6906147a371bb\" type=\"text/c..." :length 99 :completep nil)) (websocket . #s(websocket :ready-state open :client-data #s(ein:$websocket :ws #6 :kernel #s(ein:$kernel :url-or-port "https://data-science.k8s.region-001.p-use-1.braze...." :path "Untitled.ipynb" :kernelspec nil :events #<ein:events ein:events-1ff1cbe63972> :api-version 6 :session-id "e89e5459-1173-418f-8d45-c69a2eefd7f2" :kernel-id "bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :shell-channel nil :iopub-channel nil :websocket nil :base-url "/api/kernels" :kernel-url "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :ws-url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :stdin-activep nil :username "username" :msg-callbacks #<hash-table equal 0/65 0x1ff1cbe670ad> :oinfo-cache #<hash-table equal 0/65 0x1ff1cbe66db3> :after-start-hook nil :after-execute-hook nil) :closed-by-client t) :on-open #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>) :on-message #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>) :on-close #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>) :on-error #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>) :negotiated-protocols nil :negotiated-extensions nil :server-p nil :url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :protocols nil :extensions nil :custom-header-alist nil :conn #<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2> :server-conn nil :accept-string "wCTPssTm1pug7dZTTudiz17YgJ4=" :inflight-input "\n              exports: \"typ")) cl-struct-websocket-frame-tags cl-struct-websocket-tags t) nil (websocket-send websocket (record 'websocket-frame 'pong (progn (or (progn (and (memq ... cl-struct-websocket-frame-tags) t)) (signal 'wrong-type-argument (list 'websocket-frame frame))) (aref frame 2)) nil t)))()
  funcall((closure ((opcode . ping) (frame . #s(websocket-frame :opcode ping :payload "2e9fb87e0d8ef1ea96893aa6906147a371bb\" type=\"text/c..." :length 99 :completep nil)) (websocket . #s(websocket :ready-state open :client-data #s(ein:$websocket :ws #7 :kernel #s(ein:$kernel :url-or-port "https://data-science.k8s.region-001.p-use-1.braze...." :path "Untitled.ipynb" :kernelspec nil :events #<ein:events ein:events-1ff1cbe63972> :api-version 6 :session-id "e89e5459-1173-418f-8d45-c69a2eefd7f2" :kernel-id "bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :shell-channel nil :iopub-channel nil :websocket nil :base-url "/api/kernels" :kernel-url "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :ws-url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :stdin-activep nil :username "username" :msg-callbacks #<hash-table equal 0/65 0x1ff1cbe670ad> :oinfo-cache #<hash-table equal 0/65 0x1ff1cbe66db3> :after-start-hook nil :after-execute-hook nil) :closed-by-client t) :on-open #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>) :on-message #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>) :on-close #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>) :on-error #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>) :negotiated-protocols nil :negotiated-extensions nil :server-p nil :url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :protocols nil :extensions nil :custom-header-alist nil :conn #<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2> :server-conn nil :accept-string "wCTPssTm1pug7dZTTudiz17YgJ4=" :inflight-input "\n              exports: \"typ")) cl-struct-websocket-frame-tags cl-struct-websocket-tags t) nil (websocket-send websocket (record 'websocket-frame 'pong (progn (or (progn (and ... t)) (signal 'wrong-type-argument (list ... frame))) (aref frame 2)) nil t))))
  (let ((to-process (car --dolist-tail--))) (funcall to-process) (setq --dolist-tail-- (cdr --dolist-tail--)))
  (while --dolist-tail-- (let ((to-process (car --dolist-tail--))) (funcall to-process) (setq --dolist-tail-- (cdr --dolist-tail--))))
  (let ((--dolist-tail-- (nreverse processing-queue))) (while --dolist-tail-- (let ((to-process (car --dolist-tail--))) (funcall to-process) (setq --dolist-tail-- (cdr --dolist-tail--)))))
  (let ((current-frame) (processing-queue) (start-point 0)) (while (setq current-frame (websocket-read-frame (substring text start-point))) (setq processing-queue (cons (websocket-process-frame websocket current-frame) processing-queue)) (setq start-point (+ start-point (progn (or (progn (and ... t)) (signal 'wrong-type-argument (list ... current-frame))) (aref current-frame 3))))) (if (> (length text) start-point) (progn (progn (or (progn (and (memq ... cl-struct-websocket-tags) t)) (signal 'wrong-type-argument (list 'websocket websocket))) (let* ((v websocket)) (aset v 17 (substring text start-point)))))) (let ((--dolist-tail-- (nreverse processing-queue))) (while --dolist-tail-- (let ((to-process (car --dolist-tail--))) (funcall to-process) (setq --dolist-tail-- (cdr --dolist-tail--))))))
  websocket-process-input-on-open-ws(#s(websocket :ready-state open :client-data #s(ein:$websocket :ws #1 :kernel #s(ein:$kernel :url-or-port "https://data-science.k8s.region-001.p-use-1.braze...." :path "Untitled.ipynb" :kernelspec nil :events #<ein:events ein:events-1ff1cbe63972> :api-version 6 :session-id "e89e5459-1173-418f-8d45-c69a2eefd7f2" :kernel-id "bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :shell-channel nil :iopub-channel nil :websocket nil :base-url "/api/kernels" :kernel-url "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :ws-url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :stdin-activep nil :username "username" :msg-callbacks #<hash-table equal 0/65 0x1ff1cbe670ad> :oinfo-cache #<hash-table equal 0/65 0x1ff1cbe66db3> :after-start-hook nil :after-execute-hook nil) :closed-by-client t) :on-open #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>) :on-message #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>) :on-close #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>) :on-error #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>) :negotiated-protocols nil :negotiated-extensions nil :server-p nil :url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :protocols nil :extensions nil :custom-header-alist nil :conn #<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2> :server-conn nil :accept-string "wCTPssTm1pug7dZTTudiz17YgJ4=" :inflight-input "\n              exports: \"typ") "<!DOCTYPE HTML>\n<html>\n\n<head>\n    <meta charset=\"...")
  (progn (websocket-process-input-on-open-ws websocket (substring text (or start-point 0))))
  (if (eq 'open (progn (or (progn (and (memq (type-of websocket) cl-struct-websocket-tags) t)) (signal 'wrong-type-argument (list 'websocket websocket))) (aref websocket 1))) (progn (websocket-process-input-on-open-ws websocket (substring text (or start-point 0)))))
  (let ((start-point) (text (concat (progn (or (progn (and ... t)) (signal 'wrong-type-argument (list ... websocket))) (aref websocket 17)) output)) (header-end-pos)) (progn (or (progn (and (memq (type-of websocket) cl-struct-websocket-tags) t)) (signal 'wrong-type-argument (list 'websocket websocket))) (let* ((v websocket)) (aset v 17 nil))) (if (and (eq 'connecting (progn (or (progn (and ... t)) (signal 'wrong-type-argument (list ... websocket))) (aref websocket 1)))) (progn (if (and (setq header-end-pos (string-match "\15\n\15\n" text)) (setq start-point (+ 4 header-end-pos))) (progn (condition-case err (progn (websocket-verify-response-code text) (websocket-verify-headers websocket text) (websocket-process-headers ... text)) (error (websocket-close websocket) (funcall ... websocket ... err))) (progn (or (progn ...) (signal ... ...)) (let* (...) (aset v 1 ...))) (websocket-try-callback 'websocket-on-open 'on-open websocket)) (progn (or (progn (and ... t)) (signal 'wrong-type-argument (list ... websocket))) (let* ((v websocket)) (aset v 17 text)))))) (if (eq 'open (progn (or (progn (and (memq ... cl-struct-websocket-tags) t)) (signal 'wrong-type-argument (list 'websocket websocket))) (aref websocket 1))) (progn (websocket-process-input-on-open-ws websocket (substring text (or start-point 0))))))
  websocket-outer-filter(#s(websocket :ready-state open :client-data #s(ein:$websocket :ws #1 :kernel #s(ein:$kernel :url-or-port "https://data-science.k8s.region-001.p-use-1.braze...." :path "Untitled.ipynb" :kernelspec nil :events #<ein:events ein:events-1ff1cbe63972> :api-version 6 :session-id "e89e5459-1173-418f-8d45-c69a2eefd7f2" :kernel-id "bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :shell-channel nil :iopub-channel nil :websocket nil :base-url "/api/kernels" :kernel-url "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :ws-url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :stdin-activep nil :username "username" :msg-callbacks #<hash-table equal 0/65 0x1ff1cbe670ad> :oinfo-cache #<hash-table equal 0/65 0x1ff1cbe66db3> :after-start-hook nil :after-execute-hook nil) :closed-by-client t) :on-open #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>) :on-message #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>) :on-close #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>) :on-error #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>) :negotiated-protocols nil :negotiated-extensions nil :server-p nil :url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :protocols nil :extensions nil :custom-header-alist nil :conn #<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2> :server-conn nil :accept-string "wCTPssTm1pug7dZTTudiz17YgJ4=" :inflight-input "\n              exports: \"typ") "HTTP/1.1 403 Forbidden\15\nDate: Wed, 11 Nov 2020 21:...")
  (let ((websocket (process-get process :websocket))) (websocket-outer-filter websocket output))
  (closure ((websocket . #s(websocket :ready-state open :client-data #s(ein:$websocket :ws #4 :kernel ... :closed-by-client t) :on-open #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>) :on-message #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>) :on-close #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>) :on-error #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>) :negotiated-protocols nil :negotiated-extensions nil :server-p nil :url "wss://data-science.k8s.region-001.p-..." :protocols nil :extensions nil :custom-header-alist nil :conn #<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2> :server-conn nil :accept-string "wCTPssTm1pug7dZTTudiz17YgJ4=" :inflight-input "\n              exports: \"typ")) (conn . #<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2>) (key . "jbZIs24ROgwuwwQuX4z/4w==") (url-struct . #s(url :type "wss" :user nil :password nil :host "data-science.k8s.region-001.p-use-1...." :portspec 443 :filename "/api/kernels/bef991cb-ed4f-4e0d-8497..." :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous t)) (name . "websocket to wss://data-science.k8s....") (custom-header-alist) (nowait) (on-error . #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>)) (on-close . #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>)) (on-message . #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>)) (on-open . #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>)) (extensions) (protocols) (--cl-rest-- :on-open #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>) :on-message #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>) :on-close #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>) :on-error #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>)) (url . "wss://data-science.k8s.region-001.p-...") cl-struct-websocket-frame-tags cl-struct-websocket-tags t) (process output) (let ((websocket (process-get process :websocket))) (websocket-outer-filter websocket output)))(#<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2> "HTTP/1.1 403 Forbidden\15\nDate: Wed, 11 Nov 2020 21:...")

Steps to reproduce the problem

had to do

(defun websocket-ensure-connected (websocket)
  nil)

to work around https://github.com/ahyatt/emacs-websocket/issues/75

System info:

("EIN system info"
 :emacs-version
 "GNU Emacs 28.0.50 (build 3, x86_64-apple-darwin19.6.0, NS appkit-1894.60 Version 10.15.7 (Build 19H15))
 of 2020-11-09"
 :emacs-bzr-version "8dc237270f88a6abce4df9a1235b38288792ab71"
 :window-system ns
 :emacs-variant nil
 :os
 (:uname
  "Darwin BZ-C02XR5CGJG5L 19.6.0 Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64 x86_64
"
  :lsb-release nil)
 :notebook "WARNING: Package(s) not found: notebook
"
 :ipython nil
 :image-types
 (imagemagick png gif tiff jpeg xpm xbm pbm)
 :image-types-available
 (imagemagick png gif tiff jpeg xpm xbm pbm)
 :request
 (:backend curl)
 :ein
 (:version "20201028.1857"
       :source-dir "/Users/sam.steingold/.config/emacs/elpa/ein-20201028.1857/")
 :lib
 ((:name "websocket"
     :path "~/.config/emacs/elpa/websocket-20200419.2124/websocket.elc"
     :featurep t
     :version-var websocket-version
     :version "1.12")
  (:name "anaphora"
     :path "~/.config/emacs/elpa/anaphora-20180618.2200/anaphora.elc"
     :featurep t
     :version-var nil
     :version nil)
  (:name "request"
     :path "~/.config/emacs/elpa/request-20201026.2324/request.elc"
     :featurep t
     :version-var request-version
     :version "0.3.0")
  (:name "deferred"
     :path "~/.config/emacs/elpa/deferred-20170901.1330/deferred.elc"
     :featurep t
     :version-var deferred:version
     :version "0.5.0")
  (:name "polymode"
     :path "~/.config/emacs/elpa/polymode-20200606.1106/polymode.elc"
     :featurep t
     :version-var nil
     :version nil)
  (:name "exec-path-from-shell"
     :path
     "~/.config/emacs/elpa/exec-path-from-shell-20201105.2236/exec-path-from-shell.elc"
     :featurep t
     :version-var nil
     :version nil)
  (:name "dash"
     :path "~/.config/emacs/elpa/dash-20200803.1520/dash.elc"
     :featurep t
     :version-var nil
     :version nil)
  (:name "with-editor"
     :path "~/.config/emacs/elpa/with-editor-20201030.1232/with-editor.elc"
     :featurep nil
     :version-var nil
     :version nil)))
dickmao commented 3 years ago

If

grep 100.107.210.216 ~/.emacs.d/request/curl-cookie-jar | grep username

returns nothing, which is what I expect, then you need to look at the kernel json files in jupyter --runtime-dir on the server machine. You have claimed your server is "nothing special" but I remain unconvinced.

If it does return an entry, and that entry isn't reflected in M-x url-cookie-list, then there's a problem in ein-websocket.el. This seems unlikely, not because ein-websocket.el is particulary well written, but rather because there's not much that could break there.

sam-s commented 3 years ago

the relevant line in ~/.config/emacs/request/curl-cookie-jar is

data-science.k8s.region-001.p-use-1.braze.com   FALSE   /   FALSE   0   _xsrf   2|682c5193|ZZZZ|1603214402

moreover, ein:notebooklist-login does not ask me for a password anymore (and shows the list of notebooks). however, url-cookie-list fails with "No cookies are defined".

the kernel file created in response to the ein request above is kernel-d84ce358-14de-4252-996c-c5f5d45188ef.json:

{
  "shell_port": 55941,
  "iopub_port": 38979,
  "stdin_port": 34231,
  "control_port": 48685,
  "hb_port": 48739,
  "ip": "127.0.0.1",
  "key": "a9b98a0a-dad9102e9e463e610c965357",
  "transport": "tcp",
  "signature_scheme": "hmac-sha256",
  "kernel_name": ""
}

the chrome web interface seems to have no problems interacting with this kernel.

dickmao commented 3 years ago 
 data-science.k8s.region-001.p-use-1.braze.com  FALSE   /   FALSE   0   _xsrf

That's the cross-site forgery token, which is not relevant. I would rather see something like:

#HttpOnly_127.0.0.1 FALSE   /   FALSE   1607728889  username-127-0-0-1-8317

Thus it appears your server does not ask for a user, which would account for tornado complaining with 403, no current user.

however, url-cookie-list fails with "No cookies are defined".

That is perturbing.

the kernel file created in response to the ein request above is

Hmm.. that didn't help me. Maybe you could report back with the nbserver.*.json

dickmao commented 3 years ago

An empty url-cookie-list, btw, is almost certainly a showstopper. I would M-x find-library RET url-cookie and see if you have one of the various defcustoms in there set to a suspect value in your .emacs.

sam-s commented 3 years ago 
#HttpOnly_127.0.0.1   FALSE   /   FALSE   1607728889  username-127-0-0-1-8317
#HttpOnly_data-science.k8s.region-001.p-use-1.braze.com FALSE   /   FALSE   1607481770  username-data-science-k8s-region-001-p-use-1-braze-com  "2|1:0|10:1604889770|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:XXXXX=|YYYYYY"

Thus it appears your server does not ask for a user, which would account for tornado complaining with 403, no current user.

I have a modicum of control over how the server is run. Now it is run as

jupyter notebook --port=8080 --ip=0.0.0.0 --allow-root \
     --NotebookApp.iopub_data_rate_limit=1.0e10 \
     --NotebookApp.iopub_msg_rate_limit=1.0e10

How do you want me to run it?

however, url-cookie-list fails with "No cookies are defined".

That is perturbing. An empty url-cookie-list, btw, is almost certainly a showstopper. I would M-x find-library RET url-cookie and see if you have one of the various defcustoms in there set to a suspect value in your .emacs.

I don't set anything url-cookie-* in my .emacs; moreover, I run emacs -Q so that cannot be relevant.

moreover, when I run 

emacs -Q -f package-initialize --eval "(setq debug-on-error t)" --load websocket.elc --eval '(url-cookie-parse-file-netscape (expand-file-name "request/curl-cookie-jar" user-emacs-directory))'

and M-x url-cookie-list includes

data-science.k8s.region-001.p-use-1.braze.com     _xsrf                2|682c5193|ZZZZ|1603214402
data-science.k8s.region-001.p-use-1.braze.com:443 _xsrf                2|682c5193|ZZZZ|1603214402
                                                  username-data-scienc "2|1:0|10:1604889770|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:XXXX=|YYYY"

I still get the same error

Hmm.. that didn't help me. Maybe you could report back with the nbserver.*.json

{
  "base_url": "/",
  "hostname": "0.0.0.0",
  "notebook_dir": "/root/data/data-science/jupyter",
  "password": false,
  "pid": 1,
  "port": 8080,
  "secure": false,
  "sock": "",
  "token": "ZZZZ",
  "url": "http://0.0.0.0:8080/"
}
dickmao commented 3 years ago

I know as little about cookies as the next non-web programmer, but artificially parsing an old jar file is mildly suspect. It's conceivable the server insists on new cookies for each new session.

If it does return an entry, and that entry isn't reflected in M-x url-cookie-list, then there's a problem in ein-websocket.el.

So it sounds like this is the case. The jar file has the username entry, but the in-memory url-cookie-list doesn't, and worse is completely empty, which is very suspect. You should C-u C-M-x in https://github.com/millejoh/emacs-ipython-notebook/blob/bf1ca5b9b9da84849a1a6fb1057455814c7bfa33/lisp/ein-websocket.el#L73 and see why url-cookie-store isn't firing.

sam-s commented 3 years ago

I know as little about cookies as the next non-web programmer, but artificially parsing an old jar file is mildly suspect. It's conceivable the server insists on new cookies for each new session.

this is not an old file - its mtime is today.

If it does return an entry, and that entry isn't reflected in M-x url-cookie-list, then there's a problem in ein-websocket.el.

So it sounds like this is the case. The jar file has the username entry, but the in-memory url-cookie-list doesn't, and worse is completely empty, which is very suspect. You should C-u C-M-x in

https://github.com/millejoh/emacs-ipython-notebook/blob/bf1ca5b9b9da84849a1a6fb1057455814c7bfa33/lisp/ein-websocket.el#L73

and see why url-cookie-store isn't firing.

ein:websocket--prepare-cookies does not call url-cookie-store directly. cookies is computed and has a reasonable value:

(("username-data-science-k8s-region-001-p-use-1-braze..." . "\"2|1:0|10:1604889770|54:username-data-science-k8s-...") ("_xsrf" . "2|682c5193|ZZZZ|160321..."))

ein:websocket-store-cookie is called twice and each time calls url-cookie-store successfully:

======================================================================
1 -> (url-cookie-store "username-data-science-k8s-region-001-p-use-1-braze-com" "\"2|1:0|10:1604889770|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:ZZZ=|XXX\"" nil "data-science.k8s.region-001.p-use-1.braze.com:443" "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels" 0)
1 <- url-cookie-store: nil
======================================================================
1 -> (url-cookie-store "_xsrf" "2|682c5193|YYY|1603214402" nil "data-science.k8s.region-001.p-use-1.braze.com:443" "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels" 0)
1 <- url-cookie-store: nil
dickmao commented 3 years ago

It sounds like you are continuing along the path of woe of force parsing (url-cookie-parse-file-netscape (expand-file-name "request/curl-cookie-jar" user-emacs-directory)). I would not do this. I know the cookies are not chronologically old, but every new websocket instance may require a brand new cookie. Who knows? I certainly don't.

Starting from scratch, you said M-x url-cookie-list was empty. I would try to find out why that is.

sam-s commented 3 years ago

It sounds like you are continuing along the path of woe of force parsing (url-cookie-parse-file-netscape (expand-file-name "request/curl-cookie-jar" user-emacs-directory)).

no, I am not doing that.

Starting from scratch, you said M-x url-cookie-list was empty. I would try to find out why that is.

emacs -Q -f package-initialize --eval "(setq debug-on-error t)" --eval '(load "websocket")' --eval '(defun websocket-ensure-connected (websocket) nil)'
M-x ein:notebooklist-login RET https://data-science.k8s.region-001.p-use-1.braze.com/ RET
M-x url-cookie-list RET
==> (error "No cookies are defined")
[New Notebook]
==> (websocket-closed #s(websocket-frame :opcode pong :payload "2e9fb87e0d8ef1ea96893aa6906147a371bb\" type=\"text/c..." :length nil :completep t))
M-x url-cookie-list RET
==> 
data-science.k8s.region-001.p-use-1.braze.com:443 _xsrf                2|682c5193|ZZZ|1603214402
                                                  username-data-scienc "2|1:0|10:1604889770|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:XXX=|YYY"
sam-s commented 3 years ago 
emacs -Q -f package-initialize --eval "(setq debug-on-error t)" --eval '(load "websocket")' --eval '(defun websocket-ensure-connected (websocket) nil)'
M-x trace-function RET url-cookie-store RET
M-x ein:notebooklist-login RET https://data-science.k8s.region-001.p-use-1.braze.com/ RET

no *trace-output*.

[New Notebook]
==> (websocket-closed #s(websocket-frame :opcode pong :payload "2e9fb87e0d8ef1ea96893aa6906147a371bb\" type=\"text/c..." :length nil :completep t))

and now *trace-output* is

======================================================================
1 -> (url-cookie-store "_xsrf" "2|682c5193|XXX|1603214402" nil "data-science.k8s.region-001.p-use-1.braze.com:443" "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels" 0)
1 <- url-cookie-store: ((#1="data-science.k8s.region-001.p-use-1.braze.com:443" [url-cookie "_xsrf" "2|682c5193|XXX|1603214402" nil "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels" #1# 0]))
======================================================================
1 -> (url-cookie-store "username-data-science-k8s-region-001-p-use-1-braze-com" "\"2|1:0|10:1604889770|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:YYY=|ZZZ\"" nil "data-science.k8s.region-001.p-use-1.braze.com:443" "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels" 0)
1 <- url-cookie-store: ([url-cookie "username-data-science-k8s-region-001-p-use-1-braze-com" "\"2|1:0|10:1604889770|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:YYY=|ZZZ\"" nil "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels" #1="data-science.k8s.region-001.p-use-1.braze.com:443" 0] [url-cookie "_xsrf" "2|682c5193|XXX|1603214402" nil "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels" #1# 0])
dickmao commented 3 years ago

Hum, that output looks good.

Your token is non-empty "ZZZZ". Your server should not just let you in without asking for it.

I've seen inconsistent behaviors when invoking the server with --ip=0.0.0.0. If you omit that switch will the server

  1. require the ZZZZ token, and
  2. stop complaining about 403

?

I am approaching the limits of what I can advise remotely. On-site troubleshooting is possible for a nominal fee.

sam-s commented 3 years ago

Your token is non-empty "ZZZZ". Your server should not just let you in without asking for it.

this is the token in nbserver-1.json, it was asked the 1st time I used ein to connect to this server, then it was saved in request/curl-cookie-jar and never asked again.

I've seen inconsistent behaviors when invoking the server with --ip=0.0.0.0. If you omit that switch will the server

  1. require the ZZZZ token, and
  2. stop complaining about 403

I have been using --ip=0.0.0.0 since times immemorial; never had a problem. moreover, I do need to supply something there, otherwise only local connections would be allowed and I am very much remote there...

I can force being asked for the token by removing curl-cookie-jar...

I am approaching the limits of what I can advise remotely. On-site troubleshooting is possible for a nominal fee.

The fee is probably not a problem, access is.

dickmao commented 3 years ago

I would be surprised if deleting the cookie jar suddenly caused the server to require ZZZZ. My understanding was the "token" has nothing to do with the cookie business. Either way, it would be instructive to know if a server requiring ZZZZ is better behaved (doesn't 403).

sam-s commented 3 years ago

huh, I renamed cookie jar and repeated https://github.com/millejoh/emacs-ipython-notebook/issues/754#issuecomment-726364156 I was asked for password, I entered 288XXXX, found in nbserver-1.json. Here is *trace-output*:

======================================================================
1 -> (url-cookie-store "username-data-science-k8s-region-001-p-use-1-braze-com" "\"2|1:0|10:1605223426|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:ZmIXXXX=|eb5XXX\"" nil "data-science.k8s.region-001.p-use-1.braze.com:443" "/api/kernels/d84ce358-14de-4252-996c-c5f5d45188ef/channels" 0)
1 <- url-cookie-store: ((#1="data-science.k8s.region-001.p-use-1.braze.com:443" [url-cookie "username-data-science-k8s-region-001-p-use-1-braze-com" "\"2|1:0|10:1605223426|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:ZmIXXX=|eb5XXX\"" nil "/api/kernels/d84ce358-14de-4252-996c-c5f5d45188ef/channels" #1# 0]))
======================================================================
1 -> (url-cookie-store "_xsrf" "2|046f02b3|01ff161c84d0637bae97d257f35e12c4|1605223410" nil "data-science.k8s.region-001.p-use-1.braze.com:443" "/api/kernels/d84ce358-14de-4252-996c-c5f5d45188ef/channels" 0)
1 <- url-cookie-store: ([url-cookie "_xsrf" "2|046f02b3|01ffXXX|1605223410" nil "/api/kernels/d84ce358-14de-4252-996c-c5f5d45188ef/channels" #1="data-science.k8s.region-001.p-use-1.braze.com:443" 0] [url-cookie "username-data-science-k8s-region-001-p-use-1-braze-com" "\"2|1:0|10:1605223426|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:ZmIXXX=|eb5XXX\"" nil "/api/kernels/d84ce358-14de-4252-996c-c5f5d45188ef/channels" #1# 0])

note that the password/token (288XXX.....) is not in the cookie jar. and, of course

(websocket-closed #s(websocket-frame :opcode pong :payload "2e9XXXX\" type=\"text/c..." :length nil :completep t))

2e9XXX is not in the jar either...

dickmao commented 3 years ago

Sorry, I can't help you. It'd be a moderate hassle but replicating the server configuration onto your local laptop and attempting to replicate the problem wholly on one machine might be very instructive.

sam-s commented 3 years ago

I would be surprised if deleting the cookie jar suddenly caused the server to require ZZZZ.

That's what did happen!

My understanding was the "token" has nothing to do with the cookie business.

My understanding is that when a server authenticates a user, it sets a cookie on the client which the client is supposed to submit in all future communications in lieu of re-authentication (this is why intercepting cookies is so dangerous). @ahyatt - could you please confirm?

Either way, it would be instructive to know if a server requiring ZZZZ is better behaved (doesn't 403).

No change.

dickmao commented 3 years ago

My understanding was the "token" has nothing to do with the cookie business.

Ah, you're right. The above statement is a complete falsehood.

sam-s commented 3 years ago

just to make it clear - the problem is HTTP 403 error when connecting to jupyter server behind a proxy

ahyatt commented 3 years ago

Are you familiar with https://github.com/ahyatt/emacs-websocket/issues/73? Right now websockets don't work correctly with the rest of emacs connection proxying. There's a fix, but it's kind of half-baked at the moment. You can try the pull request in the fix and see if it helps you.

About server authentication & cookies - this really depends on what exactly the server is doing. Normal HTTP authentication doesn't use cookies, but does requires the Authorization header to be included on each request. However, I'm sure there are many variants, and I'm not super well informed about them.

You should authenticate via whatever HTTP method your server supports at the beginning of the websocket session. Further communication shouldn't need anything, and it's only needed again when you re-connect. Those re-connections should be rare, since the websockets can last a while.