Cannot authenticate websocket with non-empty NotebookApp.base_url

[bjodah]

I've searched through the source code, even though I see references to a variable base-url it is not quite clear to me if I can configure this in ein? Looking through cutomize-group for ein, I did not see any such setting?

The use case is the following: notebook server running with jupyter_notebook_configy.py:

c.NotebookApp.allow_origin = '*'
c.NotebookApp.base_url = '/my-base-url'
c.NotebookApp.port = 8889
c.NotebookApp.ip = '0.0.0.0'
c.NotebookApp.password = 'sha1:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

And nginx is providing this to the outside world (intranet) via a reverse proxy, on port 443 (https), which matches URLs with /my-base-url:

/etc/nginx/snippets/my_jupyter_service.conf

location = /my-base-url {
  rewrite ^/(.*)$ $1/ permanent;                                                                                                                                                                                                                                              
}
location /my-base-url {
  error_page 403 = @proxy_my_jupyter_service;                                                                                                                                                                                                                                 
  deny 127.0.0.1;                                                                                                                                                                                                                                                             
  allow all;                                                                                                                                                                                                                                                                  
  # set a webroot, if there is one
  root /some-webroot;                                                                                                                                                                                                                                                         
  try_files $uri @proxy_my_jupyter_service;                                                                                                                                                                                                                                   
}
location @proxy_my_jupyter_service {
  #rewrite /jupyter(.*) $1  break;                                                                                                                                                                                                                                            
  proxy_read_timeout 300s;                                                                                                                                                                                                                                                    
  proxy_pass http://upstream_my_jupyter_service;                                                                                                                                                                                                                              
  # pass some extra stuff to the backend
  proxy_set_header Host $host;                                                                                                                                                                                                                                                
  proxy_set_header X-Real-Ip $remote_addr;                                                                                                                                                                                                                                    
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;                                                                                                                                                                                                                
}
location ~ /my-base-url/api/kernels/ {
        proxy_pass            http://upstream_my_jupyter_service;                                                                                                                                                                                                             
        proxy_set_header      Host $host;                                                                                                                                                                                                                                     
        # websocket support
        proxy_http_version    1.1;                                                                                                                                                                                                                                            
        proxy_set_header      Upgrade "websocket";                                                                                                                                                                                                                            
        proxy_set_header      Connection "Upgrade";                                                                                                                                                                                                                           
        proxy_read_timeout    86400;                                                                                                                                                                                                                                          
    }
location ~ /my-base-url/terminals/ {
        proxy_pass            http://upstream_my_jupyter_service;                                                                                                                                                                                                             
        proxy_set_header      Host $host;                                                                                                                                                                                                                                     
        # websocket support
        proxy_http_version    1.1;                                                                                                                                                                                                                                            
        proxy_set_header      Upgrade "websocket";                                                                                                                                                                                                                            
        proxy_set_header      Connection "Upgrade";                                                                                                                                                                                                                           
        proxy_read_timeout    86400;                                                                                                                                                                                                                                          
}

/etc/nginx/sites-enables/my-service

upstream upstream_my_jupyter_service {                                                                                                                                                                                                                                       
  server localhost:8889;                                                                                                                                                                                                                                                      
  keepalive 32;                                                                                                                                                                                                                                                               
} 

$ grep /etc/nginx/sites-enabled/default

        include snippets/my_jupyter_service.conf;

and this appears in the main `server { ... }` block

This works great in the web browser. I'd like to access this server using ein too though (locally, from the machine hosting the notebook server), if I run:

M-x ein:login <ret>
URL or port: 8889
Password: ******* <ret>

I get:

ein: [error] Login to http://127.0.0.1:8889 failed, error-thrown (error http 404), raw-header HTTP/1.1 404 Not Found
Server: TornadoServer/6.1
Content-Type: text/html
Date: Thu, 07 Apr 2022 15:28:25 GMT
X-Content-Type-Options: nosniff
Content-Security-Policy: frame-ancestors 'self'; report-uri /my-base-url/api/security/csp-report
Access-Control-Allow-Origin: *
Content-Length: 7417

It feels like I'm potentially very close, and I am probably just missing something obvious. Any guidance on this issue is of course much appreciated. And thank you for your continued work on ein, a marvelous package in my opinion! :)

[bjodah]

Thanks @dickmao,

I had to specify the full url, including http for ein to open my notebook-list. i.e. http://host:port/url-base, but then I could not open actual notebooks (complaints about _xsrf in the jupyter notebook log).

So I set out to reproduce the error in a well specified environment (using linux containers), but it turns out things actaully work just fine with recent versions of all dependencies.

[bjodah]

Turns out there actually seems to be an issue with ein(?) in the case when a notebook server has both:

• A non-empty base_url
• A non-empty password

I updated the repository with the minimal non-working example to highlight this.

@dickmao, is it alright if I re-open this issue?

[bjodah]

Thanks @dickmao, so far things seem to generally work for me too, did you manage to execute any cells?

Alright, I can sympathize with that. But forgive my ignorance, if you typically recommend staying away from jupyter, what would you recommend to run emacs-ipython-notebook against?

(Unfortunately I did not get to decide how I ended up in this base_url / password situtation, but rather due to draconian network/firewall rules in a multi-user environment, where I'm the emacs guy, and my collaborators prefer the web-interface...)

[bjodah]

Oh, I'm not at all sure there's a bug in ein, I might very well be doing something wrong, or the bug resides in one of the dependencies. I really do appreciate you taking the time to look into it, especially since it works in your environment.

But that repository I linked to is pretty much a reproducible environment (short of me locking down exact version numbers pip and package-install downloads). Anyhow, since the environment is "fresh" each time upon running the script, the cookie file does not go stale (since there is no file at the launch of the container). Deleting it invalidates the running session (as expected I guess), but doesn't help for the next session either.

I targeted podman (the docker replacement), here's what the session looks like: https://asciinema.org/a/xR6fxASSDjCIRyAzqzO9NP5Jr it's a bit opaque to me why the WebSocket session fails to authenticate, while the HTTP session, succeeds.

[dickmao]

Your persistence has paid off against my surly dismissiveness. EIN was altogether incapable of handling a non-empty NotebookApp.base_url.

I made a bunch of changes in 6f138be and tested against your container. I had to change "0.0.0.0" to "127.0.0.1" in launch-notebook.sh, and change "localhost" to "127.0.0.1" in launch-emacs.sh to get it to work.

[bjodah]

@dickmao, haha, any surliness of yours is greatly out-weighted by the tenaciousness with which you have followed up my questions and fixed this shortcoming in ein. Very much appreciated!

That commit of yours looks like some (very nice) elisp wizardry (I really should invest the time to become proficient in elisp given how much time i spend in Emacs). I will try your fix right away!

[bjodah]

Works like a charm, thanks again!