Jupyter via jupyterhub-proxy with multi-user sessions - how to structure TRAMP/dired hostname/path to make it work

[kubatyszko]

Hello, we have jupyterhub gateway that authenticates users and spawns jupyter notebooks in separate Kubernetes pods per user, this results in individual user's jupyter using longer URL than simply "/"

for example: https://hub.company.com/user/kuba/lab?

It seems that tramp and the emacs-jupyter assume that jupyter always uses root, and it won't let me specify the "/user/kuba/lab" path anywhere. The api path is right under there, so the full path to the api "socket" would be "/user/kuba/lab/api".

excerpt from readme: If :session is a TRAMP file name like /jpy:localhost#8888:NAME it is interpreted as corresponding to a connection to a kernel through a Jupyter notebook server located at http://localhost:8888. Is this expected and intentional, how do I go about fixing this ? Thanks

[nnicandro]

Currently the JupyterHub REST API isn't supported, but it may be possible to get at the Jupyter channels behind the /user/kuba/lab/api path even though the API isn't supported, e.g. using a reverse proxy as explained below. I can already see an issue with how to get the cookies needed by JupyterHub, so maybe a reverse proxy won't work.

I have made progress supporting the JupyterHub REST API, see https://github.com/dzop/emacs-jupyter/tree/hub-api-integration, but have not merged the changes yet since I have not needed them. I think there are a few TODOs left also, e.g. supporting named servers. If you (or anyone) would like to work on bringing in those changes, I would be happy to answer any questions that come up. Once those changes have been merged, you will be able to use /jpy:<username>@host#8888 to connect to a notebook at http://host:8888/user/<username>, see 020b812bf6dec42e39159f588ff5c0774257d505.

Since the reverse proxy trick is useful below is an Apache httpd configuration that I previously used for setting one up on my local machine. It will allow you to redirect local network requests, e.g. made to a local host like local.jpy, to a location like https://hub.company.com/user/kuba/lab. So after setting up a configuration similar to the one below and launching httpd, you could use something like /jpy:local.jpy: to refer to a notebook at https://hub.company.com/user/kuba/lab.

The configuration is mostly a stripped down version of this one that uses HTTP instead of HTTPS since I only needed to do localhost redirection. Not sure if it will work for your purposes though, you may need to adapt it.

LoadModule proxy_module lib/httpd/modules/mod_proxy.so
LoadModule xml2enc_module lib/httpd/modules/mod_xml2enc.so
LoadModule proxy_http_module lib/httpd/modules/mod_proxy_http.so
# Convert URLs that match the host into virtual host URLs in any HTML payloads
LoadModule proxy_html_module lib/httpd/modules/mod_proxy_html.so
<IfModule proxy_html_module>
Include /usr/local/etc/httpd/extra/proxy-html.conf
</IfModule>
LoadModule allowmethods_module lib/httpd/modules/mod_allowmethods.so
LoadModule proxy_wstunnel_module lib/httpd/modules/mod_proxy_wstunnel.so

Listen 80
# Allow websocket connections via HTTP headers
RegisterHTTPMethod WS

# Add a fallback VirtualHost so that the notebook doesn't leak to http://localhost
<VirtualHost *:80>
    <Location "/">
        Order deny,allow
        Deny from all
        Allow from localhost
    </Location>
</VirtualHost>

<VirtualHost *:80>
    ServerName local.jpy
    KeepAliveTimeout 3600
    ProxyPreserveHost On
    ProxyRequests Off
    <Location "/">
        Order deny,allow
        Deny from all
        Allow from localhost
        AllowMethods GET PUT PATCH POST DELETE WS
        # Convert requests to local.jpy/ into requests to http://localhost:8888/
        ProxyPass "http://localhost:8888/"
        # Modify response headers from http://localhost:8888/ to point to local.jpy/
        ProxyPassReverse "http://localhost:8888/"
    </Location>
    <Location "/api/kernels">
        ProxyPass "ws://localhost:8888/api/kernels"
        ProxyPassReverse "ws://localhost:8888/api/kernels"
    </Location>
</VirtualHost>

A few points:

1. The file path /usr/local/etc/httpd/extra/proxy-html.conf should be changed to match the one on your system.

2. I had to add a line like

   127.0.0.1 local.jpy

   to /etc/hosts for host name resolution to work on local.jpy.

3. I couldn't get past the cross origin checks in jupyter/notebook so I had to set

   c.NotebookApp.allow_origin = '*'

   in ~/.jupyter/jupyter_notebook_config.py. This is also mentioned in the linked configuration.