search

rocker-org / rocker-versioned2

Run current & prior versions of R using docker. rocker/r-ver, rocker/rstudio, rocker/shiny, rocker/tidyverse, and so on.
https://rocker-project.org
GNU General Public License v2.0
405 stars 165 forks source link

How to use with JupyterHub #515

Closed ktaletsk closed 1 year ago

ktaletsk commented 2 years ago

I am looking at instructions on how to use this image with JupyterHub (+ KubeSpawner). I am able to run this image standalone on my machine and get to RStudio, I am able to launch it in Binder and get to RStudio, but not in JupyterHub. I added the image built from this repo to the list of profiles in KubeSpawner in jupyterhub-config.py

c.KubeSpawner.profile_list.extend(
    [
        {
            "display_name": "RStudio",
            "slug": "rstudio",
            "kubespawner_override": {
                "image": "ktaletsk/rstudio:latest",
                "url": "/rstudio",
            },
        },
    ]
)

When I launch this profile, container is successfully pulled and launched, however in browser JupyterHub redirects to a broken page image

This is the log from the K8s pod:

[I 17:25:46.340 NotebookApp] Writing notebook server cookie secret to /home/rstudio/.local/share/jupyter/runtime/notebook_cookie_secret
[W 2022-08-12 17:25:47.173 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2022-08-12 17:25:47.173 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2022-08-12 17:25:47.183 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.8/dist-packages/jupyterlab
[I 2022-08-12 17:25:47.183 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 17:25:47.189 NotebookApp] Serving notebooks from local directory: /home/jovyan
[I 17:25:47.189 NotebookApp] Jupyter Notebook 6.4.12 is running at:
[I 17:25:47.189 NotebookApp] http://<K8s_pod_name>:8888/?token=172c85d49ffb8f8a036f51d3fd7c15d7e5fe474002a06fbf
[I 17:25:47.189 NotebookApp]  or http://127.0.0.1:8888/?token=172c85d49ffb8f8a036f51d3fd7c15d7e5fe474002a06fbf
[I 17:25:47.189 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 17:25:47.193 NotebookApp] No web browser found: could not locate runnable browser.
[C 17:25:47.193 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/rstudio/.local/share/jupyter/runtime/nbserver-6-open.html
    Or copy and paste one of these URLs:
        http://<pod_name>:8888/?token=172c85d49ffb8f8a036f51d3fd7c15d7e5fe474002a06fbf
     or http://127.0.0.1:8888/?token=172c85d49ffb8f8a036f51d3fd7c15d7e5fe474002a06fbf
[I 17:25:49.745 NotebookApp] 302 GET /user/<jupyterhub_username>/ (192.168.21.10) 0.630000ms
[I 17:25:49.976 NotebookApp] 302 GET /user/<jupyterhub_username>/ (192.168.21.10) 0.620000ms
[W 17:25:50.112 NotebookApp] 404 GET /user/<jupyterhub_username> (192.168.21.10) 43.370000ms referer=https://<deployment_url>/hub/spawn-pending/<jupyterhub_username>
[W 17:26:05.159 NotebookApp] 404 GET /user/<jupyterhub_username>/lab (192.168.21.10) 1.790000ms referer=None
[I 18:14:45.678 NotebookApp] 302 GET /user/<jupyterhub_username>/ (192.168.21.10) 0.880000ms
[W 18:14:45.767 NotebookApp] 404 GET /user/<jupyterhub_username> (192.168.21.10) 1.670000ms referer=None
[I 18:54:46.715 NotebookApp] 302 GET /user/<jupyterhub_username>/ (192.168.21.10) 0.590000ms
[W 18:54:46.804 NotebookApp] 404 GET /user/<jupyterhub_username> (192.168.21.10) 1.940000ms referer=None
[W 19:02:17.283 NotebookApp] 404 GET /user/<jupyterhub_username>/rsutdio (192.168.21.10) 1.710000ms referer=None

Furthermore, I can connect to my pod by running proxy and verify that RStudio web UI is accessible

kubectl port-forward pod/<pod_name> 8080:8888

image

I can conclude that either is something is wrong with the image or JupyterHub configuration. For my normal JupyterLab image, I use the following profile:

"kubespawner_override": {
        "cmd": ["jupyter-labhub"],
        "args": ["--collaborative"],
        "url": "/lab",
    },

which requires JupyterHub to be installed in the spawned image. I tried to install JupyterHub in this repo's image, but encountered errors with pip:

pip install log ### Full log: ```bash rstudio@5fb651f7f41c:~$ pip install jupyterhub Collecting jupyterhub Downloading jupyterhub-2.3.1-py3-none-any.whl (3.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.7/3.7 MB 16.0 MB/s eta 0:00:00 Requirement already satisfied: python-dateutil in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (2.8.2) Collecting SQLAlchemy>=1.1 Downloading SQLAlchemy-1.4.40-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 29.2 MB/s eta 0:00:00 Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (2.28.0) Collecting oauthlib>=3.0 Downloading oauthlib-3.2.0-py3-none-any.whl (151 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.5/151.5 kB 22.3 MB/s eta 0:00:00 Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (21.3) Collecting jupyter-telemetry>=0.1.0 Downloading jupyter_telemetry-0.1.0-py3-none-any.whl (7.1 kB) Collecting pamela Downloading pamela-1.0.0-py2.py3-none-any.whl (5.8 kB) Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (0.4) Requirement already satisfied: traitlets>=4.3.2 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (5.3.0) Requirement already satisfied: prometheus-client>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (0.14.1) Requirement already satisfied: tornado>=5.1 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (6.1) Collecting certipy>=0.1.2 Downloading certipy-0.1.3-py3-none-any.whl (22 kB) Collecting async-generator>=1.9 Downloading async_generator-1.10-py3-none-any.whl (18 kB) Collecting alembic>=1.4 Downloading alembic-1.8.1-py3-none-any.whl (209 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.8/209.8 kB 23.5 MB/s eta 0:00:00 Requirement already satisfied: jinja2>=2.11.0 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (3.1.2) Collecting Mako Downloading Mako-1.2.1-py3-none-any.whl (78 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.6/78.6 kB 9.4 MB/s eta 0:00:00 Requirement already satisfied: importlib-resources in /usr/local/lib/python3.8/dist-packages (from alembic>=1.4->jupyterhub) (5.8.0) Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.8/dist-packages (from alembic>=1.4->jupyterhub) (4.11.4) Collecting pyopenssl Downloading pyOpenSSL-22.0.0-py2.py3-none-any.whl (55 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.8/55.8 kB 12.1 MB/s eta 0:00:00 Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2>=2.11.0->jupyterhub) (2.1.1) Collecting python-json-logger Downloading python_json_logger-2.0.4-py3-none-any.whl (7.8 kB) Requirement already satisfied: jsonschema in /usr/local/lib/python3.8/dist-packages (from jupyter-telemetry>=0.1.0->jupyterhub) (4.6.0) Collecting ruamel.yaml Downloading ruamel.yaml-0.17.21-py3-none-any.whl (109 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.5/109.5 kB 18.1 MB/s eta 0:00:00 Collecting greenlet!=0.4.17 Downloading greenlet-1.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (156 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 156.6/156.6 kB 23.3 MB/s eta 0:00:00 Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->jupyterhub) (3.0.9) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/dist-packages (from python-dateutil->jupyterhub) (1.16.0) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->jupyterhub) (3.3) Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.8/dist-packages (from requests->jupyterhub) (2.0.12) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->jupyterhub) (1.26.9) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->jupyterhub) (2022.6.15) Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata->alembic>=1.4->jupyterhub) (3.8.0) Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema->jupyter-telemetry>=0.1.0->jupyterhub) (0.18.1) Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema->jupyter-telemetry>=0.1.0->jupyterhub) (21.4.0) Collecting cryptography>=35.0 Downloading cryptography-37.0.4-cp36-abi3-manylinux_2_24_x86_64.whl (4.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.1/4.1 MB 31.1 MB/s eta 0:00:00 Collecting ruamel.yaml.clib>=0.2.6 Downloading ruamel.yaml.clib-0.2.6-cp38-cp38-manylinux1_x86_64.whl (570 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 570.4/570.4 kB 32.5 MB/s eta 0:00:00 Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.8/dist-packages (from cryptography>=35.0->pyopenssl->certipy>=0.1.2->jupyterhub) (1.15.0) Requirement already satisfied: pycparser in /usr/local/lib/python3.8/dist-packages (from cffi>=1.12->cryptography>=35.0->pyopenssl->certipy>=0.1.2->jupyterhub) (2.21) Installing collected packages: pamela, ruamel.yaml.clib, python-json-logger, oauthlib, Mako, greenlet, async-generator, SQLAlchemy, ruamel.yaml, cryptography, pyopenssl, jupyter-telemetry, alembic, certipy, jupyterhub ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/usr/local/bin/mako-render' Consider using the `--user` option or check the permissions. --- Logging error --- Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/pip/_internal/utils/logging.py", line 177, in emit self.console.print(renderable, overflow="ignore", crop=False, style=style) File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/rich/console.py", line 1752, in print extend(render(renderable, render_options)) File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/rich/console.py", line 1390, in render for render_output in iter_render: File "/usr/local/lib/python3.8/dist-packages/pip/_internal/utils/logging.py", line 134, in __rich_console__ for line in lines: File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/rich/segment.py", line 245, in split_lines for segment in segments: File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/rich/console.py", line 1368, in render renderable = rich_cast(renderable) File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/rich/protocol.py", line 36, in rich_cast renderable = cast_method() File "/usr/local/lib/python3.8/dist-packages/pip/_internal/self_outdated_check.py", line 130, in __rich__ pip_cmd = get_best_invocation_for_this_pip() File "/usr/local/lib/python3.8/dist-packages/pip/_internal/utils/entrypoints.py", line 58, in get_best_invocation_for_this_pip if found_executable and os.path.samefile( File "/usr/lib/python3.8/genericpath.py", line 101, in samefile s2 = os.stat(f2) FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/pip3.8' Call stack: File "/usr/local/bin/pip", line 8, in sys.exit(main()) File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/main.py", line 70, in main return command.main(cmd_args) File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/base_command.py", line 101, in main return self._main(args) File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/base_command.py", line 223, in _main self.handle_pip_version_check(options) File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/req_command.py", line 148, in handle_pip_version_check pip_self_version_check(session, options) File "/usr/local/lib/python3.8/dist-packages/pip/_internal/self_outdated_check.py", line 237, in pip_self_version_check logger.info("[present-rich] %s", upgrade_prompt) File "/usr/lib/python3.8/logging/__init__.py", line 1446, in info self._log(INFO, msg, args, **kwargs) File "/usr/lib/python3.8/logging/__init__.py", line 1589, in _log self.handle(record) File "/usr/lib/python3.8/logging/__init__.py", line 1599, in handle self.callHandlers(record) File "/usr/lib/python3.8/logging/__init__.py", line 1661, in callHandlers hdlr.handle(record) File "/usr/lib/python3.8/logging/__init__.py", line 954, in handle self.emit(record) File "/usr/local/lib/python3.8/dist-packages/pip/_internal/utils/logging.py", line 179, in emit self.handleError(record) Message: '[present-rich] %s' Arguments: (UpgradePrompt(old='22.1.2', new='22.2.2'),) ```

If anyone has experience running this image in JupyterHub and/or can provide a minimal working example of that, I would really appreciate it and will help to update the documentation on this repo to help others.

eitsupi commented 2 years ago

@yuvipanda Do you know anything about this? Thanks.

ktaletsk commented 2 years ago

@eitsupi @yuvipanda any hints?

eitsupi commented 2 years ago

@ktaletsk Sorry, but I don't use k8s or JupyterHub, so I have nothing to say about this. You might be more likely to get an answer if you ask this in the Jupyter community e.g. https://github.com/jupyterhub/jupyter-rsession-proxy

ktaletsk commented 1 year ago

I think the issue is routing. JupyterHub in my configuration sends user to /user/< jupyterhub_username >/rstudio, but the only path that works in container is /rstudio (or /lab for that matter). I wonder if there is a configuration in JupyterHub I can change to make it work.

Binder seem to have no issues with this URL scheme: user/rocker-org-binder-0g8a6gu4/rstudio/