search

jupyterhub / yarnspawner

Spawn JupyterHub single user notebook servers in Hadoop/YARN containers.
https://jupyterhub-yarnspawner.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
19 stars 16 forks source link

Hub and single-user notebook fail to connect out-of-the-box #24

Open ajs6f opened 1 year ago

ajs6f commented 1 year ago

Thanks to much help from @mahendrapaipuri at this issue I was able to fix this bug. I can prepare a PR if that is desired.

Bug description

There is an error in YarnSpawner's https://github.com/jupyterhub/yarnspawner/blob/1c40f529ddf9fd6217e9b8ae954fd669e4276c92/yarnspawner/singleuser.py#L15-L19

After v3 of JupyterHub, HubAuth._api_request() is now an async method.

A corrected YarnSingleUserNotebookApp.start(self) looks like: 

    def start(self):
        self.oauth_callback_handler_class = HubOAuthCallbackHandler

        hub_auth = HubOAuth()
        url = url_path_join(hub_auth.api_url, "yarnspawner")
        headers = {"Authorization": f"token {hub_auth.api_token}"}
        r = requests.post(
               url,
               headers=headers,
               json={"port": self.port},
         )
        super().start()

Expected behaviour

I thought the hub would correctly wait for the single-user notebook to be spawned and then communicate therewith.

Actual behaviour

The single-user notebook spawns, the hub and notebook exchange activity messages, but the hub never correctly returns from the notebook spawn and eventually kills the notebook, even though it was running perfectly well.

How to reproduce

Install and configure YarnSpawner. This will happen every time.

Your personal set up

CentOS Linux release 7.9.2009 (Core)

# packages in environment at /cm/shared/datalake/jupyterhub/miniconda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
abseil-cpp                20211102.0           h27087fc_1    conda-forge
alembic                   1.11.1             pyhd8ed1ab_0    conda-forge
anyio                     3.7.0              pyhd8ed1ab_1    conda-forge
argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0          py310h5764c6d_2    conda-forge
asttokens                 2.0.5              pyhd3eb1b0_0  
async_generator           1.10                       py_0    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
babel                     2.11.0          py310h06a4308_0  
backcall                  0.2.0              pyhd3eb1b0_0  
beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
bleach                    6.0.0              pyhd8ed1ab_0    conda-forge
blinker                   1.6.2              pyhd8ed1ab_0    conda-forge
boltons                   23.0.0          py310h06a4308_0  
brotlipy                  0.7.0           py310h7f8727e_1002  
bzip2                     1.0.8                h7b6447c_0  
c-ares                    1.19.0               h5eee18b_0  
ca-certificates           2023.05.30           h06a4308_0  
certifi                   2023.5.7        py310h06a4308_0  
certipy                   0.1.3                      py_0    conda-forge
cffi                      1.15.1          py310h5eee18b_3  
charset-normalizer        2.0.4              pyhd3eb1b0_0  
comm                      0.1.2           py310h06a4308_0  
conda                     23.5.2          py310h06a4308_0  
conda-content-trust       0.1.3           py310h06a4308_0  
conda-package-handling    2.0.2           py310h06a4308_0  
conda-package-streaming   0.7.0           py310h06a4308_0  
configurable-http-proxy   4.0.1                   node6_0  
cryptography              39.0.1          py310h9ce1e76_0  
debugpy                   1.5.1           py310h295c915_0  
decorator                 5.1.1              pyhd3eb1b0_0  
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
exceptiongroup            1.1.1              pyhd8ed1ab_0    conda-forge
executing                 0.8.3              pyhd3eb1b0_0  
flit-core                 3.9.0              pyhd8ed1ab_0    conda-forge
grpc-cpp                  1.48.2               h5bf31a4_0  
grpcio                    1.48.2          py310h5bf31a4_0  
icu                       58.2                 he6710b0_3  
idna                      3.4             py310h06a4308_0  
importlib-metadata        6.6.0              pyha770c72_0    conda-forge
importlib_resources       5.12.0             pyhd8ed1ab_0    conda-forge
ipykernel                 6.19.2          py310h2f386ee_0  
ipython                   8.12.0          py310h06a4308_0  
ipython_genutils          0.2.0              pyhd3eb1b0_1  
jedi                      0.18.1          py310h06a4308_1  
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
json5                     0.9.6              pyhd3eb1b0_0  
jsonpatch                 1.32               pyhd3eb1b0_0  
jsonpointer               2.1                pyhd3eb1b0_0  
jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
jupyter_client            7.3.4              pyhd8ed1ab_0    conda-forge
jupyter_core              5.3.0           py310hff52083_0    conda-forge
jupyter_server            1.13.5             pyhd3eb1b0_0  
jupyter_telemetry         0.1.0              pyhd8ed1ab_1    conda-forge
jupyterhub                3.1.1           py310h06a4308_0  
jupyterhub-dummyauthenticator 0.3.1                    pypi_0    pypi
jupyterhub-yarnspawner    0.4.0           py310hff52083_2    conda-forge
jupyterlab                3.3.2              pyhd3eb1b0_0  
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.10.3             pyhd3eb1b0_1  
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.4.2                h6a678d5_6  
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libprotobuf               3.20.3               he621ea3_0  
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libstdcxx-ng              11.2.0               h1234567_1  
libuuid                   1.41.5               h5eee18b_0  
libxml2                   2.10.3               hcbfbd50_0  
libxslt                   1.1.37               h2085143_0  
lxml                      4.9.2           py310h5eee18b_0  
mako                      1.2.4              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.1           py310h7f8727e_0  
matplotlib-inline         0.1.6           py310h06a4308_0  
mistune                   0.8.4           py310h7f8727e_1000  
nbclassic                 0.5.5           py310h06a4308_0  
nbclient                  0.8.0              pyhd8ed1ab_0    conda-forge
nbconvert                 6.5.4           py310h06a4308_0  
nbformat                  5.9.0              pyhd8ed1ab_0    conda-forge
ncurses                   6.4                  h6a678d5_0  
nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
nodejs                    6.13.1                        0    conda-forge
notebook                  6.5.4           py310h06a4308_0  
notebook-shim             0.2.2           py310h06a4308_0  
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
openssl                   1.1.1u               h7f8727e_0  
packaging                 23.0            py310h06a4308_0  
pamela                    1.0.0                      py_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd3eb1b0_0  
pexpect                   4.8.0              pyhd3eb1b0_3  
pickleshare               0.7.5           pyhd3eb1b0_1003  
pip                       23.0.1          py310h06a4308_0  
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
platformdirs              3.5.1              pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0           py310h06a4308_1  
prometheus_client         0.17.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.36          py310h06a4308_0  
protobuf                  3.20.3          py310h6a678d5_0  
psutil                    5.9.0           py310h5eee18b_0  
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd3eb1b0_0  
pycosat                   0.6.4           py310h5eee18b_0  
pycparser                 2.21               pyhd3eb1b0_0  
pygments                  2.15.1             pyhd8ed1ab_0    conda-forge
pyjwt                     2.7.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 23.0.0          py310h06a4308_0  
pyrsistent                0.18.0          py310h7f8727e_0  
pysocks                   1.7.1           py310h06a4308_0  
python                    3.10.10              h7a1cb2a_2  
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.17.1             pyhd8ed1ab_0    conda-forge
python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    2_cp310    conda-forge
pytz                      2022.7          py310h06a4308_0  
pyyaml                    6.0             py310h5764c6d_4    conda-forge
pyzmq                     25.0.2          py310h6a678d5_0  
re2                       2022.04.01           h27087fc_0    conda-forge
readline                  8.2                  h5eee18b_0  
requests                  2.28.1          py310h06a4308_1  
ruamel.yaml               0.17.21         py310h5eee18b_0  
ruamel.yaml.clib          0.2.6           py310h5eee18b_1  
send2trash                1.8.2              pyh41d4057_0    conda-forge
setuptools                65.6.3          py310h06a4308_0  
six                       1.16.0             pyhd3eb1b0_1  
skein                     0.8.1           py310hff52083_3    conda-forge
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
sqlalchemy                1.3.24          py310h5764c6d_1    conda-forge
sqlite                    3.41.1               h5eee18b_0  
stack_data                0.2.0              pyhd3eb1b0_0  
terminado                 0.17.1             pyh41d4057_0    conda-forge
tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h1ccaba5_0  
toolz                     0.12.0          py310h06a4308_0  
tornado                   6.1             py310h5764c6d_3    conda-forge
tqdm                      4.65.0          py310h2f386ee_0  
traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.6.3                hd8ed1ab_0    conda-forge
typing_extensions         4.6.3              pyha770c72_0    conda-forge
tzdata                    2023c                h04d1e81_0  
urllib3                   1.26.15         py310h06a4308_0  
wcwidth                   0.2.5              pyhd3eb1b0_0  
webencodings              0.5.1                      py_1    conda-forge
websocket-client          1.5.3              pyhd8ed1ab_0    conda-forge
wheel                     0.38.4          py310h06a4308_0  
xz                        5.2.10               h5eee18b_1  
yaml                      0.2.5                h7f98852_2    conda-forge
zeromq                    4.3.4                h9c3ff4c_1    conda-forge
zipp                      3.15.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h5eee18b_0  
zstandard                 0.19.0          py310h5eee18b_0

Configuration ```python [sorokaa@dl-test-02 miniconda]$ cat /etc/jupyterhub/jupyterhub_config.py c.JupyterHub.hub_ip = '' c.Application.log_level='DEBUG' c.JupyterHub.bind_url = 'http://:9090' c.JupyterHub.cookie_secret_file = '/etc/jupyterhub/jupyterhub_cookie_secret' c.JupyterHub.db_url = 'sqlite:////var/jupyterhub/jupyterhub.sqlite' c.JupyterHub.hub_bind_url = 'http://dl-test-02.cm.cluster:9091' c.JupyterHub.ssl_cert = '/etc/jupyterhub/cert-bundle.pem' c.JupyterHub.ssl_key = '/etc/jupyterhub/server.key' c.Application.logging_config = { 'handlers': { 'file': { 'class': 'logging.FileHandler', 'level': 'DEBUG', 'filename': '/var/log/jupyterhub/jupyterhub.log', } }, 'loggers': { '': { 'level': 'DEBUG', # NOTE: if you don't list the default "console" # handler here then it will be disabled 'handlers': ['console', 'file'], }, } } c.JupyterHub.authenticator_class = 'dummyauthenticator.DummyAuthenticator' # Optionally add a shared global password to be used by all users c.DummyAuthenticator.password = "test" c.JupyterHub.spawner_class = 'yarnspawner.YarnSpawner' #c.YarnSpawner.localize_files = { # 'environment': { # 'source': 'hdfs:///jupyterhub/example.tar.gz', # 'visibility': 'public' # } #} c.YarnSpawner.prologue = 'source /cm/shared/datalake/jupyterhub/miniconda/bin/activate' # The memory limit for a notebook instance. c.YarnSpawner.mem_limit = '2 G' # The cpu limit for a notebook instance c.YarnSpawner.cpu_limit = 1 c.YarnSpawner.debug = True c.JupyterHub.activity_resolution = 30 c.YarnSpawner.poll_interval = 30 c.YawnSpawner.args = ["--debug"] c.YarnSpawner.default_url = '/lab' ```
welcome[bot] commented 1 year ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

ajs6f commented 1 year ago

This appears to be the same issue dealt with in https://github.com/jupyterhub/yarnspawner/pull/23. I have not tried the fix there, and it's not obvious to me why it was never merged?

raibrahim commented 2 months ago

@ajs6f have you tested it with Jupyterhub 5.1.0?

I'm encountering a runtime warning and unable to access the provided URLs after the server starts. The warning and logs are as follows

[I 2024-09-16 14:32:17.212 YarnSingleUserLabApp manager:348] notebook_shim | extension was successfully linked. [W 2024-09-16 14:32:17.216 YarnSingleUserLabApp serverapp:2108] Customizing authentication via ServerApp.login_handler_class=<class 'jupyterhub.singleuser.mixins.make_singleuser_app..JupyterHubLoginHandler'> is deprecated in Jupyter Server 2.0. Use ServerApp.identity_provider_class. Falling back on legacy authentication. [I 2024-09-16 14:32:17.267 YarnSingleUserLabApp manager:368] notebook_shim | extension was successfully loaded. [I 2024-09-16 14:32:17.272 YarnSingleUserLabApp manager:368] jupyter_lsp | extension was successfully loaded. [I 2024-09-16 14:32:17.274 YarnSingleUserLabApp manager:368] jupyter_server_terminals | extension was successfully loaded. [I 2024-09-16 14:32:17.277 LabApp] JupyterLab extension loaded from /opt/jupyterhub/miniconda/envs/yarnpy/lib/python3.10/site-packages/jupyterlab [I 2024-09-16 14:32:17.277 LabApp] JupyterLab application directory is /opt/jupyterhub/miniconda/envs/yarnpy/share/jupyter/lab [I 2024-09-16 14:32:17.278 LabApp] Extension Manager is 'pypi'. [I 2024-09-16 14:32:17.312 YarnSingleUserLabApp manager:368] jupyterlab | extension was successfully loaded. [I 2024-09-16 14:32:17.321 YarnSingleUserLabApp manager:368] notebook | extension was successfully loaded. /opt/jupyterhub/miniconda/envs/yarnpy/lib/python3.10/site-packages/yarnspawner/jupyter_labhub.py:18: RuntimeWarning: coroutine 'HubAuth._api_request' was never awaited self.hub_auth._api_request(method='POST', RuntimeWarning: Enable tracemalloc to get the object allocation traceback [I 2024-09-16 14:32:17.340 YarnSingleUserLabApp mixins:636] Starting jupyterhub-singleuser server version 5.1.0 [I 2024-09-16 14:32:17.393 YarnSingleUserLabApp serverapp:3005] Serving notebooks from local directory: /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1726492361358_0007/container_1726492361358_0007_01_000001 [I 2024-09-16 14:32:17.393 YarnSingleUserLabApp serverapp:3005] Jupyter Server 2.14.2 is running at: [I 2024-09-16 14:32:17.393 YarnSingleUserLabApp serverapp:3005] http://node2:53601/user/hadoop/lab?token=2606dbaa48f2042c36a24f6ce344bc135b7bbfeec999be4f [I 2024-09-16 14:32:17.394 YarnSingleUserLabApp serverapp:3005] http://127.0.0.1:53601/user/hadoop/lab?token=2606dbaa48f2042c36a24f6ce344bc135b7bbfeec999be4f [I 2024-09-16 14:32:17.394 YarnSingleUserLabApp serverapp:3006] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 2024-09-16 14:32:17.406 YarnSingleUserLabApp serverapp:3068] To access the server, open this file in a browser: file:///.jupyter/jpserver-26397-open.html Or copy and paste one of these URLs: http://node2:53601/user/hadoop/lab?token=2606dbaa48f2042c36a24f6ce344bc135b7bbfeec999be4f http://127.0.0.1:53601/user/hadoop/lab?token=2606dbaa48f2042c36a24f6ce344bc135b7bbfeec999be4f

Even though the server appears to start and the URLs are generated, I am unable to access the JupyterLab interface via the provided URLs (e.g., http://node2:53601/user/hadoop/lab?token=...).

Afterward, the skein application running on Yarn is killed, and the user fails to spawn. The user cannot access their environment or run JupyterLab.

What could be causing this runtime warning, and could it be related to the access issue?

Any guidance would be greatly appreciated.

Thank you!

ajs6f commented 2 months ago

I'm sorry to say that we gave up on this mode of deployment, @raibrahim. We are moving on to using Kubernetes for resource management.