Closed ricky-lim closed 4 years ago
Thank you for getting in touch.
Please could you share more about your config, especially the docker image you are using, and I'll take a look.
I may have some of this already, but it could be good to get the latest anyway, and also useful for anyone else reading.
Yes sure. Here is with more detail specs
jupyterhub/k8s-hub:0.9.0
jupyterhub/singleuser:1.2
cdsdashboards==0.3.5
Below I received the error message, that the notebook file (*.ipynb
) is not found.
I checked into the dashboard-pod, and the notebook is located in /home/ril/gaussian_process_regression.ipynb
instead of in /home/jovyan/gaussian_process_regression.ipynb
I was wondering if there is away to set the home directory, other than /home/jovyan
, during the dashboard build step ?
I have tried to run the start.sh
at c.VariableKubeSpawner.default_presentation_cmd = ['start.sh', 'python3', '-m', 'jhsingle_native_proxy.main', '--allow-root']
, unfortunately I still received the same error
Here is my config:
hub:
allowNamedServers: true
extraConfig:
00-auth: |
# Custom authenticator to set NB_USER, NB_UID and NB_GID
c.JupyterHub.authenticator_class = RZAzureADAuthenticator
c.Authenticator.enable_auth_state = True
01-spawners: |
c.Spawner.cmd = ['start.sh', 'jupyterhub-singleuser', '--allow-root']
c.KubeSpawner.http_timeout = 600
c.KubeSpawner.start_timeout = 600
02-cds-handlers: |
from cdsdashboards.hubextension import cds_extra_handlers
c.JupyterHub.extra_handlers = cds_extra_handlers
c.VariableMixin.proxy_force_alive = True
c.VariableMixin.proxy_last_activity_interval = 600
c.VariableMixin.proxy_request_timeout = 600
04-cds-kube: |
c.JupyterHub.spawner_class = 'cdsdashboards.hubextension.spawners.variablekube.VariableKubeSpawner'
c.CDSDashboardsConfig.builder_class = 'cdsdashboards.builder.kubebuilder.KubeBuilder'
c.JupyterHub.redirect_to_server = False
c.JupyterHub.default_url = '/hub/dashboards'
c.Spawner.default_url = '/lab'
c.VariableKubeSpawner.default_presentation_cmd = ['start.sh', 'python3', '-m', 'jhsingle_native_proxy.main', '--allow-root']
singleuser:
uid: 0
storage:
type: dynamic
capacity: 10Gi
dynamic:
pvcNameTemplate: claim-{username}
volumeNameTemplate: volume-{username}
storageAccessModes: [ ReadWriteMany ]
extraVolumes:
- name: jupyterhub-workspace
persistentVolumeClaim:
claimName: jupyterhub-workspace
extraVolumeMounts:
- name: jupyterhub-workspace
mountPath: /home/jovyan/shared
homeMountPath: /home/{username}
extraEnv:
CHOWN_HOME: 'yes'
Below is the Error message
Error report from ContainDS Dashboards
Command Running:
python3 -m voila ./gaussian_process_regression.ipynb --port=54658 --no-browser --Voila.base_url=/user/ril/dash-gaussian-process-regression/ --Voila.server_url=/
Error output:
None
Standard output:
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/opt/conda/lib/python3.7/site-packages/voila/__main__.py", line 12, in <module>
main()
File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 663, in launch_instance
app.initialize(argv)
File "/opt/conda/lib/python3.7/site-packages/voila/app.py", line 340, in initialize
raise ValueError('argument is neither a file nor a directory: %r' % arg)
ValueError: argument is neither a file nor a directory: './gaussian_process_regression.ipynb'
Thanks you again in advance for your help :)
You've probably just omitted these lines from your config for brevity, but it doesn't quite make sense to be running:
because these don't have the ContainDS Dashboards components in them by default. (Or maybe you have another way to install them?)
Do you mean:
In particular, I don't believe your singleuser image contains the latest jhsingle_native_proxy 0.5.6 (because that passes through absolute paths, whereas the one in your 'error message' section above is relative). So the singleuser image definitely needs updating.
No images are specified in your config (although clearly cdsdashboards is installed at some level because you're seeing the cdsdashboards error page). So it would be great if you can just clarify that? Or are you building your own Docker images - if so, it would be really useful to see the Dockerfiles.
Mainly as background, what is the main reason for wanting to change the name from jovyan inside the container? (Assuming you mount /home/jovyan to the correct {username} on the volume, they should still see the correct files. But maybe you want to see correct usernames propagated into the central storage, or just make it more user-friendly within the container.)
Why have you mounted the extra shared folder at /home/jovyan/shared if you don't want jovyan to be the username? (Maybe extraVolumeMounts doesn't allow substitution of {username} - I haven't checked.)
Finally, are you seeing both the 'error message' web output ('Error report from ContainDS Dashboards') from your last post AND the log output from the very first post?
Sorry for all the questions! But if we can get you onto the latest versions, and fully understand your config, and then you're still seeing the errors I should be able to try it out for myself to see what's going on...
Thanks,
Dan
Hi Dan,
Correct, I built as my own docker images. I upgraded the jhsingle-native-proxy==0.5.6
Here is the dockerfile for singleuser
ARG TAG=1.2
ARG BASE_REPO=jupyterhub/singleuser
FROM $BASE_REPO:$TAG
# Add conda environment
USER root
COPY conda-env conda-env
RUN fix-permissions conda-env
USER $NB_UID
RUN conda env update --name base --file conda-env/base.yaml
# Set the voila default template
COPY jupyter_notebook_config_singleuser.py /etc/jupyter/
RUN cat /etc/jupyter/jupyter_notebook_config_singleuser.py >> /etc/jupyter/jupyter_notebook_config.py
RUN rm /etc/jupyter/jupyter_notebook_config_singleuser.py
# Set conda and pip environment
RUN conda init bash
COPY conda-env/.condarc /etc/conda/.condarc
COPY conda-env/conda-init.sh /etc/profile.d/conda-init.sh
# Fix permissions on /etc/jupyter as root
USER root
RUN fix-permissions /etc/jupyter/
Here is base.yaml
name: base
channels:
- conda-forge
- defaults
dependencies:
- python=3.7
- pip=20.1.1
- matplotlib=3.3.0
- voila=0.1.21
- bqplot=0.12.15
- vim>=8.2
- git>=2.28.0
- pip:
- ipyfetch==0.1.3
- voila-material==0.3.0
- pyprojroot==0.2.0
- dvc==1.4.0
- jhsingle-native-proxy==0.5.6
I needed to set the uid
and guid
as I mounted an existing NFS volume that requires a set of correct permissions.
However, the username is indeed is more of a convenience.
I also tried to keep using jovyan
as its user and get the /home/jovyan
instead of /home/ril
.
Unfortunately I still get the permission error while trying to serve a voila notebook (as shown below)
I think it's because of the unmatched uid
between jupyter-server and dashboard-server on /home/jovyan/.jupyter
INFO:tornado.application:b" with open(os.path.join(env['jupyter_config'], 'migrated'), 'w') as f:\n"
INFO:tornado.application:b"PermissionError: [Errno 13] Permission denied: '/home/jovyan/.jupyter/migrated'\n"
INFO:tornado.application:b'[Voila] Using contents: services/contents\n'
INFO:tornado.application:b'[Voila] Using contents: services/contents\n'
INFO:tornado.application:b'ERROR:tornado.access:500 GET / (127.0.0.1) 8.11ms\n'
With the updated jhsingle-native-proxy==0.5.6
, however I got the unexpected absolute path as
/home/jovyan/./gaussian_process_regression.ipynb
I was wondering, is there a way I could set the absolute path in the dashboard UI, such as /home/ril/gaussian_process_regression.ipynb
.
I think that will solve my issue.
Thanks again in advance for your time and support.
Standard output:
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/opt/conda/lib/python3.7/site-packages/voila/__main__.py", line 12, in <module>
main()
File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 663, in launch_instance
app.initialize(argv)
File "/opt/conda/lib/python3.7/site-packages/voila/app.py", line 340, in initialize
raise ValueError('argument is neither a file nor a directory: %r' % arg)
ValueError: argument is neither a file nor a directory: '/home/jovyan/./gaussian_process_regression.ipynb'
A little update.
If I execute the command with the expected home directory on /home/ril/gaussian_process_regression.ipynb
instead of /home/jovyan/gaussian_process_regression.ipynb
, on the pod directly. It works.
# Get into the pod
kubectl exec -it jupyter-ril-dash-2dgaussian-2dprocess-2dregression /bin/bash
# Voila command
python3 -m voila /home/ril/gaussian_process_regression.ipynb --port=36206 --no-browser --Voila.base_url=/user/ril/dash-gaussian-process-regression/ --Voila.server_url=/ --debug
I was wondering if there is a way in the configuration to set a dynamic home directory for the dashboard ? Another alternative, I think is to allow the dashboard developer to set an absolute path.
What is your advice, please?
Thanks again for your time. Look forward to hearing from you.
Thank you for working on this further!
Please could you let us know more about RZAzureADAuthenticator? That seems to be where NB_USER etc are set.
I don't believe 'jovyan' is hard-coded into the cdsdashboards or jhsingle-native-proxy code, but of course /home/jovyan is the default working folder for the Docker image you are using.
If you start a normal Jupyter notebook server, does that start up in /home/ril or /home/jovyan?
If this all works for standard Jupyter servers, we need to understand why CHOWN_HOME and NB_USER etc aren't being recognized by start.sh script when run through cdsdashboards.
If it doesn't work for standard Jupyter servers, then we should fix that first!
For the jupyter notebook server, the start.sh
seems to work and it does start up in /home/ril
.
Aha, for the authenticator, the NB_USER
is set during pre_spawn_start
hook
Below is the snippet of RZAzureADAuthenticator
class RZAzureADAuthenticator(AzureAdOAuthenticator):
@gen.coroutine
def pre_spawn_start(self, user, spawner):
auth_state = yield user.get_auth_state()
self.log.info('pre_spawn_start auth_state: %s' % auth_state)
self.log.info('pre_spawn_start user: %s' % user)
if not auth_state:
return
spawner.environment['NB_UID'] = auth_state['uid']
# This is set to 'ril'
spawner.environment['NB_USER'] = user.name
# On the config.yaml
auth:
type: custom
custom:
className: 'RZAzureADAuthenticator'
config:
client_id: xxxxx
client_secret: xxxxx
tenant_id: xxxxx
oauth_callback_url: https://jupyterhub.tst/hub/oauth_callback
state:
enabled: true
cryptoKey: xxxxx
admin:
users:
- ril
As I looked into the log from the hub during the creation of the dashboard, the pre_spawn_start
seems to work.
[I 2020-10-07 06:52:19.651 JupyterHub <string>:28] pre_spawn_start user: <User(ril 2/2 running)>
[I 2020-10-07 06:52:19.652 JupyterHub <string>:29] pre_spawn_start username: ril
Although there was an error prior to that.
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<DashboardBaseMixin.pipe_spawner_progress() done, defined at /usr/local/lib/python3.6/dist-packages/cdsdashboards/hubextension/base.py:129> exception=AttributeError("'NoneType' object has no attribute 'done'",)>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cdsdashboards/hubextension/base.py", line 136, in pipe_spawner_progress
if builder._build_future.done():
AttributeError: 'NoneType' object has no attribute 'done'
Here is the log from the dashboard-server:
Set username to: jovyan
usermod: no changes
Executing the command: python3 -m jhsingle_native_proxy.main --destport=0 python3 {-}m voila {presentation_path} {--}port={port} {--}no-browser {--}Voila.base_url={base_url}/ {--}Voila.server_url=/ --presentation-path=./hom
e/ril/gaussian_process_regression.ipynb --ip=0.0.0.0 --port=8888 {--}debug --debug --request-timeout=600 --last-activity-interval=600
sudo: setrlimit(RLIMIT_CORE): Operation not permitted
INFO:tornado.application:SuperviseAndProxyHandler http_get 51385
DEBUG:tornado.application:No user identified
Along with your pointer, it seems the start.sh
does not seem to run with cdsdashboard because of NB_USER
was not set properly.
Looking into the container's pod, it is run as root
which we would like to avoid and NB_USER
is jovyan
(base) root@jupyter-ril-dash-2dgaussian-2dprocess-2dregression:~# id
uid=0(root) gid=1000 groups=1000
(base) root@jupyter-ril-dash-2dgaussian-2dprocess-2dregression:~# echo $HOME
/home/jovyan
(base) root@jupyter-ril-dash-2dgaussian-2dprocess-2dregression:~# echo $NB_USER
jovyan
I was wondering, is there a specific hook for the dashboard spawner to ensure the NB_USER
is set from authentication ?
I think the problem is that the environment variables, such as NB_USER
, set during pre_spawn_start
does not seem to propagate to singleuser pod's container with cdsdashboards.hubextension.spawners.variablekube.VariableKubeSpawner
Thank you in advance for your kind guidance. I look forward to hearing from you.
@ricky-lim sorry for the delay here, but I think I've reproduced most of your setup.
I think it might make sense to add GRANT_SUDO: 'yes' in addition to CHOWN_HOME: 'yes'. I'm not too sure if you need the --allow-root bit...
In any case, cdsdashboards was causing you a problem. The pre_spawn_hook was being called, but then the environment dict was subsequently overwritten. Instead, I've checked in a change that attempts to merge it. This preserves the NB_USER in my tests, although I haven't thought too hard about any other side effects yet.
Please could you try it out by using this commit. If you're using my standard images, just try using the hub image from Docker Hub: ideonate/cdsdashboards-jupyter-k8s-hub:sha-d60c383
Hi Dan,
Thank you for your effort to reproduce the setup.
Yes, I have tried it and it works as expected, now
Are you going to tag the release for cdsdashaboard ?
Cheers
Great to hear that worked - thank you for letting me know.
I will make sure this functionality is brought into the next tagged release, but I'm not sure exactly when.
Hopefully you are OK to use the commit version instead for now - let me know if not for some reason.
Thanks,
Dan
This is now in version 0.4.0
Hi Dan,
In my setup, the user of the jupyter is set with her/his username coupled to the permissions from an active directory.
With each user has its own homedir, such as '/home/ril' instead of
home/jovyan
Unfortunately while trying to execute voila, I encountered the following error, which I think might be caused by trying to set username to jovyan.
I was wondering if you have encountered this issue before and I'll be very grateful if you could help, please.
Thank you in advance for your time and consideration.
The log with its error messages