dkipping commented 5 years ago

Hello,

we tried to replace the hub image by our own extended image to install some more dependencies on the hub. We already tried creating the image in several ways:

The most simple one:

FROM jupyterhub/k8s-hub:7f6781e

RUN pip3 install packages ...
COPY our_files /path/to/files

Then we tried to incorporate the original USER switch and CMD from the k8s-hub image just to be sure we are running root and messing up the CMD execution.

... same as previous dockerfile ...

USER ${NB_USER}
CMD ["jupyterhub", "--config", "/srv/jupyterhub_config.py"]

We also tried to build the image from the original dockerfile (cloning the zero-to-jupyterhub-k8s repository) with both docker and chartpress.

But no matter what, we always end up with this error:

Traceback (most recent call last):
      File "/home/jovyan/.local/lib/python3.6/site-packages/jupyterhub/app.py", line 1955, in launch_instance_async
        await self.initialize(argv)
      File "/home/jovyan/.local/lib/python3.6/site-packages/jupyterhub/app.py", line 1639, in initialize
        self.load_config_file(self.config_file)
      File "<decorator-gen-5>", line 2, in load_config_file
      File "/home/jovyan/.local/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
        return method(app, *args, **kwargs)
      File "/home/jovyan/.local/lib/python3.6/site-packages/traitlets/config/application.py", line 598, in load_config_file
        raise_config_file_errors=self.raise_config_file_errors,
      File "/home/jovyan/.local/lib/python3.6/site-packages/traitlets/config/application.py", line 562, in _load_config_files
        config = loader.load_config()
      File "/home/jovyan/.local/lib/python3.6/site-packages/traitlets/config/loader.py", line 457, in load_config
        self._read_file_as_dict()
      File "/home/jovyan/.local/lib/python3.6/site-packages/traitlets/config/loader.py", line 489, in _read_file_as_dict
        py3compat.execfile(conf_filename, namespace)
      File "/home/jovyan/.local/lib/python3.6/site-packages/ipython_genutils/py3compat.py", line 198, in execfile
        exec(compiler(f.read(), fname, 'exec'), glob, loc)
      File "/srv/jupyterhub_config.py", line 333, in <module>
        raise ValueError("Unhandled auth type: %r" % auth_type)
    ValueError: Unhandled auth type: None

No config at /etc/jupyterhub/config/values.yaml
No config at /etc/jupyterhub/secret/values.yaml

Has somebody already replaced the hub image with another one and knows how to cope with this error? Do we need to generate the values.yaml files somehow or still copy them?

We are using helm chart v0.7.0, so not a developer release

dkipping commented 5 years ago

Was caused by missmatching versions of the hub image and the helm chart for zero-to-jupyterhub... So the tags have to be matching with the correct version number

yugushihuang commented 5 years ago

Hi dkipping,

I have encountered exactly the same situation as yours and exactly same log mess as yours but could not resolve it. Could you please explain in more details of how you resolve it? What is the version mismatch mean?

Thank you very much for your help in advance!

dkipping commented 5 years ago

Hi @yugushihuang , sure.

So we resolved this issue by making sure to use the correct version of the hub image.

You can find the sources of the hub image here: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/tree/master/images/hub

And the image on dockerhub here: https://hub.docker.com/r/jupyterhub/k8s-hub/

When using the image from dockerhub as a base image, you have to make sure that the version (the tag) matches the helm chart you are using.

So when using the helm chart 0.7.0 for your jupyterhub (zero-to-jupyterhub setup), you will have to use the matching tag 0.7.0.

So in the dockerfile:

FROM jupyterhub/k8s-hub:0.7.0

dkipping commented 5 years ago

Another tip: Do not set custom ENTRYPOINT or CMD in the Dockerfile if it is not absolutely necessary. For more info on why and which approach works better, have a look at this discussion: https://github.com/jupyterhub/kubespawner/issues/149#issuecomment-453202346

yugushihuang commented 5 years ago

Hi @dkipping,

Thank you so much! So if I use my-own-repo/my-own-image, I still need to make the tag as "0.7.0" right?

dkipping commented 5 years ago

No, how you call and tag your custom image does not matter. The only thing that matters is the version of the base image (the k8s-hub image) and of the helm chart.

So when you verified that, you can name and tag your image how you want to.

yugushihuang commented 5 years ago

Hi @dkipping,

Since I modify source code of jupyterhub, I rewrote the dockerfile in (https://github.com/jupyterhub/zero-to-jupyterhub-k8s/tree/master/images/hub) and build the custom image. But the error "Unhandled auth type: None" just keep coming out.

dkipping commented 5 years ago

Hi @yugushihuang,

So you Also still have the error message on the missing values.yaml?

Could show how your dockerfile and your config.yaml looks like? Of course without secrets and sensitive information

consideRatio commented 5 years ago

The configuration file provided with the docker image and utilized by jupyterhub, called jupyterhub_config.py, has evolved over time along with the chart. So make sure to utilize not only the same dockerfile but also let it import the matching config file (jupyterhub_config.py).

dkipping commented 5 years ago

@consideRatio True.

When just building upon the dockerfile by adding layers (RUN, ADD/COPY) that should however still work and not cause problems, unless you overwrite the jupyterhub_config.py, or am I missing something?

consideRatio commented 5 years ago

ah no then it remains and all is good! wrote from mobile and was not sure the custom image was built from scratch or not

yugushihuang commented 5 years ago

Hi @dkipping,

My docker file is almost same as https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/images/hub/Dockerfile. Just one line of difference. Instead of RUN pip3 install jupyterhub, Mine was "RUN pip3 install jupyterhub_new". I have modifed jupyterhub's source code a little bit for my own usage and upload to python repo. I do not think this matter, since my error is still the missing "values.html".

dkipping commented 5 years ago

Oh so you have essentially copied the Dockerfile and then changed one line... Well then my tip with the base image will not work, since your base image is Ubuntu then. (As in the original dockerfile).

You would then have to make sure that you are using the Dockerfile from a commit status, where its tag / version was 0.7.0 (searching through the history of the repository) as using the current Dockerfile means that the content might have changed since 0.7.0

Then also the tagging of your custom image might be important (tagging it as 0.7.0)

I do think that this approach with creating an own PyPi package is very risky, since it will be not only harder to maintain and update your custom image but this is also hiding away the actual changes and might break the whole usage of the hub wherever the original jupyterhub package is used/referenced.

If you have to alter source code and it are not too many files, I would probably suggest just building a custom Dockerfile based on the k8s-hub image by using

FROM julyterhub/k8s-hub:0.7.0 This already makes the original Dockerfile be part of yours. You can now think of your dockerfile just containing everything from the original image and what you write is appended to the end of the original image.

Then you don't have to copy the whole Dockerfile. And if now you modify files in the hub source code, you could overwrite them with a

COPY your-Adapted-File.py /path/to/Original-File.py

(Making sure that the source code is also the status as in the 0.7.0 version of the chart (that would be 0.9.2 for jupyterhub release I think)

Even though most changes should be working with configuring instead of rewriting the jupyterhub.

yugushihuang commented 5 years ago

Hi @dkipping.

Thank you so much for detailed explanation! I have gave it a try and just built a dockerfile with content like ==>

FROM jupyterhub/k8s-hub:0.7.0

ARG TEST=0.0.0

But the same error still persist.....

Now I am wondering is something run with the helm chart I modified?

I just modified line 43 of values.yaml as

name:

yugushihuang commented 5 years ago

Hi @dkipping, I think it is not related to how I modify the helm chart, because if I modified the line 43 back to name: jupyterhub/k8s-hub everything works fine... As long as I change it to a new docker image, even the image is just "From jupyterhub/k8s-hub:0.7.0" and nothing more... it fails.

If it is possible, can you share the successful running dockerfile you edit? Just help me get a sense of how should I edit..

Thank you so so much!

dkipping commented 5 years ago

So our Dockerfile looks like following:

FROM jupyterhub/k8s-hub:0.7.0

USER root
RUN ... installations
USER ${NB_USER}

RUN pip install some_packages
COPY a_file a_path/a_file

So nothing special at all.

I think it has to do with you modifying the values.yaml. Do you by values.yaml mean the config file, that you are using for installing / upgrading the helm chart?

I only do configuration in a config.yaml file and then run: helm upgrade jhub jupyterhub/jupyterhub --namespace jhub --values path/to/config.yaml to apply the changes. In the config.yaml file, I apply the changed image as follows:

...
hub:
  # using customized hub image to integrate some packages and files
  image:
    name: 'my.private.registry.com/custom-hub-image'
    tag: 'v1'
...

Since we are pulling from a private registry, we also need to add an imagepullsecret. But if you are hosting it publicly on dockerhub, then you do not need that.

yugushihuang commented 5 years ago

Hi @dkipping,

Thank you~!

After doing configuration in "config.yaml", the old error is gone! But I do have a new permission error....

dkipping commented 5 years ago

Hey @yugushihuang, Good to hear, at least the first part!

For us to be able to help you, please always provide the error messages and a bit more context like: what have you tried so far to resolve it or maybe provide your config.yaml and the dockerfile (you can leave out sensitive information and replace it with placeholders)

yugushihuang commented 5 years ago

Hi @dkipping,

My error is -->

[I 2019-01-18 19:04:09.785 JupyterHub app:1673] Using Authenticator: dummyauthenticator.dummyauthenticator.DummyAuthenticator [I 2019-01-18 19:04:09.785 JupyterHub app:1673] Using Spawner: kubespawner.spawner.KubeSpawner [I 2019-01-18 19:04:09.786 JupyterHub app:1016] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret [W 2019-01-18 19:04:09.806 JupyterHub app:1131] JupyterHub.hub_connect_port is deprecated as of 0.9. Use JupyterHub.hub_connect_url to fully specify the URL for connecting to the Hub. [W 2019-01-18 19:04:09.808 JupyterHub app:1173] No admin users, admin interface will be unavailable. [W 2019-01-18 19:04:09.808 JupyterHub app:1174] Add any administrative users to c.Authenticator.admin_users in config. [I 2019-01-18 19:04:09.808 JupyterHub app:1201] Not using whitelist. Any authenticated user will be allowed. [I 2019-01-18 19:04:09.854 JupyterHub app:1855] Hub API listening on http://0.0.0.0:8081/hub/ [I 2019-01-18 19:04:09.855 JupyterHub app:1857] Private Hub API connect url http://100.65.86.31:8081/hub/ [I 2019-01-18 19:04:09.855 JupyterHub app:1870] Not starting proxy [I 2019-01-18 19:04:09.855 JupyterHub app:1876] Starting managed service cull-idle [I 2019-01-18 19:04:09.856 JupyterHub service:302] Starting service 'cull-idle': ['/usr/local/bin/cull_idle_servers.py', '--timeout=3600', '--cull-every=600', '--concurrency=10', '--url=http://127.0.0.1:8081/hub/api'] [I 2019-01-18 19:04:09.858 JupyterHub service:114] Spawning /usr/local/bin/cull_idle_servers.py --timeout=3600 --cull-every=600 --concurrency=10 --url=http://127.0.0.1:8081/hub/api [E 2019-01-18 19:04:09.860 JupyterHub service:125] Permission denied trying to run '/usr/local/bin/cull_idle_servers.py'. Does have access to this file? [C 2019-01-18 19:04:09.861 JupyterHub app:1880] Failed to start service cull-idle Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1878, in start service.start() File "/usr/local/lib/python3.6/dist-packages/jupyterhub/services/service.py", line 335, in start self.spawner.start() File "/usr/local/lib/python3.6/dist-packages/jupyterhub/services/service.py", line 119, in start cwd=self.cwd or None, File "/usr/lib/python3.6/subprocess.py", line 709, in init restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1344, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) PermissionError: [Errno 13] Permission denied: '/usr/local/bin/cull_idle_servers.py'

My Docker file just -->

FROM ubuntu:18.04

RUN apt-get update && \ apt-get install -y --no-install-recommends \ git \ python3 \ python3-dev \ python3-pip \ python3-setuptools \ libcurl4-openssl-dev \ libssl-dev \ build-essential \ && \ apt-get purge && apt-get clean

ARG NB_USER=jovyan ARG NB_UID=1000 ARG HOME=/home/jovyan

ENV LANG C.UTF-8

RUN adduser --disabled-password \ --gecos "Default user" \ --uid ${NB_UID} \ --home ${HOME} \ --force-badname \ ${NB_USER}

ARG JUPYTERHUB_VERSION=0.9.*

ADD requirements.txt /tmp/requirements.txt RUN pip3 install --no-cache-dir \ jupyterhub==${JUPYTERHUB_VERSION} \ -r /tmp/requirements.txt

ADD jupyterhub_config.py /srv/jupyterhub_config.py

ADD z2jh.py /usr/local/lib/python3.6/dist-packages/z2jh.py ADD cull_idle_servers.py /usr/local/bin/cull_idle_servers.py

WORKDIR /srv/jupyterhub

So we can actually write a db file here

RUN chown ${NB_USER}:${NB_USER} /srv/jupyterhub

JupyterHub API port

EXPOSE 8081

USER ${NB_USER} CMD ["jupyterhub", "--config", "/srv/jupyterhub_config.py"]

I did not change the original one and just want to experiment and see if I can using the zero to k8s provided docker file to produce same results.

Thank you very much for your help!

yugushihuang commented 5 years ago

Hi @dkipping,

I get the idea! I should delete

USER ${NB_USER} CMD ["jupyterhub", "--config", "/srv/jupyterhub_config.py"]

And everything works fine!

Thank you!!!!!!!!!!!!!!!!!!!!!

saurav-bhagat commented 4 years ago

@yugushihuang Hey I'm also getting the same error: https://discourse.jupyter.org/t/permission-denied-starting-cull-idle-servers-py-in-hub-container/3676

How did you resolve this?

jupyterhub / zero-to-jupyterhub-k8s

Using custom hub image fails with missing configs #1015

FROM jupyterhub/k8s-hub:0.7.0

ARG TEST=0.0.0

name:

So we can actually write a db file here

JupyterHub API port