search

jupyter-on-openshift / jupyterhub-quickstart

OpenShift compatible version of the JupyterHub application.
Apache License 2.0
101 stars 107 forks source link

How can I get the image s2i-minimal-notebook and jupyterhub #9

Closed IssacPan closed 6 years ago

IssacPan commented 6 years ago

I will deploy the project jupyter-on-openshift at Private Clouds.It means I can't connect to Internet. I have tried modify the "source:git" in "https://raw.githubusercontent.com/jupyter-on-openshift/jupyterhub-quickstart/master/images.json" to "source:contextDir" ,but I found in the project which has more action needing Internet which like "pip install"etc. Also I have tried using the "manage.openshift.com" to deploy the project ,but when I confirm the plan for init,there is an error "There was an error processing your request for access: reCAPTCHA verification failed, please try again." for hours. May I get the 2 images directly? If you can tell me how can I deploy the project at private clouds is much better. my email: issacpan@qq.com thank you so much

GrahamDumpleton commented 6 years ago

Bit confused what you are asking for. If you are using an OpenShift environment, the first command in the README for the repo says to do:

oc create -f https://raw.githubusercontent.com/jupyter-on-openshift/jupyter-notebooks/master/images.json

This will build the s2i-minimal-notebook image in your OpenShift project.

Similarly, to build the JupyterHub image, as the README says, you run:

oc create -f https://raw.githubusercontent.com/jupyter-on-openshift/jupyterhub-quickstart/master/images.json

The images aren't hosted on DockerHub and need to built.

If the issue is that your OpenShift environment has zero access to the Internet and so cannot pull down repositories from GitHub so as to do the build, then it would be necessary to somehow get a copy of the two GitHub repositories to a machine inside your network that can access your OpenShift environment. I would then need to give you manual instructions for setting up a binary input build to build the images from your local copy of the GitHub repositories.

IssacPan commented 6 years ago

Hi Graham Thanks for your reply. There are more details :we design to deploy a system for finanical analysis with Jupyter Notebook.And I think the K8S+JupyterHub is the best Plan.Now I am doing some pre-work for technical feasibility. Our company has a private clouds environment with OpenShift.But the environment is forbidden to connect to Internet for security reason. At first, I do my work with this guide:https://zero-to-jupyterhub.readthedocs.io/en/latest/setup-jupyterhub.html. I solved some problems，but the project need too much right ,which like "privileged:Ture".We can't grant the project so many rights in our environment. Fortunately，I see the doc in Jupyter.org updated，and find your solution.I think your solution is just meet our requirement.But our environment can't connect to Internet,so I can't use the "git","pip" etc.I tried to deploy a OpenShift cluster in my personal Macbook,but I met some other problem like"Unable to mount volumes for pod "jupyterhub-2-build_myproject(6d792d72-446c-11e8-b03c-025000000001)": timeout expired waiting for volumes to attach/mount for pod "myproject"/"jupyterhub-2-build". list of unattached/unmounted volumes=[builder-dockercfg-2l22k-push builder-token-c5442] ".Also I tried the manage.openshift.com,but when I comfirm the "OpenShift Online Starter Plan",I always see the error"There was an error processing your request for access: reCAPTCHA verification failed, please try again." It's so sorry to trouble you.Looking forward to your reply.

------------------ 原始邮件 ------------------ 发件人: "Graham Dumpleton"notifications@github.com; 发送时间: 2018年4月20日(星期五) 下午5:33 收件人: "jupyter-on-openshift/jupyterhub-quickstart"jupyterhub-quickstart@noreply.github.com; 抄送: "Issac"3672556@qq.com; "Author"author@noreply.github.com; 主题: Re: [jupyter-on-openshift/jupyterhub-quickstart] How can I get theimage s2i-minimal-notebook and jupyterhub (#9)

Bit confused what you are asking for. If you are using an OpenShift environment, the first command in the README for the repo says to do: oc create -f https://raw.githubusercontent.com/jupyter-on-openshift/jupyter-notebooks/master/images.json
This will build the s2i-minimal-notebook image in your OpenShift project.

Similarly, to build the JupyterHub image, as the README says, you run: oc create -f https://raw.githubusercontent.com/jupyter-on-openshift/jupyterhub-quickstart/master/images.json
The images aren't hosted on DockerHub and need to built.

If the issue is that your OpenShift environment has zero access to the Internet and so cannot pull down repositories from GitHub so as to do the build, then it would be necessary to somehow get a copy of the two GitHub repositories to a machine inside your network that can access your OpenShift environment. I would then need to give you manual instructions for setting up a binary input build to build the images from your local copy of the GitHub repositories.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

GrahamDumpleton commented 6 years ago

This is the best I can do right now as am travelling.

These steps still assume you are connected to the internet as I don't know what steps you need to take to be able to get remote images into your cluster, or how to download template files so you can load them into your cluster. You will need to work out what you need to do accordingly. Ask if you are not sure.

First up, if you have no access to do S2I builds with packages installed from PyPi, possibly easier to use the Jupyter Project images for Jupyter notebooks. These are:

You can find details on these at:

Work out which would have the packages you need and get that image down and pushed into your cluster with an image stream. Just be aware that these images are really big. The limit on the size of images and the container file system for pods in the OpenShift cluster will need to be anywhere between 3Gi and 8Gi depending on the image used. OpenShift Online is limited to 3Gi and so only the minimal-notebook can be used with it.

With internet access, you would normally import the image into the cluster, and create an image stream, by running the command:

oc import-image jupyter/minimal-notebook:latest --confirm

This will create an image stream in the project called minimal-notebook.

If you want to deploy just a Jupyter Notebook instance, and you don't need JupyterHub, you can use the template at:

Details on this at:

Normally this would be loaded into the OpenShift environment using:

With it loaded, you should be able to use:

oc new-app --template jupyter-notebook --param NOTEBOOK_IMAGE=minimal-notebook:latest

You are overriding NOTEBOOK_IMAGE template parameter so it uses the image you manually imported and which is referenced by the image stream minimal-notebook, rather than the image from Docker Hub.

If you need to use JupyterHub, I have created an image build on Docker Hub. To import this image, you would use:

oc import-image jupyteronopenshift/jupyterhub-quickstart:latest --confirm

This will create an image stream called jupyterhub-quickstart.

Next you need the JupyterHub template. This is from:

The template would normally be loaded using:

oc create -f https://raw.githubusercontent.com/jupyter-on-openshift/jupyterhub-quickstart/master/templates.json

With this template, you can then deploy JupyterHub using:

oc new-app --template jupyterhub-deployer --param JUPYTERHUB_IMAGE=jupyterhub-quickstart:latest --param NOTEBOOK_IMAGE=minimal-notebook:latest --param JUPYTERHUB_CONFIG='c.KubeSpawner.singleuser_supplemental_gids=[100]'

You override JUPYTERHUB_IMAGE to say to use the jupyterhub-quickstart image you imported. You override NOTEBOOK_IMAGE to use the Jupyter Project image you imported. You need to override JUPYTERHUB_CONFIG as the Jupyter Project images need to have additional supplemental group of gid 100 added.

With these steps above, so long as you can get those images and templates loaded into your project, you should be good to go. You will then need to customise the JUPYTERHUB_CONFIG further as necessary. This can be done after the fact by edit the config map for it and triggering a new deployment.

Hopefully this address your problems with lack of access for S2I builds to work.

IssacPan commented 6 years ago

thanks a lot.Your suggestion is very useful,I think it will solve my problem. I will try it!

IssacPan commented 6 years ago

I think I am already close to success，but there are still some problems. As your suggestion，I did the following work: 1、docker pull jupyter/minimal-notebook:latest docker pull jupyteronopenshift/jupyterhub-quickstart:latest docker pull postgresql (I did't find postgresql:9.5 on hub.openshift.com or hub.docker.com，can you give me a Official download address？) 2、 save the three images to tar files and copy them to private cloud environment with USB. 3、download https://raw.githubusercontent.com/jupyter-on-openshift/jupyterhub-quickstart/master/templates.json ，and copy it to private cloud environment with USB 4、In the private cloud enviroment ,I did: a、create a new project A b、load the three tar files with "docker load xx.tar xxx" c、oc create -f templates.json d、load the three images(hub/notebook/postgresql) as imagestream in the new project A with Web controller，and the pods works e、oc new-app --template jupyterhub-deployer --param JUPYTERHUB_IMAGE=jupyterhub-quickstart:latest --param NOTEBOOK_IMAGE=minimal-notebook:latest --param JUPYTERHUB_CONFIG='c.KubeSpawner.singleuser_supplemental_gids=[100]' then the resources are created，and the application created success. f. but there is still some problem in the pod "jupyterhub-db" the Health monitoring log is:

Time Severity Reason Message 5:35:35 PM Warning Unhealthy Readiness probe failed: sh: cannot set terminal process group (-1): Inappropriate ioctl for device sh: no job control in this shell psql: FATAL: role "jupyterhub" does not exist 32 times in the last 6 minutes 5:30:27 PM Normal Started Started container 5:30:27 PM Normal Created Created container 5:30:26 PM Normal Pulled Container image "registry.openshift.xxx.cn/postgresql@sha256:617953c9279b09834ec0ed9340fc76f4e2cd8caa056b983cadebf3e5fb5a5c0d" already present on machine 5:30:25 PM Normal Successful Mount Volume MountVolume.SetUp succeeded for volume "pvc-0376c451-46d7-11e8-815c-0050569d7274" 5:30:25 PM Normal Successful Mount Volume MountVolume.SetUp succeeded for volume "default-token-7nqkr" 5:30:25 PM Normal Successful Mount Volume MountVolume.SetUp succeeded for volume "config"

     the pod events are:

/usr/lib/python2.7/site-packages/supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security. 'Supervisord is running as root and it is searching ' 2018-04-23 09:30:27,878 CRIT Supervisor running as root (no user in config file) 2018-04-23 09:30:27,934 INFO RPC interface 'supervisor' initialized 2018-04-23 09:30:27,934 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2018-04-23 09:30:27,934 INFO supervisord started with pid 7 2018-04-23 09:30:28,936 INFO spawned: 'postgresql' with pid 10 2018-04-23 09:30:29,044 INFO exited: postgresql (exit status 0; not expected) 2018-04-23 09:30:30,046 INFO spawned: 'postgresql' with pid 20 2018-04-23 09:30:30,062 INFO exited: postgresql (exit status 0; not expected) 2018-04-23 09:30:32,065 INFO spawned: 'postgresql' with pid 30 2018-04-23 09:30:32,080 INFO exited: postgresql (exit status 0; not expected) 2018-04-23 09:30:35,084 INFO spawned: 'postgresql' with pid 33 2018-04-23 09:30:35,100 INFO exited: postgresql (exit status 0; not expected) 2018-04-23 09:30:36,101 INFO gave up: postgresql entered FATAL state, too many start retries too quickly

 Is there something wrong with the image postgresql? what is the "role "jupyterhub" does not exist"?
IssacPan commented 6 years ago

Solve the ""role "jupyterhub" does not exist"" with CREATE USER jupyterhub CREATE DATABASE jupyterhub in the image PSQL

GrahamDumpleton commented 6 years ago

Hmmm, I must have missed that previous message. Looks a bit like it was trying to reuse a persistent volume which had a database already created in it in a way which was setup as required. If the persistent volume was empty, it would have created the database first time as was required. Anyway, looks like you solved it now.