Use a proper init script from phusion

[giovannipizzi]

See why here: https://github.com/phusion/baseimage-docker#contents

For services already provided by phusion, check e.g. https://github.com/aiidateam/torquessh_base-docker/blob/17731c2452187adb03661683ba9e25b11b079946/Dockerfile#L36 (note that what one needs to do differs depending on the base image from Phusion).

This to provide more init scripts: https://github.com/aiidateam/torquessh_base-docker/blob/17731c2452187adb03661683ba9e25b11b079946/Dockerfile#L50

and CMD should always remain CMD ["/sbin/my_init"]

[ltalirz]

In particular, for postgres this means:

• initialize postgres DB directory in docker image (initdb)
• tell postgres system service to use this database

• in start-singleuser:

  • check whether 'template1' DB is already present

  • if not, create it

  • when making start-singleuser a system service as well, depend on postgres.service

[yakutovicha]

This is to summarize what I learned while trying to figure out how to use the standard my_init system provided by phusion.

1. According to the phusion baseimage README there are two ways to execute scripts at container startup: running them as daemons or place them into the my_init folder

2. What I observed: they both get launched by /sbin/my_init program. The scripts from my_init get executed first, the daemons get executed second.

3. PROBLEM1: This implies that one cannot start postgres as a daemon, because postgres should be started before AiiDA.

4. According to the phusion README the "proper" way to launch scripts and daemons at startup is to add CMD ["/sbin/my_init"] to the docker file. Note: there can be only one CMD in the docker file.

5. When launching the docker container DockerSpawner automatically adds two command line parameters: '--ip="0.0.0.0"', '--port=8888'. According to these lines the proper way to delete them would be to add to the following lines to the jupyterhub_config.py file:

   c.DockerSpawner.ip = '' # should be a string
   c.DockerSpawner.port = 0 # should be an integer

6. PROBLEM2: Adding the line c.DockerSpawner.port = 0 to the jupyterhub_config.py file causes jupyterhub fail to start:

   docker.errors.APIError: 500 Server Error: Internal Server Error ("driver failed programming external
   connectivity on endpoint jupyter-aliaksandr_2Eyakutovich_40epfl_2Ech
   (2568d4735ef1cd76e0200fdda04f7ce566e17a6c19537628fe8e4d2cb8e84b78):  (iptables failed:
   iptables --wait -t nat -A DOCKER -p tcp -d 127.0.0.1 --dport 32975 -j DNAT --to-destination
   172.17.0.2:0 ! -i docker0: iptables v1.6.0: Port `0' not valid

So as I conclusion I see two possible solutions:

1. If we want to keep using phusion, we should: 1.1. Place postgres launcher in my_init folder (not use them as daemons, PROBLEM1) and Fix the PROBLEM2 of the Docker spawner. or 1.2 Create our own CMD start script.

2. Use another base image. @ltalirz proposed to consider the ones provided by jupyterhub people:

   jupyter notebook docker containers:
   data science nb uses
   scipy notebook uses
   minimal notebook uses
   base notebook uses
   ubuntu 16.04

   @ltalirz, @giovannipizzi what do you think?

[yakutovicha]

In the meanwhile I made a temporary fix of PROBLEM2 on the dev AiiDA lab server to let the users to transfer their notebooks to aiida 1.0. I simply added args = [] right after this line

The postgres and start-singleuser.sh are now launched from /etc/my_init folder. The rabbitmq, aiidalab-jupyterhub-singleuser, and aiida are launched as services.

[yakutovicha]

I've created an issue https://github.com/jupyterhub/dockerspawner/issues/319 to discuss the possible solution(s) to the PROBLEM2

[yakutovicha]

Current status of things with respect to moving to Kubernetes.

1. The reason why Phusion image exists is that there is no lightweight init system that properly manages zombie tasks. The motivation is described here in more details. Recommended to read - quite an educational article.
2. According to the article above docker could have taken care of that, but probably they have higher priorities, so this hasn't yet happened.
3. The /sbin/my_init script (the lightweight init system) must be run as root. See this comment and this comment that explains in more details why running as root is unavoidable.
4. For some systems, though, this causes a big problem, like reported here. Using Heroku (another cloud platform) with Docker containers makes it impossible to use proper Phusion init system.
5. Currently, since we are going to move to Kubernetes, it is important to make sure that the container starts by root. I tried to follow this suggestion, but with no success. The problem is not yet solved. The CMD script is run by a non-root user.

Mentioning @giovannipizzi and @oschuett in case you have some ideas. @ltalirz is well aware of the problems.

[yakutovicha]

So we can close the story, finally.

1) I couldn't use /sbin/my_init script as it is, because JupyterHub docker spawner (as well as the Kubernetes docker spawner) gives some default parameters that I couldn't switch off, see https://github.com/jupyterhub/dockerspawner/issues/319. So the way I could overcome the problem is by creating my_my_init that has only calls /sbin/my_init and doesn't care about command line parameters.

2) The kubespawner issue (N5 in the previous comment) could be resolved by adding

c.KubeSpawner.uid = 0

to the kubernetes config file

[yakutovicha]

To keep in mind, that there is another init: https://github.com/krallin/tini