Usability improvements to che-server container model

[wernight]

I propose a series of improvements to the che-server Docker container to make it easier to work with local Docker installations and orchestrators.

For example the launcher cannot be used with docker-compose (at least to have docker-compose ps correctly report running processes and such) and cannot be used with Kubernetes. Also I'm sure there are many other Docker runner which wouldn't work well with the current launcher.

What I'd suggest is something like:

• [x] Have che-server handle sign TERM to perform the clean shutdown (currently code in ~/che/bin/che.sh stop).
• [x] Reduce the number of mount required for persistence to /data (or whatever name) with storage and workspaces contained under it (and only those). Ideally move them to the root to have a simpler more constant mounting point.
• [x] #2598 If possible avoid requiring to mount lib-copy by copying the files another way.
• [x] Reduce required arguments (once #1560 is done it should actually be able to use default arguments, until then only --remote:<HOST_OF_DOCKER> is required and it could be a ENV variable for now)

That'd simplify startup, shutdown, reduce complexity, make it a lot easier to use on platforms like Kubernetes... to something like:

$ docker run -d --name docker --privileged -v $PWD/storage:/data/storage docker:dind
$ docker run --rm \
    -p 8080:8080 \
    -v $PWD:/data \
    --link docker \
    codenvy/che-server

I believe this is possible and nothing more would be required.

Note: I believe currently /home/user/che/storage must be mounted on the docker host under a specific path. I'd suggest making that path configurable via an environment variable.

[l0rd]

@wernight thanks for your analysis. I particulary apreciate some of the points that you have evoked. If you look at this document you will find out that we share the same vision.

However I believe that the launcher makes running Che an easier experience. This is a technique that Docker Inc itself use to install its Docker Trusted Registry. And you are right when you say that the launcher container is not intended to be used with Docker compose or Kubernetes. The launcher is a way to provide a basic "orchestrator" for users that don't want to install compose. Users that have Compose can still use it and run the che-server container directly.

[wernight]

Thanks. I've no objection to keeping the launcher if some people feel it's better. I'm mostly pointing out that it's largely not necessary given some improvements to che-server. In my example I'm not using docker-compose or Kubernetes; it just enables to use those. It could run on a single command as well.

All my points could be to basically refactor and improve the launcher/che-server and then people could use either simply. It'd mostly make che-server easier to run.

PS: That linked document seems to me better served as either an issue or a wiki page.

[TylerJewell]

Like with any product, there are many improvements that can be made. But in this case, the launcher is essential to the overall strategy. When we first put out Che earlier this year, at least 50% of all of our tickets were related to people who struggled to get Che to even run as a Tomcat server. We then packaged it as a che-server container, and that poses a large number of new challenges. Since we have put out the che-launcher and the new CLI, the new-user startup issues have nearly disappeared. The drop in actual issues is incredible, actually.

There are a number of reasons that the che-launcher exists:

1. It is not possible to have a single container for Che that provides truly universal syntax for all operating systems. We all know that it's Docker. But, unfortunately, given the limitations that I will try to itemize, the che-server does need to be configured with different che.properties, volume mounts, and environment variables depending upon the underlying OS & intended configuration of the product. The che-launcher is a container pattern that allows us to handle the OS-specific logic within the a terminating container, so that the right che-server configuration is launched each time. It doesn't remove the ability to run the che-server, but if you are going to run it, you then have to learn the weird nuances.
2. There are many issues with incompatibilities between different Docker daemon versions. And the version of the docker daemon is beyond our control. But we can control the Docker client that is used. We can point out a variety of issues with different Docker client versions working against different daemon versions. So it's easier to have a consistent docker client (1.6) in the che-launcher and then bake in the logic necessary for dealing with that particular client working on different daemon versions.
3. The che-lib/ thing is something that we cannot get away from without introducing other forms of complexity. The che-lib/ volume mount is there as a tactic to allow the che-server image to push files onto the host. When the che-server boots, the files inside the container are copied over to /lib-copy, which is mounted onto the host. This allows those files to appear on the host. You cannot just mount /lib directly because files get overwritten. And we need certain files on the host because the workspaces that we create need to volume mount those files themselves. This is how our agents and terminal get into the workspace. In a truly distributed system, we may end up setting a type of agent registry that allows newly activated workspaces to connect to the local registry to download these files instead of volume mounting them, but we expect this to be slower than volume mounting. Lots of people misttype this volume mount and it's a problem. So we hide that from them in the che-launcher.

The net result is that if you are new to Che, or just new to Docker, the che-launcher is more fool-proof and easier to approach, working on virtually any OS / Docker / Version combination. However, it does make pure container deployment more challenging.

Having said that, there are some reallly important improvements that can be made to the che-server: 1: First, every che.property can be settable as an environment variable. Even though we allow that in the CLI, that is just a hack. The CLI actually converts those environment variables into a temporary che.properties file that gets volume mounted into che-server. There is this pull request, where we are going to rename all of the che.properties (more user friendly) AND the che-server itself will accept any environment variable. This will eliminate a lot of stuff where you have to mount a custom che.properties file to configure the product.

2: The che.sh script inside the che-launcher is outdated. It was originally authored at a time when users were running Tomcat natively outside of a container. It still provides that purpose. It can probably be broken into two parts - a script for starting Tomcat natively, and another script for starting Tomcat inside of a Docker container. But if you look in that script, it has significant checks for docker and various utilities and it will fail if they are not configured properly. It could be simplified.

3: It is not recommended to run Che using privileged mode. I know you are doing that, but that introduces a whole pile of security issues. The longer term strategy is to abstract away the layer where Che server needs to interact with the Docker daemon into an SPI. Then there can be pluggable providers implemented for different deployment platforms. We (and RedHat) will build a provider for OpenShift, which means it should run on Kube as well. Swarm could be another provider. Rancher could be a provider. Once these various providers are there, then the che-launcher will need to have smarts to allow for using compose / docker / swarm / kubernetes activation + deployment to keep things consistent.

4: The other proposed changes are good changes to make. The system should be using volumes-from to register data & storage volumes. It will be cleaner that way. I particularly think the che-server handling the TERM signal is something we should have done a long time ago.

Since this is a broader community and the che-launcher is serving an important role here, you won't mind if we customize the title of this issue to something that is topically appropriate.

[wernight]

No issue with your comments, except that I'm not using privileged mode, or more accurately, not for Che, only to just docker:dind as it's required to run docker-in-docker.

For the che-lib, isn't possible to open an https port to serve those file and point right back at you (i.e. che-server) and then in the new started container do a wget or curl?

[l0rd]

@wernight in #2598 description we have provided a solution for che-lib. @sbryzak is currently working on that. What do you think about the solution?

I'm curious: why do you need docker:dind? As far as I know its usage is discouraged even by its author.

[wernight]

The discouraged is mostly for the original docker:dind. Now it uses by default --storage-driver=vfs to avoid data corruption (slower but safer).

There are two important reasons:

• Access to docker host (from Kubernetes it's possible but not trivial to access the host docker), making the --remote:<host> flag hard to set up. I think it's possible but didn't bother to find a solution.
• CRITICAL: Kubernetes spawns on multiple machines but you need to have Docker and Che on the same host for volume mounts to work. The only way to mount disks on both (without manually hacking) is to have docker daemon run on Kubernetes.

Updated the README to include all points.

[wernight]

Commented on that issue #2598

[JamesDrummond]

@l0rd dind can be useful in certain situations but prone to some issues. One example of troubles I had was getting it to use the host network instead on the container internal network.

I was actually thinking of using dind to introduce support git ssh compose build syntax for multiple users in Codenvy. However in most cases using the docker host daemon is a better option.

@wernight Would there be an issue for you to use the host docker daemon instead of os a seperate dind daemon? If not, take a look at how we use it to develop Che with Che. https://eclipse-che.readme.io/v5.0/docs/get-started-with-che-in-che . The trick is to export CHE_PROPERTY_machine_server_extra_volume=/var/run/docker.sock:/var/run/docker.sock so that all machines have access to the host docker.sock file. As you know this can cause vulnerabilities to the host machine but is fine in a trusted environment.

[wernight]

@JamesDrummond Yes it's an issue. For the reasons I listed. I'm not even talking here about having docker from within Che workspace, it's just to be able to run Che on Kubernetes that it's required.

For the second part, docker from within Che, it might also be required if one can get past #2771

[wernight]

I see you've questions about why docker:dind, but may be also keep them on https://github.com/wernight/kubernetes-che . I don't see it being relevant for this particular issue.

[TylerJewell]

@wernight - #2786 now handles the SIGTERM interception.

[TylerJewell]

@wernight - Could you check out the pull request? We have made a number of improvements and it now supports a single volume data mount. The syntax of how to use the new container is contained in the pull request and we are working up some docs. I have verified this on Windows, which is the trickiest platform as there are a couple special considerations that need to be made. But this should cover most of the original ideas that you had.

It will take some time for us to finalze this and then to push it live, as everyone who has built third party utilities around the old way will have them break when this goes live.

[TylerJewell]

Closing this issue as its old and superseded by a number of other improvements.