CoreOS Support

[cookandy]

Hi,

I was wondering if you have ever successfully gotten PanteraS running on CoreOS. I have been looking at what it will take to move from Ubuntu to CoreOS. One difference I noticed already is that the Stable version of CoreOS uses Docker 1.8.3, whereas PanteraS uses 1.9.1. I think the container would certainly need to be downgraded to the same version of Docker. Would this cause any problems with Mesos, etc.?

Thanks

[kopax]

Just by curiosity, why did you choose CoreOS as guest OS ? I have heard that CoreOS wasn't really appreciated in the IT sectors and I still don't know why.

[cookandy]

Our primary reason for migrating to CoreOS is to support Kubernetes (for other projects). Our data center admins also like the upgrade strategies CoreOS uses. Our entire stack is containerized, so it kinda makes sense to use an OS built for containers. I get mixed feelings when I bring up CoreOS - some think it's ready, some don't. We'll be running on the stable channel for production, hence the older Docker version.

I have done some very basic testing with PanteraS and I think it will run well. Using machine metadata I am able to specify which machines should have the container running, and can manage the start/stop of PanteraS using fleet.

[sielaq]

Running docker container is agnostic from native system's distribution. So you can try to run it on CoreOS alba or beta release that supports 1.9 version. Docker 1.8.x had too many bugs that are not recommended for prod. https://coreos.com/releases/

[cookandy]

Thanks. Can you let me know what bugs you are referring to please? I'm not sure why CoreOS would use Docker 1.8.x in their "Stable" release, if it has bugs. We won't be able to use the "beta" or "alpha" channels in Prod I'm afraid.

[sielaq]

Current in my mind: I'm not sure why this is not closed yet but from 1.5 to 1.8.x we had orphaned bridge & network interfaces: https://github.com/docker/docker/issues/12239 every stopped/ killed container left its interface unclean, after few weeks of heavy deployment we ended with few thousands of interfaces (sic!) In the end (with so many interfaces) some system components slows down. That was not acceptable.

Moreover, one of the coolest feature of 1.9 is possibility to build with specific env - which was a supper needed especially when you build containers behind proxies. Before that we had a lot of nasty hacks.

[sielaq]

btw. you can always over-bind your docker version from native system to the container - so they will match each other. adding to docker-compose.yml sth like:

  volumes:
   - "/usr/local/bin/docker:/usr/local/bin/docker"

docker binary is just needed for mesos slave. mesos get the task and talks to docker like docker -H unix:///tmp/docker.sock pull <image>

If this solution is good for you I can make a ENV flag like LOCAL_DOCKER=true or NATIVE_DOCKER=true that will bind it for you

[kopax]

We have no testing protocole for that so I won't decide to change my guest OS now but I am really curious on how CoreOS as better perf for containerization than Debian OS.

btw: @sielaq will you add a new version soon with all the latest patch ?

[sielaq]

@kopax what kind of new version, what kind of patch ?

[kopax]

When you release the $LISTENING_IP I wasn't able to test it out. Since then, I saw that you corrected some dns issues. If it's fixed can you include them in a new version ? 0.1.9 was made a month ago

[sielaq]

I'm waiting with new release for marathon. I wanna include new consul 0.6.1, marathon 0.14 and mesos 0.26 since this is recommended version for this marathon.

[cookandy]

@sielaq, yes I think having LOCAL_DOCKER option would be very helpful.

[cookandy]

@kopax: I'll be sure to let you know how CoreOS works for us. I can already tell that managing PanteraS will be slightly easier using fleet. For example, I can create metadata on each server, and then ensure it's always running on every node by using a global deployment service

[Unit]
Description=PanteraS bootstrap
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
ExecStart=/opt/bin/docker-compose -f /home/core/PanteraS/docker-compose.yml up
ExecStop=/opt/bin/docker-compose -f /home/core/PanteraS/docker-compose.yml stop
ExecStop=/opt/bin/docker-compose -f /home/core/PanteraS/docker-compose.yml rm -f

[X-Fleet]
Global=true
MachineMetadata=PanteraS=true

This will start the PanteraS container on each host and ensure it stays running. There is some controversy around running docker compose on CoreOS, but that's the way I'm going for now.

[sielaq]

@cookandy sorry for so late, I got many tasks. I decided to use HOST_DOCKER=true it describe better what it does. if you have docker in a different path you will have to provide VOLUME_DOCKER by yourself.

[cookandy]

many thanks @sielaq. I reviewed the change and it looks like everything should work. I'll test and if there are any issues I'll let you know. Closing issue for now. Cheers.

[cookandy]

Hi @sielaq - I finally just got around to trying this fix, but I am having a problem with it. In CoreOS, docker is located at /bin/docker. So I've updated my docker-compose.yml to read:

  volumes:
    - "/etc/resolv.conf:/etc/resolv.conf.orig"
    - "/var/spool/marathon/artifacts/store:/var/spool/store"
    - "/var/run/docker.sock:/tmp/docker.sock"
    - "/var/lib/docker:/var/lib/docker"
    - "/sys:/sys"
    - "/tmp/mesos:/tmp/mesos"
    - "/bin/docker:/usr/local/bin/docker"

However, when the container starts, I see the following error form mesos:

mesos-slave stderr | Failed to create a containerizer: Could not create DockerContainerizer: Failed to create docker: Failed to get docker version: Failed to execute 'docker -H unix:///tmp/docker.sock --version': exited with status 127

When I get inside of the running container and try to run docker, I get the following error:

docker: error while loading shared libraries: libdevmapper.so.1.02: cannot open shared object file: No such file or directory

Any ideas?

[sielaq]

did you try to mount(bind) also this lib ?

[sielaq]

yea it should work /usr/lib/libdevmapper.so.1.02:/usr/lib/libdevmapper.so.1.02 https://groups.google.com/forum/#!topic/coreos-user/4nyJh8lYaiA

[cookandy]

Thanks @sielaq, I'll give it a try and let you know.

[sielaq]

any news?

[cookandy]

Hi, sorry for the delay. I actually just ended up swapping out the Docker version with 1.8.3 at the time. It looks like CoreOS stable just got support for Docker 1.9.1, so I'll just update the Dockerfile to use that version. :) BTW, CoreOS seems to be working well.