Hooks to run tasks after container launch / shutdown

[phildougherty]

Previously with the mesos-docker executor, we had made some modifications that stored data about containers and their IP:PORT pairs in redis once they had been launched, and cleaned up redis once they were stopped or killed. This was being used for routing via openresty+lua redis. Since the switch to deimos, we have been looking for a way to accomplish the same thing.

One option would be to put the logic into our docker images and have them register/de-register themselves. This would work fine for our internal images that we build ourselves, but would not work with pre-built images from the docker index.

The options I see are:

1. Write something that keeps redis in sync using data from the marathon API/events
2. Write something similar to confd that writes out openresty/nginx upstreams from marathon API or zookeeper data.
3. Modify deimos to allow running a script or task after successful container launch, or shutdown.

I'm basically just looking for advice and best practices, and want to see if you guys had any plans for easier integration with service discovery before making any decisions about which way to go.

[tarnfeld]

Interesting ideas. I mentioned this to @solidsnack a while back since it's something I was interested in using. The third option would get a +1 from me... or even just a set of pluggable modules that a user could chose between, rather than choosing a specific way of announcing.

I think this would work pretty well given the kill/rm invocation of deimos.

[solidsnack]

(3) has a lot going for it, in terms of flexibility. What arguments do you think the script should receive?

[phildougherty]

I was picturing something like cli flags --start-hook-path and --decommission-hook-path, and deimos would make the whole TaskInfo available to those scripts.

[sethdmoore]

So I've tried to implement this myself...

My original thought was that I could add arbitrary fields to the marathon task like so

{
   "container": {
    "image": "docker:///myregistry/runner",
    "options": ["-v", "/mnt/log/containers/:/mnt/log/containers/", "//", "start", "web"],
    "exec": {
        "prelaunch": ["/bin/sh", "-c", "/usr/bin/touch", "/var/notify"],
        "postdestroy": ["/bin/sh", "-c", "/usr/bin/touch", "/var/destroy"]
    }
  },
  "cmd": "",
  "env": {"NODE_ENV":"myenv", "STATSD":"my.graphite.host", "SLUG_URL":"http://example.com:8080/my.api/######.tar.gz"},
  "cpus": 0.16,
  "mem": 256.0,
  "instances": 1
}

And then parse the new fields in containerizer/docker.py

But unfortunately, the Marathon API strips arbitrary fields.

My next thought was that I could add it to the deimos config, but that's much less generic on a container basis. Something like:

[hooks]
prelaunch: ["/bin/sh", "-c", "/usr/bin/touch", "/var/notify"]
postdestroy: ["/bin/sh", "-c", "/usr/bin/touch", "/var/destroy"]

As far as script arguments, the easiest way to handle that in a generic way would be to use subprocess.Popen(command, env=hook_env), where hook_env could contain the docker container port, uuid, et cetera.

Then it's up to the user to do what they will with that information.

Any thoughts or ideas on my approach(es)?

[tarnfeld]

This is quite interesting. The reason you won't see any additional fields go through when defining things inside container is because that dictionary maps directly onto the ContainerInfo protobuf message, which is a static structure.

I'd vote for the second one, not only because the former would be difficult to implement, but the use case I had in mind would probably be something quite generic and a feature of the slave rather than a feature of the container itself. If you're doing something specific for a certain type of container when it launches and terminates, you're probably better off doing that elsewhere, perhaps inside the executor even. A good example of the former could be something like registering DNS entries automatically.