How do you integrate pipework with restart and orchestration systems?

[deitch]

The question pretty much says it all. If I want to give a container an IP directly on the network (not Docker bridge), I pretty much run a script that does:

   # generate static IP, default route, which physical eth to use
   . /usr/local/bin/find_network_info.sh
   CID=$(docker run -d --net=none image)
   pipework $PHYS_IF $CID $CONT_IP@$CONT_GW

That works great. But anything that relies on native docker fails:

1. Restarts
2. Orchestration tools (Rancher, Kub, Fleet, Marathon, etc.)

... because all of the above use Docker directly, and I had to run 2 scripts - 1 to generate the correct interface, container IP, container GW; 1 pipework to attach them.

How do people set these environments up so they are properly managed with restarts and orchestration?

[jpetazzo]

One possibility is to use a combination of --net host, --privileged (or --cap-add), and --net container:.... It is not very complicated, but since I have (unfortunately) zero time to dedicate to pipework, I don't think I'll be able to make that happen :(

[deitch]

@jpetazzo I don't get it. --net host or --privileged just gives the container control over the environment (and messes it up a bit), so how... oh, wait. Do you mean if you give the container --net host or at least --cap-add NET_ADMIN that the container itself could add the macvlan interface as part of its startup?

If I understand how to do it, I would be happy to put in a pull request, but I need to understand your suggestion first.

[pierrecdn]

As jpetazzo said here, "you're wandering in uncharted territory!"

From my own experience and trying, there are some ways to do it, but they are still undocumented, not bundled "as-is" in the different tools, so far experimental here and there...

• At Docker level : you've to intercept the docker commands on the daemon socket
  • by default it's a unix-socket (unix:///var/run/docker.sock)
  • that's how dind, powerstrip, weave and other works.
  • there is some issues open to formally allowing to write pre/post hooks on docker events, see for example https://github.com/docker/docker/issues/6982
• At orchestrator level : basically your solution launch a docker run and do other things around. It's here you've to patch.
  • mesos : see containerizer/hooks/isolator. Require some mastery of mesos architecture and datamodel, C++ etc.
  • I didn't look in-depth the others.

For now, I understand that everyone has it's own constraints, but the "network plumbing in an orchestration context" subject is hot, so stay tuned !

I hope we will see docker, mesosphere, etc. industrializing this area soon, so many users like me will delete dirty patchs from unmaintainable public/private forks.

[deitch]

"wandering in uncharted territory!" I like that!

I did dig into powerstrip to see how they work, basically it is a proxy shim that allows creation of events. Obviously the plugin API will make that unnecessary, but as @jpetazzo said, it is still very fluid.

I have little desire to start ripping apart orchestrators, because then you need to do a lot of different things for each one. To be fair, though, most of them have some method of doing post-creation work.

I was thinking of (<= 1.7) using the events API (some orchestrators go down that path) from another container. Then you can look for a label, like com.customnet.ip=1.2.3.4 and on creation assign that IP. It is not that fancy or powerful, but would solve the fixed IP thing. And since all orchestrators support some form of tagging....

[dreamcat4]

Perhaps that's already solved. If you invoke pipework via my docker image. It works well in orchestration tools, and restarting / from bootup etc. Been around for ~6 months now

https://github.com/jpetazzo/pipework#cleanup

[deitch]

Hi @dreamcat4. Yes, that is the route I took. I had forgotten this issue was still around or I would have closed it. I used your plugin (with some pipework modifications, which I PRed back to here.