Proposal: Use Swarm place holders to supply target host and port to Docker run

[vbichov]

Overview

Some (many?) containers must know at docker run time, what is the hostname/port of the host they are running on.

Apps that explicitly expose their IP for other apps for example. Many clustered applications / DBs do this, as well as Consul-registrator, Consul-template etc.

example:

docker run -d --net=host --name registrator -v /var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator consul://$(hostname -i):8500

Problem

When using Swarm, one does not know the target hostname/IP because the target host is unknown by design.

The needed data cannot be provided and so, Swarm cannot be used to run such containers.

It may be tempting for one to view this as a "Docker" daemon issue: In a similar fashion as Swarm does, one may use a Docker client to connect to a Docker daemon and attempt to run such a container. One will not be able to use: docker run ... -e my_ip=$(hostname -i) ...

However, that is not the case. When a user or automated system connect to a remote docker daemon, they possess the information of what host is being targeted.

Not so when using Swarm. Swarm, and only Swarm holds the information of what the target host is, and Swarm is not in a sharing mood.

Solution

Add Swarm placeholders: swarm_target_ip, swarm_target_host.

When finding these in the swarm run command, Swarm will replace these with the needed data.

pros: simple, works. cons: ugly. alternatives: ???

code example for the fix: (works great)

In cluster/swarm/cluster.go

// CreateContainer aka schedule a brand new container into the cluster.
func (c *Cluster) CreateContainer(config *cluster.ContainerConfig, name string) (*cluster.Container, error) {
    c.scheduler.Lock()
    defer c.scheduler.Unlock()

    // Ensure the name is avaliable
    if cID := c.getIDFromName(name); cID != "" {
        return nil, fmt.Errorf("Conflict, The name %s is already assigned to %s. You have to delete (or rename) that container to be able to assign %s to a container again.", name, cID, name)
    }

    // Associate a Swarm ID to the container we are creating.
    config.SetSwarmID(c.generateUniqueID())

    n, err := c.scheduler.SelectNodeForContainer(c.listNodes(), config)
    if err != nil {
        return nil, err
    }

    TranslateSwarmVars(config, n)

    if nn, ok := c.engines[n.ID]; ok {
        container, err := nn.Create(config, name, true)
        if err != nil {
            return nil, err
        }

        st := &state.RequestedState{
            ID:     container.Id,
            Name:   name,
            Config: config,
        }
        return container, c.store.Add(container.Id, st)
    }

    return nil, nil
}

//Translate Swarm target host variables
func TranslateSwarmVars(config *cluster.ContainerConfig,  node *node.Node) {

    config.Hostname = strings.Replace(config.Hostname, "swarm_target_ip", node.IP, -1)
    config.Hostname = strings.Replace(config.Hostname, "swarm_target_host", node.Name, -1)

    for i := 0; i < len(config.Env); i++ {
        config.Env[i] = strings.Replace(config.Env[i], "swarm_target_ip", node.IP, -1)
        config.Env[i] = strings.Replace(config.Env[i], "swarm_target_host", node.Name, -1)
    }
    for i := 0; i < len(config.Cmd); i++ {
        config.Cmd[i] = strings.Replace(config.Cmd[i], "swarm_target_ip", node.IP, -1)
        config.Cmd[i] = strings.Replace(config.Cmd[i], "swarm_target_host", node.Name, -1)
    }
    for i := 0; i < len(config.Entrypoint); i++ {
        config.Entrypoint[i] = strings.Replace(config.Entrypoint[i], "swarm_target_ip", node.IP, -1)
        config.Entrypoint[i] = strings.Replace(config.Entrypoint[i], "swarm_target_host", node.Name, -1)
    }
}

[aluzzardi]

Thanks @vbichov for this.

I'm wondering if we can find a higher level solution - there are some ongoing efforts in Docker to provide service discovery (/cc @mavenugo @icecrime).

Perhaps, with service discovery, a container such as registrator could self-discover its network address.

[vbichov]

@aluzzardi I hope this will help. We've been using it, and it really made a difference for us. (may not be pretty, but it made the issue just go away.) I'm unaware of any Docker native service discovery efforts, I would be happy to hear more about that.

[ddgnani]

Thanks for posting this. Interesting and i'll follow here for docker native efforts.

[mavenugo]

@vbichov @ddgnani As @aluzzardi mentioned, we are adding the native service discovery support in docker engine via the new experimental networking features. Please refer to these docs/blog

https://github.com/docker/docker/blob/master/experimental/networking.md https://github.com/docker/docker/blob/master/experimental/compose_swarm_networking.md https://github.com/docker/docker/blob/master/experimental/network_overlay.md http://blog.docker.com/2015/07/docker-tutorial-13-experimental-networking/

Please note that this is still experimental (https://github.com/docker/docker/blob/master/experimental/README.md) and we are working towards making it part of engine release soon. If you have any feedback/input, please add them to https://github.com/docker/docker/issues/14083.

[vbichov]

@mavenugo Thank you for your post, interesting stuff. We are already working with Hashicorp's Consul and Registrator much to the same effect.

Forgive me for not commenting on the 14083 - that thread has become dreadfully long. Had I have, I would have commented that managing "/etc/hosts" files seems unwise. extremely so. Also, I would have asked about containers that have identical names. (if that question was asked and answered already, I haven't seen it)

@mavenugo , @aluzzardi I'm afraid to say that I don't see how these features would in any way resolve the issue this proposal was meant to address, could you please explain how this would work?

[pbecotte]

That solution could solve some of the SAME problems that something like Consul does- but it won't help RUN something like Consul, where you have to pass command line arguments to the process when you launch it.

[OferE]

+1. Very important feature.

my problem is like this: my containers runs a service - and need to expose it to the outside world by binding the ports of the service to the machine.

I want that each container will register itself to a service discovery. The consul discovery service requires that i will register my service to an agent.

since i don't want to install an agent on all my containers - i plan to have just one agent per host and all the containers on that host should register to it.

in order to do it - i need my containers to know the ip of the host.

since i am using swarm and not docker - i run the containers from a remote location. i don't know where swarm will decide to put my containers - so i need to get tjhe ip by using some api - like suggested in this issue.

Is there any way to solve this?

[dvenza]

Any news? With Swarm being able to restart containers on different hosts in case of failure, tracking their external ip:port address is even more important than ever.

[DerekTBrown]

+1

[bluefishforsale]

Running elasticsearch in unicast mode without host networking requires at least the --net-host flag with a value of an IP address that is reachable from other hosts participating in the cluster.

That value must be known at docker run, and this is a problem with launching elasticsearch clusters via swarm.

I also run other services which need to know their host name and IP, but the elasticsearch example is one I believe others can easily understand.

[jjn2009]

In my case I need my TLS certs to line up with a DNS name while talking to an application running outside of docker but on the same host machine. I have etcd running on the host machine with TLS to coordinate docker swarm, I want my containers to use this etcd cluster to manage configuration for the containers.

So inside swarm I can get the ip of the host machine like so ip route | awk '/default/ { print $3 }' which is fine for many purposes but when I talk to that IP I need to use the proper dns name for TLS to work. Like others have stated swarm doesn't seem to have a way to pass this information into the container.

Potential solution:

• create a unique DNS name for each swarm node which can be resolved by said swarm node
• ensure each swarm node's hostname is this DNS name
• run container with /etc/hostname mounted
• configure services to use this file to find the host which it is scheduled to and talk to its services which would be 'local' to that container

I think the only downside of this is that depending on the network setup you could potentially be routing your traffic in a way which negates any of the benefits of talking to the local machine in the first place.

if the container at runtime could map core-01.example.com to the result of ip route | awk '/default/ { print $3 }' this could be avoided but from what I understand messing with /etc/hosts in the container is bad idea, is this still the case with docker 1.10+?

[thutcheson-jha]

Yeah something like this is definitely needed. And while it's nice that the new combined swarm+engine stuff is going to include discovery features, I'd like to see swarm continue with the "batteries included by removable" principle.

I'm currently running consul agent as a container (rather than a full deamon process) on each swarm node, and registrator is connecting to that local instance. The consul agent container is running in host networking mode, so it is only available on one particular ip. I need to be able to wire containers managed by swarm to the local host ip so they can connect to the local consul agent. In other words, this is not a value that can be specified ahead-of-time in a compose file, as the actual ip will be dependent on each swarm node.

Right now I'm just using the master consul ip, which is fixed (so its value can be specified to docker compose). But that is brittle - if the master consul node is not available, the containers will fail to start.

[ChristianKniep]

I created an issue in docker to introduce environment variables into containers on a docker-engine level. https://github.com/docker/docker/issues/26234 IMHO it would be extending the idea proposed here, as one could introduce DOCKER_HOST. My usecase is to know the hostname a container is running on.

[ldshi]

One and half year, any update on this?

[bgou]

+1 For me we need to deploy some audit container that's running on all hosts (in a docker swarm), and we need to set the environment variable on each container to the node that's running that container. How do I do that?

[andr83]

+1 For now I can suggest dirty solution: mount file with required variables to container and export after on start. Start a service with something like:

--mount type=bind,source=/etc/variables,target=/etc/variables,readonly \

[abcdedf]

+1 It is a very needed feature for automation.

[loeveol]

+1 Very need for this function, I hope to be resolved as soon as possible.

[vitalcode]

A very needed feature

[prydin]

Aaaand... I just hit this issue as well. Any updates? Trying to write code that automatically and dynamically sets up services on Swarm and it's almost impossible without this functionality.

[ecoulon]

+1 Very important feature

[therealgambo]

+1

I'm surprised this feature doesn't exist yet, with the amount of services that now operate in a cluster environment, knowing what the host IP is crucial to automatically handle scaling, failover and replication at an application level.

[lupant]

+1 quote for missionrulz

[aleksandr-vin]

Trying to form Akka cluster with Consul (and ContructR as a coordinator), I need to know the FQDN or IP of the container started in swarm mode, at the start of the app, to register it in cluster. Can't do it using ENV.

[realcbb]

+1

[dongluochen]

Trying to form Akka cluster with Consul (and ContructR as a coordinator), I need to know the FQDN or IP of the container started in swarm mode, at the start of the app, to register it in cluster. Can't do it using ENV.

@aleksandr-vin Do you run your cluster with Docker Swarm mode? That is a different project https://github.com/docker/swarmkit/. The different is explained at https://github.com/docker/swarm#swarm-disambiguation.

[theindra]

+1 We have mongos running on the node itself and to connect to it, we need the public ip address of the node where the service is deployt.

[s-bernard]

+1

[kstobbel]

+1

[bvipparla]

+1

[liptga]

+1

[laugimethods]

To solve that (kind of) issue (needed for https://github.com/Logimethods/smart-meter/tree/multi-stage-build), I actually worked on my own solution: https://github.com/Logimethods/docker-eureka . I tried to keep it as lightweight as possible. Therefore I mostly used Bash Script to do so (+ a Python Server).

It offers those features:

• DEPENDS_ON (a dedicated port is open), WAIT_FOR (ping to answer)
• READY_WHEN & FAILED_WHEN (based on the logs)
• Access to the local containers of a Docker Service (through hosts entries + environment variables)

Take note that it is still a work in progress... Comments are welcome!

BTW, this project based on Docker multi-stage build, which is not yet available on Docker Hub (https://github.com/docker/hub-feedback/issues/1039)

[simingy]

+1

[elek]

+1

[benjaminstokes]

+1

[danipl]

+1

The community is becoming to need it critically, we're working with technologies (as Kafka, Hazelcast, and so on...) which need to advertise their host ips when are exposed to outside the SDN docker network.

[pprasse]

+1

[rms1000watt]

+1

[damonYuan]

+1

[IIIRADIII]

+1

[GiannisSialmas]

+1

[ap1459777123]

+1

[XUWeijiang]

+1

[junneyang]

+1

[junneyang]

+1 It is a very needed feature for automation.

[RaymondLeiLi]

+1

[RabihHallage]

+1

[huangmingyuan]

+1

[ssalvatori]

+1

[ssalvatori]

+1