Feature request: Provide .Node.Labels in service templating

[mback2k]

Hello everyone,

I would like to request the addition of the Docker node labels to service templating. Just like service labels can be accessed during templating, I would like to access the labels of the node the task container is running on.

That would allow a small amount of node specific behaviour and configuration to be passed into service tasks. What do you think?

[chrisbecke]

Sometimes docker swarm is used to deploy - in turn - clustered services, and given the lack of CSI support the best way to achieve these is to deploy to a fixed set of nodes, with local mounts. And being able to use labels as a confguration point for this would be very useful:

services:
  etcd:
    image: quay.io/etcd
    hostname: {{index .Node.Labels "etcd.host"}}
    deploy:
       replicas: global
       placement:
         constraints:
         - node.labels.etcd.host != null

Now if only there also was a way to test that a label was not set as a placement constraint.

[chrisbecke]

minio is another service let down by the current semantics.

This fragment is elegant. And wrong.

services:
  minio:
    image: minio/minio:latest
    command: server http://minio{1...4}/data
    hostname: minio{{.Task.Slot}}
    volumes:
      - /mnt/minio:/data
    deploy:
      placement:
        max_replicas_per_node: 1
        constraints:
          - node.labels.minio==true

It fails because eventually the tasks are restarted on their nodes in the wrong slot order, and minio doesn't like the resulting mismatch between the hostnames and disk ids.

[martin-marko]

I can't believe this is still not a thing. All I want is to be able to set an environment variable based on a node label's value

[tomalaci]

I would like to echo what @martin-marko has said. I was searching for this feature and how to use it but it appears it is still not added.

The main use-case has already been described by @chrisbecke such as setting up stateful services that require pinning specific tasks to specific nodes. Another example would be setting up Apache Zookeepers or many other clustered services that require setting up identifiers for sharding or replication.

I am also glad that @chrisbecke also has shown a subtle flaw by relying purely on task slot ids to control for which instance is what as it will likely create data corruption for many stateful services when the service is inevitably restarted and the task slots are distributed in a different way across different nodes. Related: https://github.com/moby/moby/issues/30770

Currently, for setting up any clustered, stateful service you need to create separate service definition for each shard and replica you intend to use in your clustered solution. This causes a lot of repeating configuration with the only change typically being a tiny identifier between them. Abstracting that away with Node labels would be a large quality-of-life improvement for maintaining stack deployment configs.

The only time you could safely rely on purely .Task.Slot IDs to manage things like etcd, Zookeeper and other stateful services is when your persistent storage volumes are non-local/remote (such as NFS). That way it won't matter if a task is shuffled between nodes.

P.S. This issue was also echoed in swarmkit part: https://github.com/docker/swarmkit/issues/2942