search

weaveworks-experiments / flux-classic

Microservice routing
http://weaveworks.github.io/flux/
Apache License 2.0
82 stars 10 forks source link

Don't remove other hosts' containers #6

Closed squaremo closed 8 years ago

squaremo commented 8 years ago

The algorithm in the agent which tells amber about which containers are instances of which services does this:

  1. Look at each container running on this host, and record it as an instance of any service it matches;
  2. Look at each instance, and if it does not correspond to a container running on this host, remove it.

This has the effect, when running on more than one host, of removing all instances.

squaremo commented 8 years ago

This is only a problem when trying to reconcile the the source of truth for container lifecycle (docker) with the source of truth for services (etcd), when the agent starts. While running, we have enough information to add and remove containers accurately.

An obvious tactic would be to label each instance with a host identifier, and remove only those that belong to the host on which the agent runs. Another would be to use leases, and garbage collect.

squaremo commented 8 years ago

If the host crashes (or the docker daemon is stopped, for that matter), we'd want the instances to go away; this suggests a heartbeat mechanism.

As a stop-gap, associating each instance with a host identifier will be adequate. Assuming the host IP given to the agent identifies the host, we can use that.

squaremo commented 8 years ago

As a stop-gap, associating each instance with a host identifier will be adequate. Assuming the host IP given to the agent identifies the host, we can use that.

Done as described. Needs tests though.

squaremo commented 8 years ago

Test in eeb83683f3ec2214cd9aaa230ce1cfbdd076ecf6