search

kontena / kontena-registrator

Other
0 stars 0 forks source link

Do not remove shared etcd nodes #5

Closed SpComb closed 7 years ago

SpComb commented 7 years ago

Each Kontena::Registrator::Service tracks etcd nodes that are shared by multiple Docker containers, and is able to correctly maintain the shared etcd node as containers go up and down. However, if multiple kontena-registrator processes/policies register the same etcd node, then each Kontena::Etcd::Writer will currently remove the shared etcd nodes once there are no more local Docker containers for that Policy.

The Kontena::Etcd::Writer needs something to detect shared vs exclusive nodes, and only remove nodes from etcd that no other writer is refreshing.

It is probably better to not explicitly remove etcd nodes, and instead let them be expired by etcd if no other registrator is refreshing them.

SpComb commented 7 years ago

The Kontena::Etcd::Writer implementation at kontena-etcd@112a2b472 now uses etcd CAS operations to detect concurrent writes to the same etcd node, ensuring that each writer is writing the same value, and ensuring that each writer only deletes those etcd nodes that it has itself written, and that no other nodes are writing.

Shared nodes are tracked by comparing the etcd set operation result's prevNode; if this prevNode.modifiedIndex does not match the node returned by our most recent update/refresh op, then the node is shared with some other writer, and the node will not be removed from etcd. Once the last seen foreign prevNode.expiration timestamp is exceeded, the node is no longer shared, and can be removed when no longer updated.

SpComb commented 7 years ago

Closing as this seems to work reasonably well in general now.

See https://github.com/SpComb/kontena-etcd/issues/6 for some remaining implementation issues.