search

kontena / kontena-ipam

Kontena Docker IP Address Management Plugin
Apache License 2.0
5 stars 2 forks source link

Cleanup orphaned subnets on etcd conflicts/errors #43

Open SpComb opened 8 years ago

SpComb commented 8 years ago

The current AddressPools::Request logic first attempts to reserve a Subnet and then create a new AddressPool pool. This may lead to subnets reserved in etcd that are not actually in use in any pool if two nodes race on a request for the same pool.

Miscellaneous etcd errors within Subnet#reserve may also leave overlapping, orphaned subnets within etcd.

Scenario: Both nodes attempt to reserve the same subnet, the first one succeeds and the second one retries, and allocates a second subnet. One node will create the AddressPool, and the other node will get that AddressPool, leaving their subnet orphaned.

This should be some kind of idempotent after-the-fact fsck-style thing, because any errors during the etcd cleanup will also cause any such cleanup to fail.

SpComb commented 8 years ago

I don't know how likely this kind of scenario is. Some kind of busy-retry loop with multiple nodes could theoretically lead to a lot of conflicts with orphaned subnets?