Htools primary/secondary instance placement for geographic diversity

(1) When a cluster has nodes in different geographical locations but is 
otherwise fully-connected, it is desirable to have a way to make the instance 
allocator place primary and secondary in different availability zones so that 
they are less likely to fail simultaneously.

The following proposal for achieving this is based on using node tags.

cluster tag:    htools:snodeextags:diverse

node 1,2,3 tag: site:FOO
node 4,5,6 tag: site:BAR

instance tag:   diverse:site

hbal/hail ensures that the primary and secondary are placed on nodes with
different values of the site:XXX node tag.

(Presumably this also implies that the primary and secondary must be placed
on nodes with *some* site:XXX tag, i.e. nodes with no such tag are excluded
from consideration)

***

If this area is being worked on, I have a couple of additional suggestions
to be considered.

(2) Extend the htools tags to limit placement of instances to nodes with
specified node tags.

node 1 tags:  cpu:amd, country:dk
node 2 tags:  cpu:intel, country:dk
node 3 tags:  cpu:amd, country:fr
node 4 tags:  cpu:intel, country:fr

instance tag:  node_tag:cpu:amd
instance tag:  pnode_tag:country:dk
instance tag:  snode_tag:country:fr

would ensure that both primary and secondary are placed on nodes with tag 
cpu:amd, the
primary node has tag country:dk, and the secondary node has tag country:fr.

(Should "node_tag", "pnode_tag", "snode_tag" be hard-coded prefixes, or should 
there be
cluster tags to enable this functionality?)

(3) Extend the htools instance exclusion functionality so that it can
prevent primary instances running on two nodes with the same value of some
node tag, rather than just the same node.

Use case: we have two instances providing DNS, and we want the primaries to
be placed in different availability zones.

Not sure exactly how that would work. Perhaps something like this:

cluster tag:    htools:nodeextags:diverse_service:site

node 1,2,3 tag: site:FOO
node 4,5,6 tag: site:BAR

instance tag:   diverse_service:dns

Two instances with the same diverse_service:YYY tag are not allowed to be
placed on primary nodes which have the same node tag site:XXX

(Again, presumably this implies that the primary nodes must have some value
for the site:XXX tag)

(What about secondary nodes?)

Original issue reported on code.google.com by bcandler...@googlemail.com on 16 Apr 2014 at 1:24

olopez32 / ganeti

Htools primary/secondary instance placement for geographic diversity #801