Split DB + Rabbit and API processes

[mkoderer]

We need to have dedicated DB+Rabbit nodes. Please ensure also the migration for a mixed setup to a dedicated one.

[tpatzig]

failover should still work for DB + rabbit. ideally they do not run on the same node.

[vuntz]

As discussed, the idea would be to split the 5-node cluster in a 2-node part for rabbitmq and database, and a 3-node part for the API services. This is not the usual setup we want, just some way to make this work here, for this existing cloud.

So some location constraints would do the magic here, and I understand it's acceptable to have them added manually.

cc @aspiers

[aspiers]

@vuntz I'm not yet convinced that's a good idea - what if the DB and MQ end up on the smaller side of a network split? Admittedly it's hard to think of a failure case which would cause that. More likely (but still hopefully unlikely) is that the two DB/MQ nodes both fail and then Pacemaker doesn't fail over those services due to the constraints. The normal advantage of a 5-node cluster is that it can tolerate up to 2 node failures, but this would be preventing that ... unless we make the location constraints finite.

[mkoderer]

Let's discuss this internally and open it up if needed