search

fabric8io / fabric8

fabric8 is an open source microservices platform based on Docker, Kubernetes and Jenkins
http://fabric8.io/
1.76k stars 504 forks source link

Attaching a group of MQ brokers to different persistent journal files #3802

Open ghost opened 9 years ago

ghost commented 9 years ago

A particular challenge I have when using MQ is configuring a group of brokers to attach to different persistent journal files. I’ve opened this issue to capture my thoughts so far as I’m sure I won’t be the only person who hits this problem.

Imagine I want 2 instances of MQ fronted by a Kubernetes Service, such that consumers/producers are load balanced between the two brokers. Messages are persistent and i’m using kahadb. I can’t store the journal file in the container’s ephemeral file system because it must be long-lived. Therefore, I’d store them on a volume outside of the container (e.g. HostDir locally mounted to shared storage, glusterfs volume etc). So if using HostDir, broker1 would connect to /mnt/mq/journal1 and broker2 would connect to /mnt/mq/journal2 (here assuming a HostDir is mounted on all minions).

I can see a few challenges here:

At the moment I cannot do this using a single pod template with an RC of size of 2, because each Pod is given the same configuration. My workaround is to create a separate Pod template per broker instance, each with an RC of 1 and with a different ENV_VAR defining the journal file. This works but is pretty horrible and won't support auto scaling.

A brain dump of various options I've come up with:

  1. Use Nominal Services if/when they become available. I think this is most likely the best long term solutions as the Kubernetes ticket 260 talks about very similar issues, but it’s early days at the moment. I found this issue linked from this Kubernetes issue 1064, which precisely describes the journal re-attachment issue .
  2. Add a plugin/persistence solution in AMQ that would choose a ‘free’ journal file from a given directory of journal file directories (or create one). I guess this would work, but its limited to use of disk-based journals (as opposed to JDBC) and disk based locks.
  3. Manage assignment of brokers using an additional controller process e.g. monitor pods and set a unique broker file location via an ENV_VAR from a set of valid values.
  4. Same as above but have the controller talk directly to AMQ to assign a broker file over jolokia. No idea if this is possible.

Another challenge is auto scaling as journals may become orphaned with messages on scale down, but maybe this can be managed by the mq-controller? https://github.com/fabric8io/quickstarts/tree/master/apps/fabric8-mq-controller

gravis commented 9 years ago

I have the exact same issue with NSQ (another MQ system). I don't see other solution today than create a service/pod per instance...