search

SNAS / openbmp

OpenBMP Server Collector
www.openbmp.org
Eclipse Public License 1.0
230 stars 76 forks source link

Multiple kafka clusters #58

Open DamianZaremba opened 6 years ago

DamianZaremba commented 6 years ago

Currently it's possible to specify multiple brokers for a single cluster.

In order to provider cluster level redundancy, it would be nice to be able to 'dual publish', so the exact same messages are sent 2 (or really N) times.

I've not looked deeply into the existing code to see how easy this would be, but would like to gather some feedback on if others have this use case and thus would a pull request be accepted for it.

I imagine something like;

kafka:
  brokers:
    cluster1:
      - xxxx
      - yyyy
    cluster2:
      - xxxx
      - yyyy

Where cluster1/cluster2 are just identifiers for logging/stats or such.

The main unknown is what the current behaviour in the cluster being down is; if it fails safe at publish time this is probably rather easier, if it assumes a hard dep then likely harder.

TimEvens commented 6 years ago

This would be good but it’ll increase resource usage a bit.

psikala commented 5 years ago

How about deploying two BMP -stations? 1. push to cluster one, and 2. to cluster two.

allamiro commented 3 years ago

I'm not sure about this but I think its more like environment specific and you may need to follow Kafka guidelines on how to setup cluster failover.

https://docs.confluent.io/5.5.0/multi-dc-deployments/replicator/replicator-failover.html

But definitely it would be good to have with the overall setup