search

gchq / stroom

Stroom is a highly scalable data storage, processing and analysis platform.
https://gchq.github.io/stroom-docs/
Apache License 2.0
433 stars 55 forks source link

Use an embedded zookeeper for cluster management #2070

Closed at055612 closed 1 year ago

at055612 commented 3 years ago

Tasks like leader election and cluster locking within the cluster would be made easier and more robust if we could use zookeeper to manage that cluster wide state. It seems it is possible to fully embed zk in stroom to avoid the requirement of a zk cluster as has been done at NiFi.

https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/nifi-state-management/content/embedded_zookeeper.html

It would make config slightly more tricky in that 3 or 5 nodes would need to have slightly different config than the others, however it should make stroom more resilient to node failure. For those envs where a dedicated zk cluster is available, we can make it configurable to use embedded or use an external one as NiFi has done.

at055612 commented 3 years ago

See here for integrating zk https://github.com/apache/nifi/blob/e2e137fced39f184e5998f7c4befe03e70f53016/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/state/server/ZooKeeperStateServer.java#L84

at055612 commented 3 years ago

See also https://sourcedelica.com/blog/2013/02/adventures-in-clustering-part-2/ and here for discussions on the limitations of using embedded zk http://zookeeper-user.578899.n2.nabble.com/Embedding-ZK-in-another-application-td4951714.html

at055612 commented 1 year ago

Closing this as it is likely we will adopt hazelcast