search

apache / incubator-heron

Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter
https://heron.apache.org/
Apache License 2.0
3.65k stars 598 forks source link

Add a initialization toggle to the Helm chart #3771

Closed nicknezis closed 2 years ago

nicknezis commented 2 years ago

There is an issue with the way the Bookkeeper cluster is initialized. It is currently only initialized when running the minikube platform mode. This causes issues for any of the other platform modes. Also this causes an issue whenever the Init container runs again, because it wipes out any underlying data. This is an issue in Kubernetes clusters that may restart frequently. For example K3D and KinD clusters.

The proposed fix is to provide an initialize setting which will be defaulted to false, but can be explicitly set during installation of the Helm chart. This is how Pulsar Helm chart handles Bookkeeper initialization. Here is the Pulsar documentation that mentions the initialize setting: https://pulsar.apache.org/docs/en/helm-deploy/#deploy-pulsar-cluster-using-helm

nicknezis commented 2 years ago

This is the example in the Pulsar helm chart. https://github.com/apache/pulsar-helm-chart/blob/192b3ca2ef80c1e78acd98f64a7f5fa64136ccf8/charts/pulsar/templates/bookkeeper-cluster-initialize.yaml#L19