search

banzaicloud / koperator

Oh no! Yet another Apache Kafka operator for Kubernetes
Apache License 2.0
784 stars 195 forks source link

launching Kafka components using StatefulSet insted of Deployments #744

Open ElyRonen opened 2 years ago

ElyRonen commented 2 years ago

Hello, My name is Ely, I am a customer success engineer at Statehub, we provide a global PersistentVolumes allowing stateful applications on Kubernetes to access their PVs across clusters (regions and clouds) to access

I’m currently creating a blog post about launching Kafka cluster using Koperator while leveraging our technology for a stateful Kafka cluster that will allow fast failover between one Kubernetes cluster to the other.

While testing your guide and launching a fully functional example Kafka cluster I noticed that all the components launched are using deployments and not StatefulSet, thus providing the PVC a name with a suffix that is auto-generated.

Is it possible to deploy the Kafka and Zookeeper pods using a statefulSet with PersistentVolumeClaims instead of a deployment? Providing a global name that will be recognized across clusters will allow us to launch a new Kafka cluster using the data and volumes from the previous cluster. in order for that to happen, we need zookeeper and Kafka volumes to have a unique and global ID and not be generated. I browsed through the repository but couldn’t find an example of a config that will use STS instead of deployment.

I’d appreciate your opinion, Thanks in advance, Ely

adamantal commented 2 years ago

Hi Ely,

Thanks for opening this issue! We are happy that you're using the operator.

Firstly, As opposed to other Kafka operators, Koperator runs the Kafka brokers as bare Kubernetes Pods instead of using Deployments or StatefulSets. Secondly, the Koperator takes into account any volume related constraint that you provide through the KafkaCluster CR configuration. More concretely you can set the properties of the PVC generated for each broker in brokerConfig.storageConfigs.pvcSpec. As an example please check simplekafkacluster.yaml with the default settings:

      storageConfigs:
        - mountPath: "/kafka-logs"
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10Gi

If you have any further questions, please let us know. We'd also be happy if you can send us the blog post, when you're finished with it.

Thanks, Adam

baluchicken commented 2 years ago

Hi @ElyRonen please look my answer here how can you create manually your own PV and PVC for the operator. https://github.com/banzaicloud/koperator/issues/572#issuecomment-805668287