[bitnami/kafka] the persistance.existingClaim accepts strings(single value) how can I setup multiple pvc's for multiple kafka brokers

[Aadarsh-Verma]

Name and Version

bitnami/kafka 16.2.0

What steps will reproduce the bug?

Install latest bitnami/kafka chart with these custom values below.

persistance:
  existingClaim: kafka-pvc

Are you using any custom parameters or values?

persistance:
  existingClaim: kafka-pvc

What is the expected behavior?

The expected behavior should be that I should be able to create multiple brokers attaching them to multiple pvc's

What do you see instead?

When I create multiple brokers they get attached to the same pvc and only one broker runs and others shut down.

Additional information

I am using bitnami/kafka helm chart latest version. I am using local storage that does not support dynamic provisioning therefore I need to manually create my pv and pvc and use the existingClaim property for helm to use the created pvc but I can only specify a single pvc and thus cannot configure multiple pvc for running 2 or more kafka brokers. On doing so only a single broker starts and other brokers shut down. https://artifacthub.io/packages/helm/bitnami/kafka#uninstalling-the-chart Is there a way to specify multiple pvc's through the existingClaim property. This is the statefulset yaml file yaml file

[joancafom]

Hi @Aadarsh-Verma

I don't think you can provide multiple multiple PVC with the current configuration, but still, there is a simple workaround that comes to my mind.

I am using local storage that does not support dynamic provisioning therefore I need to manually create my pv and PVC

Okay, so as you are creating them manually and beforehand, let's say you want to run a Kafka cluster with 2 brokers (for instance). Let's also say you name your release theta. Then, create 2 PVCs (and their corresponding PVs) and name them data-theta-kafka-0 and data-theta-kafka-1 respectively:

$ kubectl get pvc
NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-theta-kafka-0       Bound    pvc-78da9569-bf94-4022-b55b-b309e2df499f   8Gi        RWO            standard       28m
data-theta-kafka-1       Bound    pvc-3ef32d2a-9ee5-4756-adf5-a1ad0852bb07   8Gi        RWO            standard       28m

If you install a new theta release of Kafka, I think K8s will try to use them instead of creating new ones:

$ helm install theta bitnami/kafka --set replicaCount=2
...
$ kubectl get pods
NAME                READY   STATUS    RESTARTS   AGE
theta-kafka-0       1/1     Running   2          12m
theta-kafka-1       1/1     Running   1          12m
theta-zookeeper-0   1/1     Running   0          12m

$ kubectl get pvc
NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-theta-kafka-0       Bound    pvc-78da9569-bf94-4022-b55b-b309e2df499f   8Gi        RWO            standard       31m
data-theta-kafka-1       Bound    pvc-3ef32d2a-9ee5-4756-adf5-a1ad0852bb07   8Gi        RWO            standard       31m
data-theta-zookeeper-0   Bound    pvc-f2a02d46-381b-4c4a-b459-0e5a9a6a9c54   8Gi        RWO            standard       12m

Notice the different timestamps in the AGE field 🙂 Could you give it a try?

[Aadarsh-Verma]

Thanks worked like a charm

[prionkor]

@Aadarsh-Verma could you elaborate this a bit? What should I put in the existing claim field in the values file? I understand the name convention you said here.

[crazykingit]

@prionkor , I have tested this on the mysql bitnami chart and this works for me.

There are 2 choices. 1st is to download and use a dynamic provisioner. Unfortunately, I am not on a network that has access to the registry, so I did the following instead as a workaround.

The persistent volume claims from bitnami are created using the following naming convection. "data-" followed by the service name, followed by "-replica id". For example,

data-mariadb-galera-0
data-mariadb-galera-1
data-mariadb-galera-2

These claims are being created and applied automatically when you use the replicaCount in the values.yaml. I just created the persistent volumes manually to allow these automatic persistent volume claims to work.

here is my PV for the 1st replica:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv-0
spec:
  claimRef:
    name: data-mariadb-galera-0
    namespace: mysql-database
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /nfs/mysql/0
  storageClassName: nfs-mariadb

There is a line in the values.yaml to disable the use of dynamic provisioning. When I did this, I have to define a storage class and use it in the persistent volume yaml above. I also left the existingClaim field unmodified.

values.yaml:

...
existingClaim: ""
...
## @param persistence.storageClass ...
## If set to "-", storageClassName: "", which disables dynamic provisioning
...

storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-mariadb
provisioner: kubernetes.io/hostPath
reclaimPolicy: Retain
volumeBindingMode: Immediate