Retained messages being lost after pod restart even after using Persistent volume

[ManishJain-TomTom]

We're running a cluster with 2 VerneMq brokers. Everything works fine if we start one broker at one time, but as soon as we need to re-start both the brokers, all the retained messages are lost.

To fix the issue, we tried configuring persistent volume with vernemq, and we can see that the claim is bound to vernemq and volume is created. Even after this when we tested our scenario by restarting both the pods, we found that the retained messages were not synced. This is leading to data loss.

As per our understanding, all we need to do to use persistent storage with VerneMq is to set this value to true:

persistentVolume:
  ## If true, VerneMQ will create/use a Persistent Volume Claim
  ## If false, use local directory
  enabled: true

This should create the relevant persistent volume and map it correctly to our pods. When we run our cluster with this setting, we can see that Persistent volumes are created and mapped, but the messages are still being lost.

I've been trying for around 2 weeks now to find a solution, could be that I am missing something small. Will appreciate any help I can get.

Code in statefulset.yaml that creates and maps the volumes:

{{- if .Values.persistentVolume.enabled }}
  volumeClaimTemplates:
    - metadata:
        name: data
        annotations:
        {{- range $key, $value := .Values.persistentVolume.annotations }}
          {{ $key }}: {{ $value }}
        {{- end }}
      spec:
        accessModes:
        {{- range .Values.persistentVolume.accessModes }}
          - {{ . | quote }}
        {{- end }}
        resources:
          requests:
            storage: {{ .Values.persistentVolume.size }}
      {{- if .Values.persistentVolume.storageClass }}
      {{- if (eq "-" .Values.persistentVolume.storageClass) }}
        storageClassName: ""
      {{- else }}
        storageClassName: "{{ .Values.persistentVolume.storageClass }}"
      {{- end }}
      {{- end }}
{{- else }}
        - name: data
{{- end }}

Volume mapping:

• mountPath: /vernemq/data name: data

[ivanopassari]

Hi, do you have updates about this issue? I have the same behaviour.

[An0nymous0]

Same problem