All Graylog Nodes Are Master in AWS EKS

Deployed helm chart tried 1.17.2, all Graylog nodes are master nodes. The pod label graylog-role label is not set.

Version of Helm and Kubernetes:

$ helm version
version.BuildInfo{Version:"v3.6.3", GitCommit:"d506314abfb5d21419df8c7e7e68012379db2354", GitTreeState:"dirty", GoVersion:"go1.16.5"}

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:17:57Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.5-eks-bc4871b", GitCommit:"5236faf39f1b7a7dabea8df12726f25608131aa9", GitTreeState:"clean", BuildDate:"2021-10-29T23:32:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1

Which version of the chart: 1.17.2

What happened: Multiple master node

What you expected to happen: Only one master node

How to reproduce it (as minimally and precisely as possible):

helm upgrade --install --namespace "graylog"  "graylog" kongz/graylog --version 1.7.12 -f values.1.7.12.yaml

$ cat values.1.7.12.yaml 
tags:
  install-mongodb: false
  install-elasticsearch: false
graylog:
  image:
    repository: graylog/graylog:4.2.5
  plugins:
    - name: graylog-enterprise-plugins-4.2.5.tgz
      url: https://downloads.graylog.org/releases/graylog-enterprise/graylog-enterprise-plugins-4.2.5.tgz
    - name: graylog-output-syslog-4.0.8.jar
      url: https://github.com/wizecore/graylog2-output-syslog/releases/download/v4.0.8/graylog-output-syslog-4.0.8.jar
  replicas: 3
  resources:
    limits:
      cpu: "3.75"
      memory: "6.5Gi"
    requests:
      cpu: "1"
      memory: "2Gi"
  heapSize: "3g"
  service:
    type: LoadBalancer
    port: 443
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:****:certificate/*******
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
    master:
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
        port: 9000
  input:
     tcp:
       service:
         annotations:
           service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
         type: LoadBalancer
         loadBalancerIP:
       ports:
         - name: gelf
           port: 12201
         - name: syslog
           port: 12202
  externalUri: graylog.*******.com
  externalUriTLS: true
  nodeSelector:
    node.*******.com/component: graylog
  persistence:
    storageClass: allowexpand-gp3
    size: 50Gi
  journal:
    maxSize: 45G
  updateStrategy: OnDelete
  #updateStrategy: RollingUpdate
  terminationGracePeriodSeconds: 300
  #terminationGracePeriodSeconds: 30

  config: |
    output_batch_size = 2500
    inputbuffer_processors = 1
    processbuffer_processors = 5
    outputbuffer_processors = 2

  mongodb:
    uri: mongodb://mongodb-headless.graylog.svc.cluster.local:27017/graylog?replicaSet=rs0 
  elasticsearch:
    hosts: https://graylog:*****@opensearch.*******.com
    version: 7

Anything else we need to know:

$ kubectl get pod graylog-0 -o=json | jq -r '.metadata.labels'
{
  "app.kubernetes.io/instance": "graylog",
  "app.kubernetes.io/managed-by": "Helm",
  "app.kubernetes.io/name": "graylog",
  "app.kubernetes.io/version": "4.0.6",
  "controller-revision-hash": "graylog-58bd58bc4b",
  "helm.sh/chart": "graylog-1.7.12",
  "statefulset.kubernetes.io/pod-name": "graylog-0"
}

Graylog deployed on self-managed (brought up with kubespray) with similar vaules file has the correct labelled pods with only one master node. self-managed elasticsearch is used.

Also the ELB is net set to SSL and the cert is not associated (after deploy, I have to manaually perform setup)

KongZ / charts

All Graylog Nodes Are Master in AWS EKS #112