Splunk Operator: Indexer Clusters - Decreasing replica count does not shutdown the indexer pods

[gjanders]

Please select the type of request

Bug

Tell us more

Describe the request

• I have a multi-site cluster with 2 sites, site1, site2. I have 4 indexers/site (replicaCount: 4), I want to decrease to 2 indexers/site

Expected behavior

• I expect 2 indexers in each site to shutdown and the pods to exit

Splunk setup on K8S helm chart overrides:

splunk-operator:
  enabled: false

indexerCluster:
  enabled: true
  name: mobiles-idxc-site1
  replicaCount: 2
  etcVolumeStorageConfig:
    ephemeralStorage: false
    storageCapacity: 5Gi
  # storageClassName: gp2

  varVolumeStorageConfig:
    ephemeralStorage: false
    storageCapacity: 1000Gi
  # storageClassName: gp2
  resources:
     requests:
       memory: "40Gi"
       cpu: "20"
     limits:
       memory: "62Gi"
       cpu: "32"
  clusterManagerRef:
    name: mobiles-cm
  defaults:
    splunk:
      multisite_master: splunk-mobiles-cm-cluster-manager-service
      site: site1
  serviceTemplate:
    spec:
      ports:
      - name: tls-s2s
        port: 9998
        protocol: TCP
        targetPort: 9998
      - name: https-hec
        port: 8087
        protocol: TCP
        targetPort: 8087
  affinity:
    ## Example: schedule Splunk instance pod on a node in zone-1a
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: mobile-iot-site
            operator: In
            values:
            - "1"

Reproduction/Testing steps

• Scale to 4 indexers/site. Change to 2 replicaCount per-site, and then helm upgrade -f mobiles-indexer-cluster-site1.yaml or similar

K8s environment

• 1.23.1

K8s collector data(optional)

• Please provide data collected from the K8s collectors, if any.

Additional context(optional)

• The 2 indexers go into decommissioning status in the indexer clustering page of the manager node. Then shutting down, then they come back in the "up" status. This scenario repeats in an infinite loop. Note that I upgraded 2 helm charts at the same time to decrease from 4->2, and I let the loop run for 3-4 iterations of decomm/shutting down/up before running kubectl delete pods...

[gjanders]

If I check the indexercluster CRD it says replicas "2": kubectl edit indexercluster -n mobiles ...

But when I check the statefulset it has 4 replicas, so perhaps that is why they failed to decommission as expected? Because the statefulset did not match the indexercluster CRD? (which is managed by the operator)

[gjanders]

FYI this is still an issue in 2.5.2. Any updates on this issue?