raising replica and min_quorom to 6 does not allow node to auto join raft into ready state

[nia-potato]

Describe the bug after raising replica and min_quorom to 6 in the helm chart, running k get pods only shows 5 pods are ready instead of joining the 6 pod to raft.

To Reproduce Steps to reproduce the behavior: install chart with config of

  ha:
    enabled: true
    replicas: 6
    raft:
      enabled: true
      setNodeId: true
      config: |
*********OTHER_CONFIG******
 storage "raft" {
          path = "/vault/data"
          performance_multiplier = 1
          retry_join {
            auto_join = "provider=k8s label_selector=\"app=vault,component=server\" namespace=\"vault\" "
            leader_tls_servername = "primary-vault"
            auto_join_scheme = "https"
            leader_ca_cert_file = "/vault/userconfig/vault-tls/vault-ca.pem"
            leader_client_cert_file = "/vault/userconfig/vault-tls/vault-cert.pem"
            leader_client_key_file   = "/vault/userconfig/vault-tls/vault-key.pem"
          }
          autopilot {
            cleanup_dead_servers = "true"
            last_contact_threshold = "200ms"
            last_contact_failure_threshold = "10m"
            max_trailing_logs = 250000
            min_quorum = 6
            server_stabilization_time = "10s"
          }
        }

k get pods 

primary-vault-0   1/1     Running   0          117s
primary-vault-1   1/1     Running   0          117s
primary-vault-2   1/1     Running   0          117s
primary-vault-3   1/1     Running   0          117s
primary-vault-4   1/1     Running   0          117s
primary-vault-5   0/1     Running   0          117s

Other useful info to include: vault pod logs, kubectl describe statefulset vault and kubectl get statefulset vault -o yaml output

Expected behavior replica of 6 + min_quorum of 6 should have 6 nodes all in ready state and joined in raft.

Environment

• Kubernetes version: GKE, vault-enterprise

[maxb]

• You have not mentioned which seal method is in use for this Vault. A new node requires access to the unseal key to join the cluster, which may mean manual unsealing.

• A Raft cluster of 6 nodes is questionable, since it is no more fault-tolerant than a 5 node cluster. The failure tolerance is 2 in both cases.

• I'm pretty sure the autopilot settings in the configuration file only apply when initializing a new cluster, and to modify an existing cluster, changes must be made via the Vault API.