Master not reachable when "enableConnectionPooler"=True

Hey,

i have just started using the postgres operator of zalando. I am using the provided helm charts for the operator and the ui in the version of the v1.6.0 tag (586b46d0).

All the deployments seem to work fine. If i build a Cluster, there seems to be no issues at all. But as soon as i want to enable the connection pooler, that status in the ui is stuck at "Waiting for master to become available". I don't really find any relevant log messages about the issue.

I tried a bunch of different combinations, with other postgres versions, less/more replicas and also tried to create the postgres via kubectl directly. But in all cases the same issue occurs as soon as i enable the connection pooler.

I am using Helm v3.4.2, with the charts provided in this repository. I reseted all configuration that i made in the values.yaml's, now the only thing changed to the original is the (listening-)namespace.

The Kubernetes Cluster is version v1.19.4 and my kubectl 1.18.3. The k8s cluster contains of multiple Nodes on virtual-machines via rke, the storage is provided via longhorn.

here is my values.yaml for the operator helm chart:

image:
  registry: registry.opensource.zalan.do
  repository: acid/postgres-operator
  tag: v1.6.0
  pullPolicy: "IfNotPresent"

# Optionally specify an array of imagePullSecrets.
# Secrets must be manually created in the namespace.
# ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
# imagePullSecrets:
  # - name: myRegistryKeySecretName

podAnnotations: {}
podLabels: {}

configTarget: "ConfigMap"

# JSON logging format
enableJsonLogging: false

# general configuration parameters
configGeneral:
  # choose if deployment creates/updates CRDs with OpenAPIV3Validation
  enable_crd_validation: "true"
  # update only the statefulsets without immediately doing the rolling update
  enable_lazy_spilo_upgrade: "false"
  # set the PGVERSION env var instead of providing the version via postgresql.bin_dir in SPILO_CONFIGURATION
  enable_pgversion_env_var: "true"
  # start any new database pod without limitations on shm memory
  enable_shm_volume: "true"
  # enables backwards compatible path between Spilo 12 and Spilo 13 images
  enable_spilo_wal_path_compat: "false"
  # etcd connection string for Patroni. Empty uses K8s-native DCS.
  etcd_host: ""
  # Select if setup uses endpoints (default), or configmaps to manage leader (DCS=k8s)
  # kubernetes_use_configmaps: "false"
  # Spilo docker image
  docker_image: registry.opensource.zalan.do/acid/spilo-13:2.0-p2
  # max number of instances in Postgres cluster. -1 = no limit
  min_instances: "-1"
  # min number of instances in Postgres cluster. -1 = no limit
  max_instances: "-1"
  # period between consecutive repair requests
  repair_period: 5m
  # period between consecutive sync requests
  resync_period: 30m
  # can prevent certain cases of memory overcommitment
  # set_memory_request_to_limit: "false"

  # map of sidecar names to docker images
  # sidecar_docker_images: ""

  # number of routines the operator spawns to process requests concurrently
  workers: "8"

# parameters describing Postgres users
configUsers:
  # postgres username used for replication between instances
  replication_username: standby
  # postgres superuser name to be created by initdb
  super_username: postgres

configKubernetes:
  # default DNS domain of K8s cluster where operator is running
  cluster_domain: cluster.local
  # additional labels assigned to the cluster objects
  cluster_labels: application:spilo
  # label assigned to Kubernetes objects created by the operator
  cluster_name_label: cluster-name
  # annotations attached to each database pod
  # custom_pod_annotations: "keya:valuea,keyb:valueb"

  # key name for annotation that compares manifest value with current date
  # delete_annotation_date_key: "delete-date"

  # key name for annotation that compares manifest value with cluster name
  # delete_annotation_name_key: "delete-clustername"

  # list of annotations propagated from cluster manifest to statefulset and deployment
  # downscaler_annotations: "deployment-time,downscaler/*"

  # enables initContainers to run actions before Spilo is started
  enable_init_containers: "true"
  # toggles pod anti affinity on the Postgres pods
  enable_pod_antiaffinity: "false"
  # toggles PDB to set to MinAvailabe 0 or 1
  enable_pod_disruption_budget: "true"
  # enables sidecar containers to run alongside Spilo in the same pod
  enable_sidecars: "true"
  # namespaced name of the secret containing infrastructure roles names and passwords
  # infrastructure_roles_secret_name: postgresql-infrastructure-roles

  # list of annotation keys that can be inherited from the cluster manifest
  # inherited_annotations: owned-by

  # list of label keys that can be inherited from the cluster manifest
  # inherited_labels: application,environment

  # timeout for successful migration of master pods from unschedulable node
  # master_pod_move_timeout: 20m

  # set of labels that a running and active node should possess to be considered ready
  # node_readiness_label: ""

  # namespaced name of the secret containing the OAuth2 token to pass to the teams API
  # oauth_token_secret_name: postgresql-operator

  # defines the template for PDB (Pod Disruption Budget) names
  pdb_name_format: "postgres-{cluster}-pdb"
  # override topology key for pod anti affinity
  pod_antiaffinity_topology_key: "kubernetes.io/hostname"
  # namespaced name of the ConfigMap with environment variables to populate on every pod
  # pod_environment_configmap: "default/my-custom-config"
  # name of the Secret (in cluster namespace) with environment variables to populate on every pod
  # pod_environment_secret: "my-custom-secret"

  # specify the pod management policy of stateful sets of Postgres clusters
  pod_management_policy: "ordered_ready"
  # label assigned to the Postgres pods (and services/endpoints)
  pod_role_label: spilo-role
  # service account definition as JSON/YAML string to be used by postgres cluster pods
  # pod_service_account_definition: ""

  # role binding definition as JSON/YAML string to be used by pod service account
  # pod_service_account_role_binding_definition: ""

  # Postgres pods are terminated forcefully after this timeout
  pod_terminate_grace_period: 5m
  # template for database user secrets generated by the operator
  secret_name_template: "{username}.{cluster}.credentials.{tprkind}.{tprgroup}"
  # set user and group for the spilo container (required to run Spilo as non-root process)
  # spilo_runasuser: "101"
  # spilo_runasgroup: "103"
  # group ID with write-access to volumes (required to run Spilo as non-root process)
  # spilo_fsgroup: "103"

  # whether the Spilo container should run in privileged mode
  spilo_privileged: "false"
  # storage resize strategy, available options are: ebs, pvc, off
  storage_resize_mode: pvc
  # operator watches for postgres objects in the given namespace
  watched_namespace: "postgres-operator"  # listen to all namespaces

# configure resource requests for the Postgres pods
configPostgresPodResources:
  # CPU limits for the postgres containers
  default_cpu_limit: "1"
  # CPU request value for the postgres containers
  default_cpu_request: 100m
  # memory limits for the postgres containers
  default_memory_limit: 500Mi
  # memory request value for the postgres containers
  default_memory_request: 100Mi
  # hard CPU minimum required to properly run a Postgres cluster
  min_cpu_limit: 250m
  # hard memory minimum required to properly run a Postgres cluster
  min_memory_limit: 250Mi

# timeouts related to some operator actions
configTimeouts:
  # timeout when waiting for the Postgres pods to be deleted
  pod_deletion_wait_timeout: 10m
  # timeout when waiting for pod role and cluster labels
  pod_label_wait_timeout: 10m
  # interval between consecutive attempts waiting for postgresql CRD to be created
  ready_wait_interval: 3s
  # timeout for the complete postgres CRD creation
  ready_wait_timeout: 30s
  # interval to wait between consecutive attempts to check for some K8s resources
  resource_check_interval: 3s
  # timeout when waiting for the presence of a certain K8s resource (e.g. Sts, PDB)
  resource_check_timeout: 10m

# configure behavior of load balancers
configLoadBalancer:
  # DNS zone for cluster DNS name when load balancer is configured for cluster
  db_hosted_zone: db.example.com
  # annotations to apply to service when load balancing is enabled
  # custom_service_annotations: "keyx:valuez,keya:valuea"

  # toggles service type load balancer pointing to the master pod of the cluster
  enable_master_load_balancer: "false"
  # toggles service type load balancer pointing to the replica pod of the cluster
  enable_replica_load_balancer: "false"
  # define external traffic policy for the load balancer
  external_traffic_policy: "Cluster"
  # defines the DNS name string template for the master load balancer cluster
  master_dns_name_format: '{cluster}.{team}.{hostedzone}'
  # defines the DNS name string template for the replica load balancer cluster
  replica_dns_name_format: '{cluster}-repl.{team}.{hostedzone}'

# options to aid debugging of the operator itself
configDebug:
  # toggles verbose debug logs from the operator
  debug_logging: "true"
  # toggles operator functionality that require access to the postgres database
  enable_database_access: "true"

# parameters affecting logging and REST API listener
configLoggingRestApi:
  # REST API listener listens to this port
  api_port: "8080"
  # number of entries in the cluster history ring buffer
  cluster_history_entries: "1000"
  # number of lines in the ring buffer used to store cluster logs
  ring_log_lines: "100"

# configure interaction with non-Kubernetes objects from AWS or GCP
configAwsOrGcp:
  # Additional Secret (aws or gcp credentials) to mount in the pod
  # additional_secret_mount: "some-secret-name"

  # Path to mount the above Secret in the filesystem of the container(s)
  # additional_secret_mount_path: "/some/dir"

  # AWS region used to store ESB volumes
  aws_region: eu-central-1

  # enable automatic migration on AWS from gp2 to gp3 volumes
  enable_ebs_gp3_migration: "false"
  # defines maximum volume size in GB until which auto migration happens
  # enable_ebs_gp3_migration_max_size: "1000"

  # GCP credentials for setting the GOOGLE_APPLICATION_CREDNETIALS environment variable
  # gcp_credentials: ""

  # AWS IAM role to supply in the iam.amazonaws.com/role annotation of Postgres pods
  # kube_iam_role: ""

  # S3 bucket to use for shipping postgres daily logs
  # log_s3_bucket: ""

  # S3 bucket to use for shipping WAL segments with WAL-E
  # wal_s3_bucket: ""

  # GCS bucket to use for shipping WAL segments with WAL-E
  # wal_gs_bucket: ""

# configure K8s cron job managed by the operator
configLogicalBackup:
  # image for pods of the logical backup job (example runs pg_dumpall)
  logical_backup_docker_image: "registry.opensource.zalan.do/acid/logical-backup:v1.6.0"
  # path of google cloud service account json file
  # logical_backup_google_application_credentials: ""

  # prefix for the backup job name
  logical_backup_job_prefix: "logical-backup-"
  # storage provider - either "s3" or "gcs"
  logical_backup_provider: "s3"
  # S3 Access Key ID
  logical_backup_s3_access_key_id: ""
  # S3 bucket to store backup results
  logical_backup_s3_bucket: "my-bucket-url"
  # S3 endpoint url when not using AWS
  logical_backup_s3_endpoint: ""
  # S3 region of bucket
  logical_backup_s3_region: ""
  # S3 Secret Access Key
  logical_backup_s3_secret_access_key: ""
  # S3 server side encryption
  logical_backup_s3_sse: "AES256"
  # backup schedule in the cron format
  logical_backup_schedule: "30 00 * * *"

# automate creation of human users with teams API service
configTeamsApi:
  # team_admin_role will have the rights to grant roles coming from PG manifests
  # enable_admin_role_for_users: "true"

  # operator watches for PostgresTeam CRs to assign additional teams and members to clusters
  enable_postgres_team_crd: "false"
  # toogle to create additional superuser teams from PostgresTeam CRs
  # enable_postgres_team_crd_superusers: "false"

  # toggle to grant superuser to team members created from the Teams API
  # enable_team_superuser: "false"

  # toggles usage of the Teams API by the operator
  enable_teams_api: "false"
  # should contain a URL to use for authentication (username and token)
  # pam_configuration: https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees

  # operator will add all team member roles to this group and add a pg_hba line
  # pam_role_name: zalandos

  # List of teams which members need the superuser role in each Postgres cluster
  # postgres_superuser_teams: "postgres_superusers"

  # List of roles that cannot be overwritten by an application, team or infrastructure role
  # protected_role_names: "admin"

  # role name to grant to team members created from the Teams API
  # team_admin_role: "admin"

  # postgres config parameters to apply to each team member role
  # team_api_role_configuration: "log_statement:all"

  # URL of the Teams API service
  # teams_api_url: http://fake-teams-api.default.svc.cluster.local

# configure connection pooler deployment created by the operator
configConnectionPooler:
  # db schema to install lookup function into
  connection_pooler_schema: "pooler"
  # db user for pooler to use
  connection_pooler_user: "pooler"
  # docker image
  connection_pooler_image: "registry.opensource.zalan.do/acid/pgbouncer:master-9"
  # max db connections the pooler should hold
  connection_pooler_max_db_connections: "60"
  # default pooling mode
  connection_pooler_mode: "transaction"
  # number of pooler instances
  connection_pooler_number_of_instances: "2"
  # default resources
  connection_pooler_default_cpu_request: 500m
  connection_pooler_default_memory_request: 100Mi
  connection_pooler_default_cpu_limit: "1"
  connection_pooler_default_memory_limit: 100Mi

rbac:
  # Specifies whether RBAC resources should be created
  create: true

crd:
  # Specifies whether custom resource definitions should be created
  # When using helm3, this is ignored; instead use "--skip-crds" to skip.
  create: true

serviceAccount:
  # Specifies whether a ServiceAccount should be created
  create: true
  # The name of the ServiceAccount to use.
  # If not set and create is true, a name is generated using the fullname template
  name:

podServiceAccount:
  # The name of the ServiceAccount to be used by postgres cluster pods
  # If not set a name is generated using the fullname template and "-pod" suffix
  name: "postgres-pod"

# priority class for operator pod
priorityClassName: ""

# priority class for database pods
podPriorityClassName: ""

resources:
  limits:
    cpu: 500m
    memory: 500Mi
  requests:
    cpu: 100m
    memory: 250Mi

# Affinity for pod assignment
# Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
affinity: {}

# Tolerations for pod assignment
# Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: []

# Node labels for pod assignment
# Ref: https://kubernetes.io/docs/user-guide/node-selection/
nodeSelector: {}

controllerID:
  # Specifies whether a controller ID should be defined for the operator
  # Note, all postgres manifest must then contain the following annotation to be found by this operator
  # "acid.zalan.do/controller": <controller-ID-of-the-operator>
  create: false
  # The name of the controller ID to use.
  # If not set and create is true, a name is generated using the fullname template
  name:

Also the values.yaml for the ui helm chart:

# Default values for postgres-operator-ui.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

# configure ui image
image:
  registry: registry.opensource.zalan.do
  repository: acid/postgres-operator-ui
  tag: v1.6.0
  pullPolicy: "IfNotPresent"

# Optionally specify an array of imagePullSecrets.
# Secrets must be manually created in the namespace.
# ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
# imagePullSecrets:
#   - name: 

rbac:
  # Specifies whether RBAC resources should be created
  create: true

serviceAccount:
  # Specifies whether a ServiceAccount should be created
  create: true
  # The name of the ServiceAccount to use.
  # If not set and create is true, a name is generated using the fullname template
  name:

# configure UI pod resources
resources:
  limits:
    cpu: 200m
    memory: 200Mi
  requests:
    cpu: 100m
    memory: 100Mi

# configure UI ENVs
envs:
  # IMPORTANT: While operator chart and UI chart are idendependent, this is the interface between
  # UI and operator API. Insert the service name of the operator API here!
  operatorApiUrl: "http://postgres-operator:8080"
  operatorClusterNameLabel: "cluster-name"
  resourcesVisible: "False"
  targetNamespace: "postgres-operator"

# configure UI service
service:
  type: "ClusterIP"
  port: "80"
  # If the type of the service is NodePort a port can be specified using the nodePort field
  # If the nodePort field is not specified, or if it has no value, then a random port is used
  # notePort: 32521

# configure UI ingress. If needed: "enabled: true"
ingress:
  enabled: false
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  hosts:
    - host: ui.example.org
      paths: [""]
  tls: []
  #  - secretName: ui-tls
  #    hosts:
  #      - ui.exmaple.org

I would really appreciate any help. If you need any additional info, feel free to ask :-)

thanks in advance.

zalando / postgres-operator

Master not reachable when "enableConnectionPooler"=True #1304