operator 1.6.0 pooler not beeing deployed and pg_hba.conf not additive anymore

[haroonb]

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.6.0
• Where do you run it - cloud or metal? Kubernetes or OpenShift? VM K8s
• Are you running Postgres Operator in production? not, in this instance
• Type of issue? Bug report, question

Hi everyone, we are planning on updating our production postgres operator in the near future. I'm running a few tests on a test cluster and am running into a few issues. Currently we're using the operator from version 1.4.0.

Iis it intended, that the pg_hba section is not additive anymore, because I configured both statefulsets the same, but the pg_hba.conf is different.

The next issue I'm experiencing is, that the pooler is not beeing deployed, even though I gave the parameters for the schema and user. The operator is giving a debug message about resolved a conflict of role \"pgbouncer\" between manifest role and connection pooler role to connection pooler role" cluster-name=postgres-test/acid-test-cluster pkg=cluster, but I don't know what that is supposed to mean.

NEW
devops-m1:~$ kubectl exec -it acid-test-cluster-0 -- cat /home/postgres/pgdata/pgroot/data/pg_hba.conf
# Do not edit this file manually!
# It will be overwritten by Patroni!
hostssl all all 0.0.0.0/0 md5
host    all all 0.0.0.0/0 md5
hostssl replication postgres 0.0.0.0/0 md5
host replication postgres 0.0.0.0/0 md5
command terminated with exit code 1

OLD
devops-m1:~$ kubectl exec -it acid-postgres-cluster-0 -- cat /home/postgres/pgdata/pgroot/data/pg_hba.conf
# Do not edit this file manually!
# It will be overwritten by Patroni!
hostssl all all 0.0.0.0/0 md5
hostssl replication postgres 0.0.0.0/0 md5
host    all all 0.0.0.0/0 md5
host replication postgres 0.0.0.0/0 md5
local   all             all                                   trust
hostssl all             +zalandos    127.0.0.1/32       pam
host    all             all                127.0.0.1/32       md5
hostssl all             +zalandos    ::1/128            pam
host    all             all                ::1/128            md5
hostssl replication     standby all                md5
hostnossl all           all                all                reject
hostssl all             +zalandos    all                pam
hostssl all             all                all                md5

    - name: create postgres cluster
188       k8s:
187         definition:
186           apiVersion: "acid.zalan.do/v1"
185           kind: postgresql
184           metadata:
183             name: acid-test-cluster
182             namespace: default
181           spec:
180             dockerImage: registry.opensource.zalan.do/acid/spilo-13:2.0-p2
179             teamId: "acid"
178             numberOfInstances: 3 # 2
177             users:  # Application/Robot users
176               pgbouncer:
175               - superuser
174               - createdb
173             enableLogicalBackup: false
172             enableMasterLoadBalancer: false
171             enableReplicaLoadBalancer: false
170             enableConnectionPooler: true # enable/disable connection pooler deployment
169             enableReplicaConnectionPooler: false # set to enable connectionPooler for replica service
168             allowedSourceRanges:  # load balancers' source ranges for both master and replica services
167             - 127.0.0.1/32
166 #            databases:
165 #              foo: zalando
164 #            preparedDatabases:
163 #              bar:
162 #                defaultUsers: true
161 #                extensions:
160 #                  pg_partman: public
159 #                  pgcrypto: public
158 #                schemas:
157 #                  data: {}
156 #                  history:
155 #                    defaultRoles: true
154 #                    defaultUsers: false
153             postgresql:
152               version: "{{ postgres_version }}"
151               parameters:  # Expert section
150                 tcp_keepalives_count: "5"
149                 tcp_keepalives_interval: "60"
148                 tcp_keepalives_idle: "60"
147                 shared_buffers: "4GB"
146                 max_connections: "300"
145                 log_statement: "none"
144             volume:
143               size: "{{ postgres_pvsize }}G"
142               storageClass: rook-ceph-block
141             additionalVolumes:
140               - name: empty
139                 mountPath: /opt/empty
138                 targetContainers:
137                   - all
136                 volumeSource:
135                   emptyDir: {}
134           #    - name: data
133           #      mountPath: /home/postgres/pgdata/partitions
132           #      targetContainers:
131           #        - postgres
130           #      volumeSource:
129           #        PersistentVolumeClaim:
128           #          claimName: pvc-postgresql-data-partitions
127           #          readyOnly: false
126           #    - name: conf
125           #      mountPath: /etc/telegraf
124           #      subPath: telegraf.conf
123           #      targetContainers:
122           #        - telegraf-sidecar
121           #      volumeSource:
120           #        configMap:
119           #          name: my-config-map
118
117             enableShmVolume: true
116           #  spiloRunAsUser: 101
115           #  spiloRunAsGroup: 103
114           #  spiloFSGroup: 103
113             podAnnotations:
112               backup.velero.io/backup-volumes: pgdata
111           #  serviceAnnotations:
110           #    annotation.key: value
109           #  podPriorityClassName: "spilo-pod-priority"
108           #  tolerations:
107           #  - key: postgres
106           #    operator: Exists
105           #    effect: NoSchedule
104             resources:
103               requests:
102                 cpu: 10m
101                 memory: 100Mi
100               limits:
 99                 cpu: 5000m # 500m
 98                 memory: 5000Mi # 500Mi
 97             patroni:
 96               initdb:
 95                 encoding: "UTF8"
 94                 locale: "en_US.UTF-8"
 93                 data-checksums: "true"
 92               pg_hba:
 91                 - hostssl all all 0.0.0.0/0 md5
 90                 - host    all all 0.0.0.0/0 md5
 89                 - hostssl replication postgres 0.0.0.0/0 md5
 88                 - host replication postgres 0.0.0.0/0 md5
 87           #    slots:
 86           #      permanent_physical_1:
 85           #        type: physical
 84           #      permanent_logical_1:
 83           #        type: logical
 82           #        database: foo
 81           #        plugin: pgoutput
 80               ttl: 30
 79               loop_wait: &loop_wait 10
 78               retry_timeout: 10
 77               synchronous_mode: false
 76               synchronous_mode_strict: false
 75               maximum_lag_on_failover: 33554432
 74
 73           # restore a Postgres DB with point-in-time-recovery
 72           # with a non-empty timestamp, clone from an S3 bucket using the latest backup before the timestamp
 71           # with an empty/absent timestamp, clone from an existing alive cluster using pg_basebackup
 70           #  clone:
 69           #    uid: "efd12e58-5786-11e8-b5a7-06148230260c"
 68           #    cluster: "acid-batman"
 67           #    timestamp: "2017-12-19T12:40:33+01:00"  # timezone required (offset relative to UTC, see RFC 3339 section 5.6)
 66           #    s3_wal_path: "s3://custom/path/to/bucket"
 65
 64           # run periodic backups with k8s cron jobs
 63           #  enableLogicalBackup: true
 62           #  logicalBackupSchedule: "30 00 * * *"
 61
 60           #  maintenanceWindows:
 59           #  - 01:00-06:00  #UTC
 58           #  - Sat:00:00-04:00
 57
 56           # overwrite custom properties for connection pooler deployments
 55             connectionPooler:
 54               numberOfInstances: 3
 53               mode: "transaction"
 52               schema: "pgbouncer"
 51               user: "pgbouncer"
 50               maxDBConnections: 300 # custom
 49               resources:
 48                 requests:
 47                   cpu: 300m
 46                   memory: 100Mi
 45                 limits:
 44                   cpu: "1"
 43                   memory: 100Mi
 42
 41             initContainers:
 40             - name: date
 39               image: busybox
 38               command: [ "/bin/date" ]
 37           #  sidecars:
 36           #    - name: "telegraf-sidecar"
 35           #      image: "telegraf:latest"
 34           #      resources:
 33           #        limits:
 32           #          cpu: 500m
 31           #          memory: 500Mi
 30           #        requests:
 29           #          cpu: 100m
 28           #          memory: 100Mi
 27           #      env:
 26           #        - name: "USEFUL_VAR"
 25           #          value: "perhaps-true"
 24
 23           # Custom TLS certificate. Disabled unless tls.secretName has a value.
 22             tls:
 21               secretName: ""  # should correspond to a Kubernetes Secret resource to load
 20               certificateFile: "tls.crt"
 19               privateKeyFile: "tls.key"
 18               caFile: ""  # optionally configure Postgres with a CA certificate
 17               caSecretName: "" # optionally the ca.crt can come from this secret instead.
 16           # file names can be also defined with absolute path, and will no longer be relative
 15           # to the "/tls/" path where the secret is being mounted by default, and "/tlsca/"
 14           # where the caSecret is mounted by default.
 13           # When TLS is enabled, also set spiloFSGroup parameter above to the relevant value.
 12           # if unknown, set it to 103 which is the usual value in the default spilo images.
 11           # In Openshift, there is no need to set spiloFSGroup/spilo_fsgroup.
 10
  9           # Add node affinity support by allowing postgres pods to schedule only on nodes that
  8           # have label: "postgres-operator:enabled" set.
  7           #  nodeAffinity:
  6           #    requiredDuringSchedulingIgnoredDuringExecution:
  5           #      nodeSelectorTerms:
  4           #        - matchExpressions:
  3           #            - key: postgres-operator
  2           #              operator: In
  1           #              values:
226           #                - enabled

postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:54Z" level=info msg="Fully qualified configmap name: default/postgres-operator"
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:54Z" level=info msg="Spilo operator v1.6.0\n"
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:54Z" level=info msg="Parse role bindings" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:54Z" level=info msg="successfully parsed" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:54Z" level=info msg="Listening to all namespaces" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:54Z" level=info msg="customResourceDefinition \"postgresqls.acid.zalan.do\" is already registered and will only be updated" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="{" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ReadyWaitInterval\": 3000000000," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ReadyWaitTimeout\": 30000000000," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ResyncPeriod\": 1800000000000," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"RepairPeriod\": 300000000000," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnableCRDValidation\": true," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ResourceCheckInterval\": 3000000000," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ResourceCheckTimeout\": 600000000000," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PodLabelWaitTimeout\": 600000000000," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PodDeletionWaitTimeout\": 600000000000," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"SpiloFSGroup\": null," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PodPriorityClassName\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ClusterDomain\": \"cluster.local\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"SpiloPrivileged\": false," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ClusterLabels\": {" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="      \"application\": \"spilo\"" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   }," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"InheritedLabels\": null," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"InheritedAnnotations\": null," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"DownscalerAnnotations\": null," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ClusterNameLabel\": \"cluster-name\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"DeleteAnnotationDateKey\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"DeleteAnnotationNameKey\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PodRoleLabel\": \"spilo-role\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PodToleration\": null," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"DefaultCPURequest\": \"100m\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"DefaultMemoryRequest\": \"100Mi\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"DefaultCPULimit\": \"1\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"DefaultMemoryLimit\": \"500Mi\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"MinCPULimit\": \"250m\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"MinMemoryLimit\": \"250Mi\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PodEnvironmentConfigMap\": \"/\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PodEnvironmentSecret\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"NodeReadinessLabel\": null," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"MaxInstances\": -1," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"MinInstances\": -1," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ShmVolume\": true," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"SecretNameTemplate\": \"{username}.{cluster}.credentials\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PamRoleName\": \"zalandos\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PamConfiguration\": \"https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"TeamsAPIUrl\": \"https://teams.example.com/api/\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"OAuthTokenSecretName\": \"default/postgresql-operator\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"InfrastructureRolesSecretName\": \"/\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"InfrastructureRoles\": null," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"InfrastructureRolesDefs\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"SuperUsername\": \"postgres\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ReplicationUsername\": \"standby\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ScalyrAPIKey\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ScalyrImage\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ScalyrServerURL\": \"https://upload.eu.scalyr.com\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ScalyrCPURequest\": \"100m\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ScalyrMemoryRequest\": \"50Mi\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ScalyrCPULimit\": \"1\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ScalyrMemoryLimit\": \"500Mi\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"LogicalBackupSchedule\": \"30 00 * * *\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"LogicalBackupDockerImage\": \"registry.opensource.zalan.do/acid/logical-backup:v1.6.0\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"LogicalBackupProvider\": \"s3\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"LogicalBackupS3Bucket\": \"my-bucket-url\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"LogicalBackupS3Region\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"LogicalBackupS3Endpoint\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"LogicalBackupS3AccessKeyID\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"LogicalBackupS3SecretAccessKey\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"LogicalBackupS3SSE\": \"AES256\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"LogicalBackupGoogleApplicationCredentials\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"NumberOfInstances\": 2," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"Schema\": \"pooler\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"User\": \"pooler\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"Image\": \"registry.opensource.zalan.do/acid/pgbouncer:master-12\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"Mode\": \"transaction\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"MaxDBConnections\": 60," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ConnectionPoolerDefaultCPURequest\": \"500m\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ConnectionPoolerDefaultMemoryRequest\": \"100Mi\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ConnectionPoolerDefaultCPULimit\": \"1\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ConnectionPoolerDefaultMemoryLimit\": \"100Mi\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"WatchedNamespace\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"KubernetesUseConfigMaps\": false," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EtcdHost\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"DockerImage\": \"registry.opensource.zalan.do/acid/spilo-13:2.0-p2\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"SidecarImages\": null," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"SidecarContainers\": null," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PodServiceAccountName\": \"postgres-pod\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PodServiceAccountDefinition\": \"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"ServiceAccount\\\",\\\"metadata\\\":{\\\"name\\\":\\\"postgres-pod\\\"}}\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PodServiceAccountRoleBindingDefinition\": \"{\\\"apiVersion\\\":\\\"rbac.authorization.k8s.io/v1\\\",\\\"kind\\\":\\\"RoleBinding\\\",\\\"metadata\\\":{\\\"name\\\":\\\"postgres-pod\\\"},\\\"roleRef\\\":{\\\"apiGroup\\\":\\\"rbac.authorization.k8s.io\\\",\\\"kind\\\":\\\"ClusterRole\\\",\\\"name\\\":\\\"postgres-pod\\\"},\\\"subjects\\\":[{\\\"kind\\\":\\\"ServiceAccount\\\",\\\"name\\\":\\\"postgres-pod\\\"}]}\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"MasterPodMoveTimeout\": 1200000000000," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"DbHostedZone\": \"db.example.com\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"AWSRegion\": \"eu-central-1\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"WALES3Bucket\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"LogS3Bucket\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"KubeIAMRole\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"WALGSBucket\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"GCPCredentials\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"AdditionalSecretMount\": \"\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"AdditionalSecretMountPath\": \"/meta/credentials\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnableEBSGp3Migration\": false," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnableEBSGp3MigrationMaxSize\": 1000," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"DebugLogging\": true," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnableDBAccess\": true," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnableTeamsAPI\": false," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnableTeamSuperuser\": false," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"TeamAdminRole\": \"admin\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnableAdminRoleForUsers\": true," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnablePostgresTeamCRD\": false," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnablePostgresTeamCRDSuperusers\": false," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnableMasterLoadBalancer\": false," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnableReplicaLoadBalancer\": false," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"CustomServiceAnnotations\": null," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"CustomPodAnnotations\": null," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnablePodAntiAffinity\": true," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PodAntiAffinityTopologyKey\": \"kubernetes.io/hostname\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"StorageResizeMode\": \"pvc\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnableLoadBalancer\": null," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ExternalTrafficPolicy\": \"Cluster\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"MasterDNSNameFormat\": \"{cluster}.{team}.{hostedzone}\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ReplicaDNSNameFormat\": \"{cluster}-repl.{team}.{hostedzone}\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PDBNameFormat\": \"postgres-{cluster}-pdb\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnablePodDisruptionBudget\": false," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnableInitContainers\": true," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnableSidecars\": true," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"Workers\": 16," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"APIPort\": 8080," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"RingLogLines\": 100," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ClusterHistoryEntries\": 1000," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"TeamAPIRoleConfiguration\": {" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="      \"log_statement\": \"all\"" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   }," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PodTerminateGracePeriod\": 300000000000," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PodManagementPolicy\": \"ordered_ready\"," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"ProtectedRoles\": [" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="      \"admin\"" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   ]," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"PostgresSuperuserTeams\": null," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"SetMemoryRequestToLimit\": false," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnableLazySpiloUpgrade\": false," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnablePgVersionEnvVar\": true," pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="   \"EnableSpiloWalPathCompat\": true" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="}" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="acquiring initial list of clusters" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="added new cluster: \"default/acid-postgres-cluster\"" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="added new cluster: \"postgres-test/acid-test-cluster\"" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="SYNC event has been queued" cluster-name=default/acid-postgres-cluster pkg=controller worker=0
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="SYNC event has been queued" cluster-name=postgres-test/acid-test-cluster pkg=controller worker=1
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="there are 2 clusters running" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="started working in background" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="listening on :8080" pkg=apiserver
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="ADD event has been queued" cluster-name=default/acid-postgres-cluster pkg=controller worker=0
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="ADD event has been queued" cluster-name=postgres-test/acid-test-cluster pkg=controller worker=1
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="new node has been added: /devops-w1 ()" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="new node has been added: /devops-w2 ()" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="new node has been added: /devops-m1 ()" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="new node has been added: /devops-w3 ()" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="syncing of the cluster started" cluster-name=default/acid-postgres-cluster pkg=controller worker=0
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="fetching possible additional team members for team \"acid\"" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="team API is disabled, only returning 0 members for team \"acid\"" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="syncing secrets" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="syncing of the cluster started" cluster-name=postgres-test/acid-test-cluster pkg=controller worker=1
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="resolved a conflict of role \"pgbouncer\" between manifest role and connection pooler role to connection pooler role" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="fetching possible additional team members for team \"acid\"" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="team API is disabled, only returning 0 members for team \"acid\"" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=info msg="syncing secrets" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="secret default/postgres.acid-postgres-cluster.credentials already exists, fetching its password" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="secret postgres-test/pgbouncer.acid-test-cluster.credentials already exists, fetching its password" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="secret default/standby.acid-postgres-cluster.credentials already exists, fetching its password" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:58Z" level=debug msg="syncing master service" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:59Z" level=debug msg="secret postgres-test/postgres.acid-test-cluster.credentials already exists, fetching its password" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:59Z" level=debug msg="syncing replica service" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:59Z" level=debug msg="secret postgres-test/standby.acid-test-cluster.credentials already exists, fetching its password" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:04:59Z" level=debug msg="syncing master service" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="syncing replica service" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=info msg="volume claims do not require changes" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="syncing statefulsets" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="[{SCOPE acid-postgres-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-postgres-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-postgres-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 12 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{\"parameters\":{\"log_statement\":\"none\",\"shared_buffers\":\"4GB\",\"tcp_keepalives_count\":\"5\",\"tcp_keepalives_idle\":\"60\",\"tcp_keepalives_interval\":\"60\"},\"pg_hba\":[\"hostssl all all 0.0.0.0/0 md5\",\"hostssl replication postgres 0.0.0.0/0 md5\",\"host    all all 0.0.0.0/0 md5\",\"host replication postgres 0.0.0.0/0 md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{\"ttl\":30,\"loop_wait\":10,\"retry_timeout\":10,\"maximum_lag_on_failover\":33554432,\"postgresql\":{\"parameters\":{\"max_connections\":\"300\"}}}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {ENABLE_WAL_PATH_COMPAT true nil}]" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="set statefulset's rolling update annotation to false: caller/reason from cache" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="set statefulset's rolling update annotation to true: caller/reason statefulset changes" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=info msg="statefulset default/acid-postgres-cluster is not in the desired state and needs to be updated" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-          resources: {}," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-          imagePullPolicy: Always" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+          resources: {}" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-              name: KUBERNETES_LABELS," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-              value: application=spilo" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-            }," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-            {" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+              name: PGVERSION," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+              value: 12" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+            }," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+            {" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+              name: KUBERNETES_LABELS," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+              value: {\\application\\:\\spilo\\}" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+            }," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+            {" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-              value: {\\postgresql\\:{\\bin_dir\\:\\/usr/lib/postgresql/12/bin\\,\\parameters\\:{\\log_statement\\:\\none\\,\\shared_buffers\\:\\4GB\\,\\tcp_keepalives_count\\:\\5\\,\\tcp_keepalives_idle\\:\\60\\,\\tcp_keepalives_interval\\:\\60\\},\\pg_hba\\:[\\hostssl all all 0.0.0.0/0 md5\\,\\hostssl replication postgres 0.0.0.0/0 md5\\,\\host    all all 0.0.0.0/0 md5\\,\\host replication postgres 0.0.0.0/0 md5\\]},\\bootstrap\\:{\\initdb\\:[{\\auth-host\\:\\md5\\},{\\auth-local\\:\\trust\\},\\data-checksums\\,{\\encoding\\:\\UTF8\\},{\\locale\\:\\en_US.UTF-8\\}],\\users\\:{\\zalandos\\:{\\password\\:\\\\,\\options\\:[\\CREATEDB\\,\\NOLOGIN\\]}},\\dcs\\:{\\ttl\\:30,\\loop_wait\\:10,\\retry_timeout\\:10,\\maximum_lag_on_failover\\:33554432,\\postgresql\\:{\\parameters\\:{\\max_connections\\:\\300\\}}}}}" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+              value: {\\postgresql\\:{\\parameters\\:{\\log_statement\\:\\none\\,\\shared_buffers\\:\\4GB\\,\\tcp_keepalives_count\\:\\5\\,\\tcp_keepalives_idle\\:\\60\\,\\tcp_keepalives_interval\\:\\60\\},\\pg_hba\\:[\\hostssl all all 0.0.0.0/0 md5\\,\\hostssl replication postgres 0.0.0.0/0 md5\\,\\host    all all 0.0.0.0/0 md5\\,\\host replication postgres 0.0.0.0/0 md5\\]},\\bootstrap\\:{\\initdb\\:[{\\auth-host\\:\\md5\\},{\\auth-local\\:\\trust\\},\\data-checksums\\,{\\encoding\\:\\UTF8\\},{\\locale\\:\\en_US.UTF-8\\}],\\users\\:{\\zalandos\\:{\\password\\:\\\\,\\options\\:[\\CREATEDB\\,\\NOLOGIN\\]}},\\dcs\\:{\\ttl\\:30,\\loop_wait\\:10,\\retry_timeout\\:10,\\maximum_lag_on_failover\\:33554432,\\postgresql\\:{\\parameters\\:{\\max_connections\\:\\300\\}}}}}" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+            }," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+            {" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+              name: ENABLE_WAL_PATH_COMPAT," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+              value: true" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-      restartPolicy: Always," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-      dnsPolicy: ClusterFirst," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-      serviceAccount: postgres-pod," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-      }," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-      schedulerName: default-scheduler" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+      }" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-      kind: PersistentVolumeClaim," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-      apiVersion: v1," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-      status: {" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-        phase: Pending" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-      }" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+      status: {}" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-  }," cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-  revisionHistoryLimit: 10" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+  }" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="metadata.annotation are different" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="-  zalando-postgres-operator-rolling-update-required: false" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="+  zalando-postgres-operator-rolling-update-required: true" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=info msg="reason: new statefulset containers's postgres (index 0) environment does not match the current one" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="updating statefulset" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="patching statefulset annotations" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:00Z" level=debug msg="patching statefulset annotations" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:01Z" level=debug msg="calling Patroni API on a pod default/acid-postgres-cluster-0 to set the following Postgres options: map[max_connections:300]" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:01Z" level=debug msg="making PATCH http request: http://10.244.2.12:8008/config" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:01Z" level=debug msg="performing rolling update" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:01Z" level=debug msg="No load balancer created for the replica service" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:01Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:01Z" level=info msg="there are 3 pods in the cluster to recreate" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:01Z" level=debug msg="name=acid-postgres-cluster-0 phase=Running ip=10.244.2.12" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:01Z" level=debug msg="name=acid-postgres-cluster-1 phase=Running ip=10.244.1.16" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:01Z" level=debug msg="name=acid-postgres-cluster-2 phase=Running ip=10.244.3.17" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:01Z" level=debug msg="subscribing to pod \"default/acid-postgres-cluster-1\"" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=info msg="volume claims do not require changes" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="syncing statefulsets" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="[{SCOPE acid-test-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-test-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-test-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 12 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{\"parameters\":{\"log_statement\":\"none\",\"shared_buffers\":\"4GB\",\"tcp_keepalives_count\":\"5\",\"tcp_keepalives_idle\":\"60\",\"tcp_keepalives_interval\":\"60\"},\"pg_hba\":[\"hostssl all all 0.0.0.0/0 md5\",\"host    all all 0.0.0.0/0 md5\",\"hostssl replication postgres 0.0.0.0/0 md5\",\"host replication postgres 0.0.0.0/0 md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{\"ttl\":30,\"loop_wait\":10,\"retry_timeout\":10,\"maximum_lag_on_failover\":33554432,\"postgresql\":{\"parameters\":{\"max_connections\":\"300\"}}}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {ENABLE_WAL_PATH_COMPAT true nil}]" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=info msg="Mount additional volumes: [{Name:empty MountPath:/opt/empty SubPath: TargetContainers:[all] VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="set statefulset's rolling update annotation to false: caller/reason from cache" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="set statefulset's rolling update annotation to true: caller/reason statefulset changes" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=info msg="statefulset postgres-test/acid-test-cluster is not in the desired state and needs to be updated" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+        }," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+        {" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+          name: empty," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+          emptyDir: {}" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-          resources: {}," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-          imagePullPolicy: Always" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+          resources: {}" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-              name: KUBERNETES_LABELS," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-              value: application=spilo" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-            }," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-            {" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+              name: PGVERSION," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+              value: 12" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+            }," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+            {" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+              name: KUBERNETES_LABELS," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+              value: {\\application\\:\\spilo\\}" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+            }," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+            {" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-              value: {\\postgresql\\:{\\bin_dir\\:\\/usr/lib/postgresql/12/bin\\,\\parameters\\:{\\log_statement\\:\\none\\,\\shared_buffers\\:\\4GB\\,\\tcp_keepalives_count\\:\\5\\,\\tcp_keepalives_idle\\:\\60\\,\\tcp_keepalives_interval\\:\\60\\},\\pg_hba\\:[\\hostssl all all 0.0.0.0/0 md5\\,\\host    all all 0.0.0.0/0 md5\\,\\hostssl replication postgres 0.0.0.0/0 md5\\,\\host replication postgres 0.0.0.0/0 md5\\]},\\bootstrap\\:{\\initdb\\:[{\\auth-host\\:\\md5\\},{\\auth-local\\:\\trust\\},\\data-checksums\\,{\\encoding\\:\\UTF8\\},{\\locale\\:\\en_US.UTF-8\\}],\\users\\:{\\zalandos\\:{\\password\\:\\\\,\\options\\:[\\CREATEDB\\,\\NOLOGIN\\]}},\\dcs\\:{\\ttl\\:30,\\loop_wait\\:10,\\retry_timeout\\:10,\\maximum_lag_on_failover\\:33554432,\\postgresql\\:{\\parameters\\:{\\max_connections\\:\\300\\}}}}}" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+              value: {\\postgresql\\:{\\parameters\\:{\\log_statement\\:\\none\\,\\shared_buffers\\:\\4GB\\,\\tcp_keepalives_count\\:\\5\\,\\tcp_keepalives_idle\\:\\60\\,\\tcp_keepalives_interval\\:\\60\\},\\pg_hba\\:[\\hostssl all all 0.0.0.0/0 md5\\,\\host    all all 0.0.0.0/0 md5\\,\\hostssl replication postgres 0.0.0.0/0 md5\\,\\host replication postgres 0.0.0.0/0 md5\\]},\\bootstrap\\:{\\initdb\\:[{\\auth-host\\:\\md5\\},{\\auth-local\\:\\trust\\},\\data-checksums\\,{\\encoding\\:\\UTF8\\},{\\locale\\:\\en_US.UTF-8\\}],\\users\\:{\\zalandos\\:{\\password\\:\\\\,\\options\\:[\\CREATEDB\\,\\NOLOGIN\\]}},\\dcs\\:{\\ttl\\:30,\\loop_wait\\:10,\\retry_timeout\\:10,\\maximum_lag_on_failover\\:33554432,\\postgresql\\:{\\parameters\\:{\\max_connections\\:\\300\\}}}}}" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+            }," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+            {" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+              name: ENABLE_WAL_PATH_COMPAT," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+              value: true" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+            }," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+            {" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+              name: empty," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+              mountPath: /opt/empty" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-      restartPolicy: Always," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-      dnsPolicy: ClusterFirst," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-      serviceAccount: postgres-pod," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-      }," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-      schedulerName: default-scheduler" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+      }" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-      kind: PersistentVolumeClaim," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-      apiVersion: v1," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-      status: {" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-        phase: Pending" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-      }" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+      status: {}" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-  }," cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-  revisionHistoryLimit: 10" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+  }" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="metadata.annotation are different" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="-  zalando-postgres-operator-rolling-update-required: false" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="+  zalando-postgres-operator-rolling-update-required: true" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=info msg="reason: new statefulset containers's postgres (index 0) environment does not match the current one" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="updating statefulset" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="patching statefulset annotations" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="patching statefulset annotations" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="calling Patroni API on a pod postgres-test/acid-test-cluster-0 to set the following Postgres options: map[max_connections:300]" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="making PATCH http request: http://10.244.1.118:8008/config" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=warning msg="could not patch postgres parameters with a pod postgres-test/acid-test-cluster-0: patroni returned '<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\n        \"http://www.w3.org/TR/html4/strict.dtd\">\n<html>\n    <head>\n        <meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\">\n        <title>Error response</title>\n    </head>\n    <body>\n        <h1>Error response</h1>\n        <p>Error code: 503</p>\n        <p>Message: Service Unavailable.</p>\n        <p>Error code explanation: 503 - The server cannot process the request due to a high load.</p>\n    </body>\n</html>\n'" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="calling Patroni API on a pod postgres-test/acid-test-cluster-1 to set the following Postgres options: map[max_connections:300]" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="making PATCH http request: http://10.244.3.43:8008/config" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=warning msg="could not patch postgres parameters with a pod postgres-test/acid-test-cluster-1: patroni returned '<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\n        \"http://www.w3.org/TR/html4/strict.dtd\">\n<html>\n    <head>\n        <meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\">\n        <title>Error response</title>\n    </head>\n    <body>\n        <h1>Error response</h1>\n        <p>Error code: 503</p>\n        <p>Message: Service Unavailable.</p>\n        <p>Error code explanation: 503 - The server cannot process the request due to a high load.</p>\n    </body>\n</html>\n'" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="calling Patroni API on a pod postgres-test/acid-test-cluster-2 to set the following Postgres options: map[max_connections:300]" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=debug msg="making PATCH http request: http://10.244.2.112:8008/config" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=warning msg="could not patch postgres parameters with a pod postgres-test/acid-test-cluster-2: patroni returned '<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\n        \"http://www.w3.org/TR/html4/strict.dtd\">\n<html>\n    <head>\n        <meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\">\n        <title>Error response</title>\n    </head>\n    <body>\n        <h1>Error response</h1>\n        <p>Error code: 503</p>\n        <p>Message: Service Unavailable.</p>\n        <p>Error code explanation: 503 - The server cannot process the request due to a high load.</p>\n    </body>\n</html>\n'" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=warning msg="error while syncing cluster state: could not sync statefulsets: could not set cluster-wide PostgreSQL configuration options: could not reach Patroni API to set Postgres options: failed on every pod (3 total)" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=error msg="could not sync cluster: could not sync statefulsets: could not set cluster-wide PostgreSQL configuration options: could not reach Patroni API to set Postgres options: failed on every pod (3 total)" cluster-name=postgres-test/acid-test-cluster pkg=controller worker=1
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:02Z" level=info msg="Recieved add event for already existing Postgres cluster" cluster-name=postgres-test/acid-test-cluster pkg=controller worker=1
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:43Z" level=info msg="pod \"default/acid-postgres-cluster-1\" has been recreated" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:43Z" level=debug msg="unsubscribing from pod \"default/acid-postgres-cluster-1\" events" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:05:43Z" level=debug msg="subscribing to pod \"default/acid-postgres-cluster-2\"" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:22Z" level=info msg="pod \"default/acid-postgres-cluster-2\" has been recreated" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:22Z" level=debug msg="unsubscribing from pod \"default/acid-postgres-cluster-2\" events" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:22Z" level=debug msg="switching over from \"acid-postgres-cluster-0\" to \"default/acid-postgres-cluster-1\"" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:22Z" level=debug msg="making POST http request: http://10.244.2.12:8008/failover" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:22Z" level=debug msg="subscribing to pod \"default/acid-postgres-cluster-1\"" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:24Z" level=debug msg="successfully switched over from \"acid-postgres-cluster-0\" to \"default/acid-postgres-cluster-1\"" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:25Z" level=debug msg="unsubscribing from pod \"default/acid-postgres-cluster-1\" events" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:25Z" level=info msg="recreating old master pod \"default/acid-postgres-cluster-0\"" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:25Z" level=debug msg="subscribing to pod \"default/acid-postgres-cluster-0\"" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=info msg="ADD event has been queued" cluster-name=default/acid-test-cluster pkg=controller worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=info msg="creating a new Postgres cluster" cluster-name=default/acid-test-cluster pkg=controller worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=warning msg="master is not running, generated master endpoint does not contain any addresses" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=info msg="endpoint \"default/acid-test-cluster\" has been successfully created" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=info msg="master service \"default/acid-test-cluster\" has been successfully created" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=info msg="replica service \"default/acid-test-cluster-repl\" has been successfully created" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=debug msg="resolved a conflict of role \"pgbouncer\" between manifest role and connection pooler role to connection pooler role" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=debug msg="fetching possible additional team members for team \"acid\"" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=debug msg="team API is disabled, only returning 0 members for team \"acid\"" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=info msg="users have been initialized" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=info msg="syncing secrets" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=debug msg="created new secret \"default/pgbouncer.acid-test-cluster.credentials\", uid: \"2789b37a-a667-443a-b858-d23498408ca3\"" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=debug msg="created new secret \"default/postgres.acid-test-cluster.credentials\", uid: \"5d9659e8-3d61-4f15-8150-3cf62fb9c844\"" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=debug msg="created new secret \"default/standby.acid-test-cluster.credentials\", uid: \"81f6ccfd-7f1e-4901-ac94-5aa56f1f5fd5\"" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=info msg="secrets have been successfully created" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=info msg="pod disruption budget \"default/postgres-acid-test-cluster-pdb\" has been successfully created" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=debug msg="[{SCOPE acid-test-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-test-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-test-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 12 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{\"parameters\":{\"log_statement\":\"none\",\"shared_buffers\":\"4GB\",\"tcp_keepalives_count\":\"5\",\"tcp_keepalives_idle\":\"60\",\"tcp_keepalives_interval\":\"60\"},\"pg_hba\":[\"hostssl all all 0.0.0.0/0 md5\",\"host    all all 0.0.0.0/0 md5\",\"hostssl replication postgres 0.0.0.0/0 md5\",\"host replication postgres 0.0.0.0/0 md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{\"ttl\":30,\"loop_wait\":10,\"retry_timeout\":10,\"maximum_lag_on_failover\":33554432,\"postgresql\":{\"parameters\":{\"max_connections\":\"300\"}}}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {ENABLE_WAL_PATH_COMPAT true nil}]" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=info msg="Mount additional volumes: [{Name:empty MountPath:/opt/empty SubPath: TargetContainers:[all] VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=debug msg="created new statefulset \"default/acid-test-cluster\", uid: \"0d027892-ef7f-4b8f-8f0a-5ffe28433557\"" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=info msg="statefulset \"default/acid-test-cluster\" has been successfully created" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:06:27Z" level=info msg="waiting for the cluster being ready" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:00Z" level=debug msg="Waiting for 3 pods to become ready" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=info msg="pod \"default/acid-postgres-cluster-0\" has been recreated" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=debug msg="unsubscribing from pod \"default/acid-postgres-cluster-0\" events" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=info msg="pods have been recreated" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=debug msg="set statefulset's rolling update annotation to false: caller/reason applyRollingUpdateFlag" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=debug msg="patching statefulset annotations" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=debug msg="syncing pod disruption budgets" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=debug msg="syncing roles" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=debug msg="closing database connection" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=debug msg="syncing databases" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=debug msg="closing database connection" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=debug msg="syncing prepared databases with schemas" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=debug msg="syncing connection pooler from (nil, nil) to (nil, nil)" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=debug msg="could not get connection pooler secret pooler.acid-postgres-cluster.credentials: secrets \"pooler.acid-postgres-cluster.credentials\" not found" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=info msg="cluster has been synced" cluster-name=default/acid-postgres-cluster pkg=controller worker=0
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:07:04Z" level=info msg="Recieved add event for already existing Postgres cluster" cluster-name=default/acid-postgres-cluster pkg=controller worker=0
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:16:57Z" level=error msg="failed to create cluster: pod labels error: still failing after 200 retries" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:16:57Z" level=error msg="could not create cluster: pod labels error: still failing after 200 retries" cluster-name=default/acid-test-cluster pkg=controller worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=info msg="SYNC event has been queued" cluster-name=postgres-test/acid-test-cluster pkg=controller worker=1
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=info msg="SYNC event has been queued" cluster-name=default/acid-test-cluster pkg=controller worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=info msg="SYNC event has been queued" cluster-name=default/acid-postgres-cluster pkg=controller worker=0
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=info msg="there are 3 clusters running" pkg=controller
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=info msg="syncing of the cluster started" cluster-name=default/acid-test-cluster pkg=controller worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=info msg="syncing of the cluster started" cluster-name=postgres-test/acid-test-cluster pkg=controller worker=1
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=info msg="syncing of the cluster started" cluster-name=default/acid-postgres-cluster pkg=controller worker=0
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=debug msg="resolved a conflict of role \"pgbouncer\" between manifest role and connection pooler role to connection pooler role" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=debug msg="fetching possible additional team members for team \"acid\"" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=debug msg="team API is disabled, only returning 0 members for team \"acid\"" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=info msg="syncing secrets" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=debug msg="resolved a conflict of role \"pgbouncer\" between manifest role and connection pooler role to connection pooler role" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=debug msg="fetching possible additional team members for team \"acid\"" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=debug msg="team API is disabled, only returning 0 members for team \"acid\"" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=info msg="syncing secrets" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=debug msg="fetching possible additional team members for team \"acid\"" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=debug msg="team API is disabled, only returning 0 members for team \"acid\"" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=info msg="syncing secrets" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=debug msg="secret default/pgbouncer.acid-test-cluster.credentials already exists, fetching its password" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=debug msg="secret default/postgres.acid-postgres-cluster.credentials already exists, fetching its password" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:58Z" level=debug msg="secret postgres-test/pgbouncer.acid-test-cluster.credentials already exists, fetching its password" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:59Z" level=debug msg="secret default/postgres.acid-test-cluster.credentials already exists, fetching its password" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:59Z" level=debug msg="secret default/standby.acid-postgres-cluster.credentials already exists, fetching its password" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:34:59Z" level=debug msg="syncing master service" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:00Z" level=debug msg="secret postgres-test/postgres.acid-test-cluster.credentials already exists, fetching its password" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:00Z" level=debug msg="secret default/standby.acid-test-cluster.credentials already exists, fetching its password" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:00Z" level=debug msg="syncing master service" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:00Z" level=debug msg="syncing replica service" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:01Z" level=debug msg="secret postgres-test/standby.acid-test-cluster.credentials already exists, fetching its password" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:01Z" level=debug msg="syncing master service" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:01Z" level=debug msg="syncing replica service" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:01Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:01Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:02Z" level=info msg="volume claims do not require changes" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:02Z" level=debug msg="syncing statefulsets" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:02Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is true" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:02Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:02Z" level=debug msg="[{SCOPE acid-postgres-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-postgres-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-postgres-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 12 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{\"parameters\":{\"log_statement\":\"none\",\"shared_buffers\":\"4GB\",\"tcp_keepalives_count\":\"5\",\"tcp_keepalives_idle\":\"60\",\"tcp_keepalives_interval\":\"60\"},\"pg_hba\":[\"hostssl all all 0.0.0.0/0 md5\",\"hostssl replication postgres 0.0.0.0/0 md5\",\"host    all all 0.0.0.0/0 md5\",\"host replication postgres 0.0.0.0/0 md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{\"ttl\":30,\"loop_wait\":10,\"retry_timeout\":10,\"maximum_lag_on_failover\":33554432,\"postgresql\":{\"parameters\":{\"max_connections\":\"300\"}}}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {ENABLE_WAL_PATH_COMPAT true nil}]" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:02Z" level=debug msg="set statefulset's rolling update annotation to false: caller/reason from cache" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:02Z" level=debug msg="patching statefulset annotations" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:02Z" level=debug msg="syncing replica service" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:02Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:02Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=info msg="volume claims do not require changes" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="syncing statefulsets" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is false" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="[{SCOPE acid-test-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-test-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-test-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 12 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{\"parameters\":{\"log_statement\":\"none\",\"shared_buffers\":\"4GB\",\"tcp_keepalives_count\":\"5\",\"tcp_keepalives_idle\":\"60\",\"tcp_keepalives_interval\":\"60\"},\"pg_hba\":[\"hostssl all all 0.0.0.0/0 md5\",\"host    all all 0.0.0.0/0 md5\",\"hostssl replication postgres 0.0.0.0/0 md5\",\"host replication postgres 0.0.0.0/0 md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{\"ttl\":30,\"loop_wait\":10,\"retry_timeout\":10,\"maximum_lag_on_failover\":33554432,\"postgresql\":{\"parameters\":{\"max_connections\":\"300\"}}}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {ENABLE_WAL_PATH_COMPAT true nil}]" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=info msg="Mount additional volumes: [{Name:empty MountPath:/opt/empty SubPath: TargetContainers:[all] VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="set statefulset's rolling update annotation to false: caller/reason from cache" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=info msg="statefulset default/acid-test-cluster is not in the desired state and needs to be updated" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-          resources: {}," cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-          imagePullPolicy: Always" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="+          resources: {}" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-      restartPolicy: Always," cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-      dnsPolicy: ClusterFirst," cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-      serviceAccount: postgres-pod," cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-      }," cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-      schedulerName: default-scheduler" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="+      }" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-      kind: PersistentVolumeClaim," cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-      apiVersion: v1," cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-      status: {" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-        phase: Pending" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-      }" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="+      status: {}" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-  }," cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="-  revisionHistoryLimit: 10" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="+  }" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="metadata.annotation are different" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg=-null cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="+{" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="+  zalando-postgres-operator-rolling-update-required: false" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="+}" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=info msg="reason: new statefulset's annotations does not match the current one" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="updating statefulset" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="patching statefulset annotations" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="patching statefulset annotations" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="calling Patroni API on a pod default/acid-postgres-cluster-0 to set the following Postgres options: map[max_connections:300]" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="making PATCH http request: http://10.244.2.113:8008/config" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="syncing pod disruption budgets" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="syncing roles" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="closing database connection" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="syncing databases" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="closing database connection" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="syncing prepared databases with schemas" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="syncing connection pooler from (nil, nil) to (nil, nil)" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="No load balancer created for the replica service" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:03Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="could not get connection pooler secret pooler.acid-postgres-cluster.credentials: secrets \"pooler.acid-postgres-cluster.credentials\" not found" cluster-name=default/acid-postgres-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=info msg="cluster has been synced" cluster-name=default/acid-postgres-cluster pkg=controller worker=0
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=info msg="volume claims do not require changes" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="syncing statefulsets" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is false" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=info msg="found a statefulset with an unfinished rolling update of the pods" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="[{SCOPE acid-test-cluster nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-test-cluster.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY  &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-test-cluster.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 12 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{\"parameters\":{\"log_statement\":\"none\",\"shared_buffers\":\"4GB\",\"tcp_keepalives_count\":\"5\",\"tcp_keepalives_idle\":\"60\",\"tcp_keepalives_interval\":\"60\"},\"pg_hba\":[\"hostssl all all 0.0.0.0/0 md5\",\"host    all all 0.0.0.0/0 md5\",\"hostssl replication postgres 0.0.0.0/0 md5\",\"host replication postgres 0.0.0.0/0 md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{\"ttl\":30,\"loop_wait\":10,\"retry_timeout\":10,\"maximum_lag_on_failover\":33554432,\"postgresql\":{\"parameters\":{\"max_connections\":\"300\"}}}}} nil} {DCS_ENABLE_KUBERNETES_API true nil} {ENABLE_WAL_PATH_COMPAT true nil}]" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=info msg="Mount additional volumes: [{Name:empty MountPath:/opt/empty SubPath: TargetContainers:[all] VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="set statefulset's rolling update annotation to true: caller/reason from cache" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="patching statefulset annotations" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="calling Patroni API on a pod default/acid-test-cluster-0 to set the following Postgres options: map[max_connections:300]" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="making PATCH http request: http://10.244.1.122:8008/config" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=warning msg="could not patch postgres parameters with a pod default/acid-test-cluster-0: patroni returned '<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\n        \"http://www.w3.org/TR/html4/strict.dtd\">\n<html>\n    <head>\n        <meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\">\n        <title>Error response</title>\n    </head>\n    <body>\n        <h1>Error response</h1>\n        <p>Error code: 503</p>\n        <p>Message: Service Unavailable.</p>\n        <p>Error code explanation: 503 - The server cannot process the request due to a high load.</p>\n    </body>\n</html>\n'" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="calling Patroni API on a pod default/acid-test-cluster-1 to set the following Postgres options: map[max_connections:300]" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="making PATCH http request: http://10.244.3.45:8008/config" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=warning msg="could not patch postgres parameters with a pod default/acid-test-cluster-1: patroni returned '<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\n        \"http://www.w3.org/TR/html4/strict.dtd\">\n<html>\n    <head>\n        <meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\">\n        <title>Error response</title>\n    </head>\n    <body>\n        <h1>Error response</h1>\n        <p>Error code: 503</p>\n        <p>Message: Service Unavailable.</p>\n        <p>Error code explanation: 503 - The server cannot process the request due to a high load.</p>\n    </body>\n</html>\n'" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="calling Patroni API on a pod default/acid-test-cluster-2 to set the following Postgres options: map[max_connections:300]" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="making PATCH http request: http://10.244.2.114:8008/config" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=warning msg="could not patch postgres parameters with a pod default/acid-test-cluster-2: patroni returned '<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\n        \"http://www.w3.org/TR/html4/strict.dtd\">\n<html>\n    <head>\n        <meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\">\n        <title>Error response</title>\n    </head>\n    <body>\n        <h1>Error response</h1>\n        <p>Error code: 503</p>\n        <p>Message: Service Unavailable.</p>\n        <p>Error code explanation: 503 - The server cannot process the request due to a high load.</p>\n    </body>\n</html>\n'" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=warning msg="error while syncing cluster state: could not sync statefulsets: could not set cluster-wide PostgreSQL configuration options: could not reach Patroni API to set Postgres options: failed on every pod (3 total)" cluster-name=default/acid-test-cluster pkg=cluster worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=error msg="could not sync cluster: could not sync statefulsets: could not set cluster-wide PostgreSQL configuration options: could not reach Patroni API to set Postgres options: failed on every pod (3 total)" cluster-name=default/acid-test-cluster pkg=controller worker=2
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="calling Patroni API on a pod postgres-test/acid-test-cluster-0 to set the following Postgres options: map[max_connections:300]" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="making PATCH http request: http://10.244.1.118:8008/config" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=warning msg="could not patch postgres parameters with a pod postgres-test/acid-test-cluster-0: patroni returned '<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\n        \"http://www.w3.org/TR/html4/strict.dtd\">\n<html>\n    <head>\n        <meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\">\n        <title>Error response</title>\n    </head>\n    <body>\n        <h1>Error response</h1>\n        <p>Error code: 503</p>\n        <p>Message: Service Unavailable.</p>\n        <p>Error code explanation: 503 - The server cannot process the request due to a high load.</p>\n    </body>\n</html>\n'" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="calling Patroni API on a pod postgres-test/acid-test-cluster-1 to set the following Postgres options: map[max_connections:300]" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="making PATCH http request: http://10.244.3.43:8008/config" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=warning msg="could not patch postgres parameters with a pod postgres-test/acid-test-cluster-1: patroni returned '<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\n        \"http://www.w3.org/TR/html4/strict.dtd\">\n<html>\n    <head>\n        <meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\">\n        <title>Error response</title>\n    </head>\n    <body>\n        <h1>Error response</h1>\n        <p>Error code: 503</p>\n        <p>Message: Service Unavailable.</p>\n        <p>Error code explanation: 503 - The server cannot process the request due to a high load.</p>\n    </body>\n</html>\n'" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="calling Patroni API on a pod postgres-test/acid-test-cluster-2 to set the following Postgres options: map[max_connections:300]" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=debug msg="making PATCH http request: http://10.244.2.112:8008/config" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=warning msg="could not patch postgres parameters with a pod postgres-test/acid-test-cluster-2: patroni returned '<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\n        \"http://www.w3.org/TR/html4/strict.dtd\">\n<html>\n    <head>\n        <meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\">\n        <title>Error response</title>\n    </head>\n    <body>\n        <h1>Error response</h1>\n        <p>Error code: 503</p>\n        <p>Message: Service Unavailable.</p>\n        <p>Error code explanation: 503 - The server cannot process the request due to a high load.</p>\n    </body>\n</html>\n'" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=warning msg="error while syncing cluster state: could not sync statefulsets: could not set cluster-wide PostgreSQL configuration options: could not reach Patroni API to set Postgres options: failed on every pod (3 total)" cluster-name=postgres-test/acid-test-cluster pkg=cluster
postgres-operator-65dcbf585b-nh2bs postgres-operator time="2021-01-22T07:35:04Z" level=error msg="could not sync cluster: could not sync statefulsets: could not set cluster-wide PostgreSQL configuration options: could not reach Patroni API to set Postgres options: failed on every pod (3 total)" cluster-name=postgres-test/acid-test-cluster pkg=controller worker=1

[FxKu]

I've already commented on the pg_hba change here. It's true that it's not additive anymore - an oversight we should have mentioned in the release notes. The error messages at the end of you pasted log snippet look strange though.

The conflict message on the users can be explained like this: The operator will create a pooler user which you've specified in the connectionPooler section. Internally the pooler role will be more like a system role and not like an app user you list in the users part. Actually, you don't have to put pgbouncer role there. The operator will interpret it as two different roles first, only to find out that there's a name conflict.

[haroonb]

edit: I found out, that I can change the user to a custom user if I change the user part in the connectionPooler, but I doesn't accept a array, meaning multiple users. And even if I login as a admin user e.g. 'pgbouncer' I still can't connect to another db, because of the ssl error.

Hello @FxKu, thanks for the reply. I think I understand this implementation of pgbouncer a bit better now.

I'm running into the issue, that I can't use the pgbouncer user I created with the postgres-cluster to access the database through the pooler, since it is not added in the auth_file.txt. How can I add a users to the 'auth_file.txt'.

@devops-m1:~$ kubectl exec -it acid-test-cluster-pooler-b47588474-nfqr9 -- cat /etc/pgbouncer/auth_file.txt
"pooler" "FpLIEOjwTvoUrZGpzGAMSB8d3cEmho31cxxduyxheeEvRa0fGB1V396jeGY4OsZy"

If I try to use the pgbouncer user to login, I get a error, that ssl is required.

devops-m1:~$ psql postgres://pgbouncer@10.107.138.153/pgbouncer
psql: ERROR:  not allowed
ERROR:  SSL required

acid-test-cluster-pooler-b47588474-nmnrsacid-test-cluster-pooler-b47588474-twpbg  connection-pooler 2021-01-25 09:43:15.835 UTC [1] WARNING C-0x5605de073010: pgbouncer/pgbouncer@10.244.0.0:58320 pooler error: not allowed
connection-pooler 2021-01-25 09:43:15.836 UTC [1] WARNING C-0x5629a9f9a010: (nodb)/(nouser)@10.244.0.0:22907 pooler error: SSL required

I definetly know, that the pgbouncer user exists.

devops-m1:~$ kubectl exec -it acid-test-cluster-0 -- psql -U postgres
psql (13.1 (Ubuntu 13.1-1.pgdg18.04+1), server 12.5 (Ubuntu 12.5-1.pgdg18.04+1))
Type "help" for help.

postgres=# \du
                                     List of roles
 Role name  |                         Attributes                         |  Member of
------------+------------------------------------------------------------+-------------
 admin      | Create DB, Cannot login                                    | {pgbouncer}
 pgbouncer  | Superuser, Create DB                                       | {}
 pooler     |                                                            | {}
 postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
 robot_zmon | Cannot login                                               | {}
 standby    | Replication                                                | {}
 zalandos   | Create DB, Cannot login                                    | {}

postgres=#

Is there a possibility to customize the pgbouncer.ini file to suite our needs? The ssl requirement seems hardcoded at first glance.

devops-m1:~$ kubectl exec -it acid-test-cluster-pooler-b47588474-nfqr9 -- cat /etc/pgbouncer/pgbouncer.ini
# vim: set ft=dosini:

[databases]
* = host=acid-test-cluster port=5432 auth_user=pooler

[pgbouncer]
pool_mode = transaction
listen_port = 5432
listen_addr = *
auth_type = plain
auth_file = /etc/pgbouncer/auth_file.txt
admin_users = pooler
auth_query = SELECT * FROM pooler.user_lookup($1)
logfile = /var/log/pgbouncer/pgbouncer.log
pidfile = /var/run/pgbouncer/pgbouncer.pid

server_tls_sslmode = require
server_tls_ca_file = /etc/ssl/certs/pgbouncer.crt
server_tls_protocols = secure
client_tls_sslmode = require
client_tls_key_file = /etc/ssl/certs/pgbouncer.key
client_tls_cert_file = /etc/ssl/certs/pgbouncer.crt

log_connections = 0
log_disconnections = 0

# How many server connections to allow per user/database pair.
default_pool_size = 50

# Add more server connections to pool if below this number. Improves behavior
# when usual load comes suddenly back after period of total inactivity.
min_pool_size = 25

# How many additional connections to allow to a pool
reserve_pool_size = 25

# Maximum number of client connections allowed.
max_client_conn = 10000

# Do not allow more than this many connections per database (regardless of
# pool, i.e. user)
max_db_connections = 100

# If a client has been in "idle in transaction" state longer, it will be
# disconnected. [seconds]
idle_transaction_timeout = 600

# If login failed, because of failure from connect() or authentication that
# pooler waits this much before retrying to connect. Default is 15. [seconds]
server_login_retry = 5

# To ignore extra parameter in startup packet. By default only 'database' and
# 'user' are allowed, all others raise error.  This is needed to tolerate
# overenthusiastic JDBC wanting to unconditionally set 'extra_float_digits=2'
# in startup packet.
ignore_startup_parameters = extra_float_digits,options

143     - name: create postgres cluster
142       k8s:
141         state: present
140         definition:
139           apiVersion: "acid.zalan.do/v1"
138           kind: postgresql
137           metadata:
136             name: acid-test-cluster
135             namespace: default
134           spec:
133             dockerImage: registry.opensource.zalan.do/acid/spilo-13:2.0-p2
132             teamId: "acid"
131             numberOfInstances: 3 # 2
130             users:  # Application/Robot users
129               pgbouncer:
128               - superuser
127               - createdb
126             enableLogicalBackup: false
125             enableMasterLoadBalancer: false
124             enableReplicaLoadBalancer: false
123             enableConnectionPooler: true # enable/disable connection pooler deployment
122             enableReplicaConnectionPooler: false # set to enable connectionPooler for replica service
121             allowedSourceRanges:  # load balancers' source ranges for both master and replica services
120             - 127.0.0.1/32
119 #            databases:
118 #              foo: zalando
117 #            preparedDatabases:
116 #              bar:
115 #                defaultUsers: true
114 #                extensions:
113 #                  pg_partman: public
112 #                  pgcrypto: public
111 #                schemas:
110 #                  data: {}
109 #                  history:
108 #                    defaultRoles: true
107 #                    defaultUsers: false
106             postgresql:
105               version: "{{ postgres_version }}"
104               parameters:  # Expert section
103                 tcp_keepalives_count: "5"
102                 tcp_keepalives_interval: "60"
101                 tcp_keepalives_idle: "60"
100                 shared_buffers: "4GB"
 99                 max_connections: "300"
 98                 log_statement: "none"
 97             volume:
 96               size: "{{ postgres_pvsize }}G"
 95               storageClass: rook-ceph-block
 94             additionalVolumes:
 93               - name: empty
 92                 mountPath: /opt/empty
 91                 targetContainers:
 90                   - all
 89                 volumeSource:
 88                   emptyDir: {}
 87           #    - name: data
 86           #      mountPath: /home/postgres/pgdata/partitions
 85           #      targetContainers:
 84           #        - postgres
 83           #      volumeSource:
 82           #        PersistentVolumeClaim:
 81           #          claimName: pvc-postgresql-data-partitions
 80           #          readyOnly: false
 79           #    - name: conf
 78           #      mountPath: /etc/telegraf
 77           #      subPath: telegraf.conf
 76           #      targetContainers:
 75           #        - telegraf-sidecar
 74           #      volumeSource:
 73           #        configMap:
 72           #          name: my-config-map
 71
 70             enableShmVolume: true
 69           #  spiloRunAsUser: 101
 68           #  spiloRunAsGroup: 103
 67           #  spiloFSGroup: 103
 66             podAnnotations:
 65               backup.velero.io/backup-volumes: pgdata
 64           #  serviceAnnotations:
 63           #    annotation.key: value
 62           #  podPriorityClassName: "spilo-pod-priority"
 61           #  tolerations:
 60           #  - key: postgres
 59           #    operator: Exists
 58           #    effect: NoSchedule
 57             resources:
 56               requests:
 55                 cpu: 10m
 54                 memory: 100Mi
 53               limits:
 52                 cpu: 5000m # 500m
 51                 memory: 5000Mi # 500Mi
 50             patroni:
 49               initdb:
 48                 encoding: "UTF8"
 47                 locale: "en_US.UTF-8"
 46                 data-checksums: "true"
 45               pg_hba:
 44                 - hostssl all all 0.0.0.0/0 md5
 43                 - host    all all 0.0.0.0/0 md5
 42                 - hostssl replication postgres 0.0.0.0/0 md5
 41                 - host replication postgres 0.0.0.0/0 md5
 40                 - local   all             all                                   trust
 39                 - host    all             all                127.0.0.1/32       md5
 38                 - host    all             all                ::1/128            md5
 37                 - hostssl replication     standby all                md5
 36                 - hostnossl all           all                all                reject
 35                 - hostssl all             all                all                md5
 34           #    slots:
 33           #      permanent_physical_1:
 32           #        type: physical
 31           #      permanent_logical_1:
 30           #        type: logical
 29           #        database: foo
 28           #        plugin: pgoutput
 27               ttl: 30
 26               loop_wait: &loop_wait 10
 25               retry_timeout: 10
 24               synchronous_mode: false
 23               synchronous_mode_strict: false
 22               maximum_lag_on_failover: 33554432
 21
 20           # restore a Postgres DB with point-in-time-recovery
 19           # with a non-empty timestamp, clone from an S3 bucket using the latest backup before the timestamp
 18           # with an empty/absent timestamp, clone from an existing alive cluster using pg_basebackup
 17           #  clone:
 16           #    uid: "efd12e58-5786-11e8-b5a7-06148230260c"
 15           #    cluster: "acid-batman"
 14           #    timestamp: "2017-12-19T12:40:33+01:00"  # timezone required (offset relative to UTC, see RFC 3339 section 5.6)
 13           #    s3_wal_path: "s3://custom/path/to/bucket"
 12
 11           # run periodic backups with k8s cron jobs
 10           #  enableLogicalBackup: true
  9           #  logicalBackupSchedule: "30 00 * * *"
  8
  7           #  maintenanceWindows:
  6           #  - 01:00-06:00  #UTC
  5           #  - Sat:00:00-04:00
  4
  3           # overwrite custom properties for connection pooler deployments
  2             connectionPooler:
  1               numberOfInstances: 3
180               mode: "transaction"
  1               schema: "pooler"
  2               user: "pooler"
  3               maxDBConnections: 300 # custom
  4               resources:
  5                 requests:
  6                   cpu: 300m
  7                   memory: 100Mi
  8                 limits:
  9                   cpu: "1"
 10                   memory: 100Mi
 11
 12             initContainers:
 13             - name: date
 14               image: busybox
 15               command: [ "/bin/date" ]
 16           #  sidecars:
 17           #    - name: "telegraf-sidecar"
 18           #      image: "telegraf:latest"
 19           #      resources:
 20           #        limits:
 21           #          cpu: 500m
 22           #          memory: 500Mi
 23           #        requests:
 24           #          cpu: 100m
 25           #          memory: 100Mi
 26           #      env:
 27           #        - name: "USEFUL_VAR"
 28           #          value: "perhaps-true"
 29
 30           # Custom TLS certificate. Disabled unless tls.secretName has a value.
 31             tls:
 32               secretName: ""  # should correspond to a Kubernetes Secret resource to load
 33               certificateFile: "tls.crt"
 34               privateKeyFile: "tls.key"
 35               caFile: ""  # optionally configure Postgres with a CA certificate
 36               caSecretName: "" # optionally the ca.crt can come from this secret instead.
 37           # file names can be also defined with absolute path, and will no longer be relative
 38           # to the "/tls/" path where the secret is being mounted by default, and "/tlsca/"
 39           # where the caSecret is mounted by default.
 40           # When TLS is enabled, also set spiloFSGroup parameter above to the relevant value.
 41           # if unknown, set it to 103 which is the usual value in the default spilo images.
 42           # In Openshift, there is no need to set spiloFSGroup/spilo_fsgroup.
 43
 44           # Add node affinity support by allowing postgres pods to schedule only on nodes that
 45           # have label: "postgres-operator:enabled" set.
 46           #  nodeAffinity:
 47           #    requiredDuringSchedulingIgnoredDuringExecution:
 48           #      nodeSelectorTerms:
 49           #        - matchExpressions:
 50           #            - key: postgres-operator
 51           #              operator: In
 52           #              values:
 53           #                - enabled
~

Thank you for the clarification and assistance.