Hi,
I have a problem with pod after fail node on which master pod run, I configure operator to reboot quicker when fail node when is the operator container, but still don't help with db pod.
operator config:
apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
name: teamId-pg96
spec:
enableMasterLoadBalancer: true
enableReplicaLoadBalancer: false
teamId: "teamId"
volume:
size: 15Gi
numberOfInstances: 2
metrics:
enabled: false
users:
user:
- createdb
# role for application foo
foo_user: []
patroni:
pg_hba:
- local all all trust
- hostssl all +zalandos 127.0.0.1/32 pam
- host all all 127.0.0.1/32 md5
- hostssl all +zalandos ::1/128 pam
- host all all ::1/128 md5
- hostssl replication standby all md5
- hostnossl all all all md5
- hostssl all +zalandos all pam
- hostssl all all all md5
#databases: name->owner
databases:
main: user
postgresql:
version: "9.6"
after 20 minutes on failure:
NAME READY STATUS RESTARTS AGE
postgres-operator-76f5c5cb58-7g42w 1/1 Terminating 0 67m
postgres-operator-76f5c5cb58-ws47s 1/1 Running 0 26m
user-pg96-0 1/1 Terminating 0 48m
Name: user-pg96-0
Namespace: db
Priority: 0
Node: node1/10.4.18.118
Start Time: Thu, 10 Oct 2019 10:24:50 +0200
Labels: application=spilo
controller-revision-hash=user-pg96-bf85f74b
spilo-role=master
statefulset.kubernetes.io/pod-name=tupa1-pg96-0
team=tupa1
version=user-pg96
Annotations: kubernetes.io/psp: 00-pharos-privileged
status:
{"conn_url":"postgres://10.46.0.9:5432/postgres","api_url":"http://10.46.0.9:8008/patroni","state":"running","role":"master","version":"1....
Status: Terminating (lasts 22m)
Termination Grace Period: 300s
IP: 10.46.0.9
IPs: <none>
Controlled By: StatefulSet/user-pg96
Containers:
postgres:
Container ID: docker://4b6b8af82c69b591e0480ff2c186ba5c748dd28962d79cb85c5d9d0f0da5f23d
Image: registry.opensource.zalan.do/acid/spilo-11:1.5-p9
Image ID: docker-pullable://registry.opensource.zalan.do/acid/spilo-11@sha256:dd39c9581a56e44f80ae55fc86b7a3eb9735e5c799e056fa6cc77b63e84df307
Ports: 8008/TCP, 5432/TCP, 8080/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
State: Running
Started: Thu, 10 Oct 2019 10:27:11 +0200
Ready: True
Restart Count: 0
Limits:
cpu: 3
memory: 1Gi
Requests:
cpu: 100m
memory: 100Mi
Environment:
SCOPE: user-pg96
PGROOT: /home/postgres/pgdata/pgroot
POD_IP: (v1:status.podIP)
POD_NAMESPACE: db (v1:metadata.namespace)
PGUSER_SUPERUSER: postgres
KUBERNETES_SCOPE_LABEL: version
KUBERNETES_ROLE_LABEL: spilo-role
KUBERNETES_LABELS: application=spilo
PGPASSWORD_SUPERUSER: <set to the key 'password' in secret 'postgres.user-pg96.credentials'> Optional: false
PGUSER_STANDBY: standby
PGPASSWORD_STANDBY: <set to the key 'password' in secret 'standby.user-pg96.credentials'> Optional: false
PAM_OAUTH2: https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees
HUMAN_ROLE: zalandos
SPILO_CONFIGURATION: {"postgresql":{"bin_dir":"/usr/lib/postgresql/9.6/bin","pg_hba":["local all all trust","hostssl all +zalandos 127.0.0.1/32 pam","host all all 127.0.0.1/32 md5","hostssl all +zalandos ::1/128 pam","host all all ::1/128 md5","hostssl replication standby all md5","hostnossl all all all md5","hostssl all +zalandos all pam","hostssl all all all md5"]},"bootstrap":{"initdb":[{"auth-host":"md5"},{"auth-local":"trust"}],"users":{"zalandos":{"password":"","options":["CREATEDB","NOLOGIN"]}},"dcs":{}}}
DCS_ENABLE_KUBERNETES_API: true
Mounts:
/dev/shm from dshm (rw)
/home/postgres/pgdata from pgdata (rw)
/var/run/secrets/kubernetes.io/serviceaccount from zalando-postgres-operator-token-n26xw (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady True
PodScheduled True
Volumes:
pgdata:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pgdata-user-pg96-0
ReadOnly: false
dshm:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
zalando-postgres-operator-token-n26xw:
Type: Secret (a volume populated by a Secret)
SecretName: zalando-postgres-operator-token-n26xw
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 49m default-scheduler Successfully assigned db/user-pg96-0 to node1
Normal Pulling 49m kubelet, node1 Pulling image "registry.opensource.zalan.do/acid/spilo-11:1.5-p9"
Normal Pulled 46m kubelet, node1 Successfully pulled image "registry.opensource.zalan.do/acid/spilo-11:1.5-p9"
Normal Created 46m kubelet, node1 Created container postgres
Normal Started 46m kubelet, node1 Started container postgres
operator-log:
2019/10/10 08:47:14 Fully qualified configmap name: db/postgres-operator
2019/10/10 08:47:14 Spilo operator v1.2.0
time="2019-10-10T08:47:16Z" level=warning msg="in the operator config map, the pod service account name zalando-postgres-operator does not match the name operator given in the account definition; using the former for consistency" pkg=controller
time="2019-10-10T08:47:16Z" level=info msg="Parse role bindings" pkg=controller
time="2019-10-10T08:47:16Z" level=info msg="successfully parsed" pkg=controller
time="2019-10-10T08:47:16Z" level=info msg="Listening to all namespaces" pkg=controller
time="2019-10-10T08:47:16Z" level=info msg="customResourceDefinition \"postgresqls.acid.zalan.do\" is already registered and will only be updated" pkg=controller
time="2019-10-10T08:47:20Z" level=warning msg="in the operator config map, the pod service account name zalando-postgres-operator does not match the name operator given in the account definition; using the former for consistency" pkg=controller
time="2019-10-10T08:47:20Z" level=info msg="config: {\n\t\"ReadyWaitInterval\": 3000000000,\n\t\"ReadyWaitTimeout\": 30000000000,\n\t\"ResyncPeriod\": 300000000000,\n\t\"RepairPeriod\": 300000000000,\n\t\"ResourceCheckInterval\": 3000000000,\n\t\"ResourceCheckTimeout\": 600000000000,\n\t\"PodLabelWaitTimeout\": 600000000000,\n\t\"PodDeletionWaitTimeout\": 600000000000,\n\t\"SpiloFSGroup\": null,\n\t\"PodPriorityClassName\": \"\",\n\t\"ClusterDomain\": \"cluster.local\",\n\t\"SpiloPrivileged\": false,\n\t\"ClusterLabels\": {\n\t\t\"application\": \"spilo\"\n\t},\n\t\"InheritedLabels\": null,\n\t\"ClusterNameLabel\": \"version\",\n\t\"PodRoleLabel\": \"spilo-role\",\n\t\"PodToleration\": null,\n\t\"DefaultCPURequest\": \"100m\",\n\t\"DefaultMemoryRequest\": \"100Mi\",\n\t\"DefaultCPULimit\": \"3\",\n\t\"DefaultMemoryLimit\": \"1Gi\",\n\t\"PodEnvironmentConfigMap\": \"\",\n\t\"NodeReadinessLabel\": null,\n\t\"MaxInstances\": -1,\n\t\"MinInstances\": -1,\n\t\"ShmVolume\": true,\n\t\"SecretNameTemplate\": \"{username}.{cluster}.credentials\",\n\t\"PamRoleName\": \"zalandos\",\n\t\"PamConfiguration\": \"https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees\",\n\t\"TeamsAPIUrl\": \"https://teams.example.com/api/\",\n\t\"OAuthTokenSecretName\": \"db/postgresql-operator\",\n\t\"InfrastructureRolesSecretName\": \"/\",\n\t\"SuperUsername\": \"postgres\",\n\t\"ReplicationUsername\": \"standby\",\n\t\"ScalyrAPIKey\": \"\",\n\t\"ScalyrImage\": \"\",\n\t\"ScalyrServerURL\": \"https://upload.eu.scalyr.com\",\n\t\"ScalyrCPURequest\": \"100m\",\n\t\"ScalyrMemoryRequest\": \"50Mi\",\n\t\"ScalyrCPULimit\": \"1\",\n\t\"ScalyrMemoryLimit\": \"1Gi\",\n\t\"LogicalBackupSchedule\": \"30 00 * * *\",\n\t\"LogicalBackupDockerImage\": \"registry.opensource.zalan.do/acid/logical-backup\",\n\t\"LogicalBackupS3Bucket\": \"\",\n\t\"WatchedNamespace\": \"\",\n\t\"EtcdHost\": \"\",\n\t\"DockerImage\": \"registry.opensource.zalan.do/acid/spilo-11:1.5-p9\",\n\t\"Sidecars\": null,\n\t\"PodServiceAccountName\": \"zalando-postgres-operator\",\n\t\"PodServiceAccountDefinition\": \"\\n\\t\\t{ \\\"apiVersion\\\": \\\"v1\\\",\\n\\t\\t \\\"kind\\\": \\\"ServiceAccount\\\",\\n\\t\\t \\\"metadata\\\": {\\n\\t\\t\\t\\t \\\"name\\\": \\\"operator\\\"\\n\\t\\t }\\n\\t\\t}\",\n\t\"PodServiceAccountRoleBindingDefinition\": \"\\n\\t\\t{\\n\\t\\t\\t\\\"apiVersion\\\": \\\"rbac.authorization.k8s.io/v1beta1\\\",\\n\\t\\t\\t\\\"kind\\\": \\\"RoleBinding\\\",\\n\\t\\t\\t\\\"metadata\\\": {\\n\\t\\t\\t\\t \\\"name\\\": \\\"zalando-postgres-operator\\\"\\n\\t\\t\\t},\\n\\t\\t\\t\\\"roleRef\\\": {\\n\\t\\t\\t\\t\\\"apiGroup\\\": \\\"rbac.authorization.k8s.io\\\",\\n\\t\\t\\t\\t\\\"kind\\\": \\\"ClusterRole\\\",\\n\\t\\t\\t\\t\\\"name\\\": \\\"zalando-postgres-operator\\\"\\n\\t\\t\\t},\\n\\t\\t\\t\\\"subjects\\\": [\\n\\t\\t\\t\\t{\\n\\t\\t\\t\\t\\t\\\"kind\\\": \\\"ServiceAccount\\\",\\n\\t\\t\\t\\t\\t\\\"name\\\": \\\"zalando-postgres-operator\\\"\\n\\t\\t\\t\\t}\\n\\t\\t\\t]\\n\\t\\t}\",\n\t\"MasterPodMoveTimeout\": 1200000000000,\n\t\"DbHostedZone\": \"db.example.com\",\n\t\"AWSRegion\": \"eu-central-1\",\n\t\"WALES3Bucket\": \"\",\n\t\"LogS3Bucket\": \"\",\n\t\"KubeIAMRole\": \"\",\n\t\"AdditionalSecretMount\": \"\",\n\t\"AdditionalSecretMountPath\": \"/meta/credentials\",\n\t\"DebugLogging\": true,\n\t\"EnableDBAccess\": true,\n\t\"EnableTeamsAPI\": false,\n\t\"EnableTeamSuperuser\": false,\n\t\"TeamAdminRole\": \"admin\",\n\t\"EnableAdminRoleForUsers\": true,\n\t\"EnableMasterLoadBalancer\": true,\n\t\"EnableReplicaLoadBalancer\": false,\n\t\"CustomServiceAnnotations\": null,\n\t\"EnablePodAntiAffinity\": false,\n\t\"PodAntiAffinityTopologyKey\": \"kubernetes.io/hostname\",\n\t\"EnableLoadBalancer\": null,\n\t\"MasterDNSNameFormat\": \"{cluster}.{team}.staging.{hostedzone}\",\n\t\"ReplicaDNSNameFormat\": \"{cluster}-repl.{team}.staging.{hostedzone}\",\n\t\"PDBNameFormat\": \"postgres-{cluster}-pdb\",\n\t\"EnablePodDisruptionBudget\": true,\n\t\"Workers\": 4,\n\t\"APIPort\": 8080,\n\t\"RingLogLines\": 100,\n\t\"ClusterHistoryEntries\": 1000,\n\t\"TeamAPIRoleConfiguration\": {\n\t\t\"log_statement\": \"all\"\n\t},\n\t\"PodTerminateGracePeriod\": 300000000000,\n\t\"PodManagementPolicy\": \"ordered_ready\",\n\t\"ProtectedRoles\": [\n\t\t\"admin\"\n\t],\n\t\"PostgresSuperuserTeams\": null,\n\t\"SetMemoryRequestToLimit\": false\n}" pkg=controller
time="2019-10-10T08:47:20Z" level=debug msg="acquiring initial list of clusters" pkg=controller
time="2019-10-10T08:47:20Z" level=debug msg="added new cluster: \"db/user-pg96\"" pkg=controller
time="2019-10-10T08:47:20Z" level=info msg="\"SYNC\" event has been queued" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T08:47:20Z" level=info msg="there are 1 clusters running" pkg=controller
time="2019-10-10T08:47:20Z" level=info msg="started working in background" pkg=controller
time="2019-10-10T08:47:20Z" level=info msg="listening on :8080" pkg=apiserver
time="2019-10-10T08:47:20Z" level=info msg="\"ADD\" event has been queued" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T08:47:20Z" level=debug msg="new node has been added: \"/node1\" ()" pkg=controller
time="2019-10-10T08:47:20Z" level=debug msg="new node has been added: \"/node2\" ()" pkg=controller
time="2019-10-10T08:47:20Z" level=debug msg="new node has been added: \"/node3\" ()" pkg=controller
time="2019-10-10T08:47:20Z" level=info msg="syncing of the cluster started" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T08:47:20Z" level=debug msg="team API is disabled, returning empty list of members for team \"user\"" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:20Z" level=debug msg="syncing secrets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:20Z" level=debug msg="secret \"db/standby.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:20Z" level=debug msg="secret \"db/foo-user.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:20Z" level=debug msg="secret \"db/kraken.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:20Z" level=debug msg="secret \"db/taras.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:20Z" level=debug msg="secret \"db/postgres.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:20Z" level=debug msg="syncing services" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:20Z" level=debug msg="syncing master service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:20Z" level=info msg="could not find the cluster's master endpoint" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:21Z" level=info msg="created missing master endpoint \"db/user-pg96\"" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:21Z" level=info msg="could not find the cluster's master service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:22Z" level=info msg="created missing master service \"db/user-pg96\"" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:22Z" level=debug msg="syncing replica service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:22Z" level=debug msg="No load balancer created for the replica service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:22Z" level=debug msg="syncing persistent volumes" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:22Z" level=debug msg="skipping persistent volume \"pgdata-user-pg96-1\" corresponding to a non-running pods" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:22Z" level=debug msg="syncing statefulsets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:22Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE user-pg96 nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL version nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {KUBERNETES_LABELS application=spilo nil} {PGPASSWORD_SUPERUSER &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.user-pg96.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.user-pg96.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/9.6/bin\",\"pg_hba\":[\"local all all trust\",\"hostssl all +zalandos 127.0.0.1/32 pam\",\"host all all 127.0.0.1/32 md5\",\"hostssl all +zalandos ::1/128 pam\",\"host all all ::1/128 md5\",\"hostssl replication standby all md5\",\"hostnossl all all all md5\",\"hostssl all +zalandos all pam\",\"hostssl all all all md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:22Z" level=debug msg="statefulset's rolling update annotation has been set to false" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:22Z" level=debug msg="syncing pod disruption budgets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:22Z" level=debug msg="syncing roles" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:23Z" level=error msg="could not connect to PostgreSQL database: dial tcp: lookup user-pg96.db.svc.cluster.local on 169.254.20.10:53: no such host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:40Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:47:55Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:48:10Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:48:25Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:48:40Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:48:55Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:49:10Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:49:10Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:49:10Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T08:49:10Z" level=debug msg="cluster already exists" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T08:52:20Z" level=info msg="\"SYNC\" event has been queued" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T08:52:20Z" level=info msg="there are 1 clusters running" pkg=controller
time="2019-10-10T08:52:20Z" level=info msg="syncing of the cluster started" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T08:52:20Z" level=debug msg="team API is disabled, returning empty list of members for team \"user\"" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:20Z" level=debug msg="syncing secrets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:20Z" level=debug msg="secret \"db/postgres.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:20Z" level=debug msg="secret \"db/standby.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:20Z" level=debug msg="secret \"db/kraken.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:20Z" level=debug msg="secret \"db/taras.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:20Z" level=debug msg="secret \"db/foo-user.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:20Z" level=debug msg="syncing services" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:20Z" level=debug msg="syncing master service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:20Z" level=debug msg="syncing replica service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:20Z" level=debug msg="No load balancer created for the replica service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:20Z" level=debug msg="syncing persistent volumes" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:21Z" level=debug msg="skipping persistent volume \"pgdata-user-pg96-1\" corresponding to a non-running pods" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:21Z" level=debug msg="syncing statefulsets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:21Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is true" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:21Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE user-pg96 nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL version nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {KUBERNETES_LABELS application=spilo nil} {PGPASSWORD_SUPERUSER &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.user-pg96.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.user-pg96.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/9.6/bin\",\"pg_hba\":[\"local all all trust\",\"hostssl all +zalandos 127.0.0.1/32 pam\",\"host all all 127.0.0.1/32 md5\",\"hostssl all +zalandos ::1/128 pam\",\"host all all ::1/128 md5\",\"hostssl replication standby all md5\",\"hostnossl all all all md5\",\"hostssl all +zalandos all pam\",\"hostssl all all all md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:21Z" level=debug msg="statefulset's rolling update annotation has been set to false" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:21Z" level=debug msg="syncing pod disruption budgets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:21Z" level=debug msg="syncing roles" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:24Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:39Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:52:54Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:53:09Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:53:24Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:53:39Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:53:54Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:54:09Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:54:09Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:54:09Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T08:57:20Z" level=info msg="\"SYNC\" event has been queued" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T08:57:20Z" level=info msg="there are 1 clusters running" pkg=controller
time="2019-10-10T08:57:20Z" level=info msg="syncing of the cluster started" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T08:57:20Z" level=debug msg="team API is disabled, returning empty list of members for team \"user\"" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:20Z" level=debug msg="syncing secrets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:20Z" level=debug msg="secret \"db/foo-user.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:20Z" level=debug msg="secret \"db/kraken.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:20Z" level=debug msg="secret \"db/taras.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:20Z" level=debug msg="secret \"db/postgres.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:20Z" level=debug msg="secret \"db/standby.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:20Z" level=debug msg="syncing services" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:20Z" level=debug msg="syncing master service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:20Z" level=debug msg="syncing replica service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:20Z" level=debug msg="No load balancer created for the replica service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:20Z" level=debug msg="syncing persistent volumes" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:21Z" level=debug msg="skipping persistent volume \"pgdata-user-pg96-1\" corresponding to a non-running pods" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:21Z" level=debug msg="syncing statefulsets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:21Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is true" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:21Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE user-pg96 nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL version nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {KUBERNETES_LABELS application=spilo nil} {PGPASSWORD_SUPERUSER &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.user-pg96.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.user-pg96.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/9.6/bin\",\"pg_hba\":[\"local all all trust\",\"hostssl all +zalandos 127.0.0.1/32 pam\",\"host all all 127.0.0.1/32 md5\",\"hostssl all +zalandos ::1/128 pam\",\"host all all ::1/128 md5\",\"hostssl replication standby all md5\",\"hostnossl all all all md5\",\"hostssl all +zalandos all pam\",\"hostssl all all all md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:21Z" level=debug msg="statefulset's rolling update annotation has been set to false" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:21Z" level=debug msg="syncing pod disruption budgets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:21Z" level=debug msg="syncing roles" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:24Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:39Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:57:54Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:58:09Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:58:24Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:58:39Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:58:54Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:59:09Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:59:09Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T08:59:09Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T09:02:20Z" level=info msg="\"SYNC\" event has been queued" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T09:02:20Z" level=info msg="there are 1 clusters running" pkg=controller
time="2019-10-10T09:02:20Z" level=info msg="syncing of the cluster started" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T09:02:20Z" level=debug msg="team API is disabled, returning empty list of members for team \"user\"" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:20Z" level=debug msg="syncing secrets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:20Z" level=debug msg="secret \"db/taras.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:20Z" level=debug msg="secret \"db/postgres.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:20Z" level=debug msg="secret \"db/standby.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:20Z" level=debug msg="secret \"db/foo-user.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:20Z" level=debug msg="secret \"db/kraken.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:20Z" level=debug msg="syncing services" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:20Z" level=debug msg="syncing master service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:20Z" level=debug msg="syncing replica service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:20Z" level=debug msg="No load balancer created for the replica service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:20Z" level=debug msg="syncing persistent volumes" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:21Z" level=debug msg="skipping persistent volume \"pgdata-user-pg96-1\" corresponding to a non-running pods" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:21Z" level=debug msg="syncing statefulsets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:21Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is true" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:21Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE user-pg96 nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL version nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {KUBERNETES_LABELS application=spilo nil} {PGPASSWORD_SUPERUSER &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.user-pg96.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.user-pg96.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/9.6/bin\",\"pg_hba\":[\"local all all trust\",\"hostssl all +zalandos 127.0.0.1/32 pam\",\"host all all 127.0.0.1/32 md5\",\"hostssl all +zalandos ::1/128 pam\",\"host all all ::1/128 md5\",\"hostssl replication standby all md5\",\"hostnossl all all all md5\",\"hostssl all +zalandos all pam\",\"hostssl all all all md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:21Z" level=debug msg="statefulset's rolling update annotation has been set to false" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:21Z" level=debug msg="syncing pod disruption budgets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:21Z" level=debug msg="syncing roles" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:24Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:39Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:02:54Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:03:09Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:03:24Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:03:39Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:03:54Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:04:09Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:04:09Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:04:09Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T09:07:20Z" level=info msg="\"SYNC\" event has been queued" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T09:07:20Z" level=info msg="there are 1 clusters running" pkg=controller
time="2019-10-10T09:07:20Z" level=info msg="syncing of the cluster started" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T09:07:20Z" level=debug msg="team API is disabled, returning empty list of members for team \"user\"" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:20Z" level=debug msg="syncing secrets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:20Z" level=debug msg="secret \"db/foo-user.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:20Z" level=debug msg="secret \"db/kraken.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:20Z" level=debug msg="secret \"db/taras.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:20Z" level=debug msg="secret \"db/postgres.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:20Z" level=debug msg="secret \"db/standby.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:20Z" level=debug msg="syncing services" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:20Z" level=debug msg="syncing master service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:20Z" level=debug msg="syncing replica service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:20Z" level=debug msg="No load balancer created for the replica service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:20Z" level=debug msg="syncing persistent volumes" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:21Z" level=debug msg="skipping persistent volume \"pgdata-user-pg96-1\" corresponding to a non-running pods" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:21Z" level=debug msg="syncing statefulsets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:21Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is true" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:21Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE user-pg96 nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL version nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {KUBERNETES_LABELS application=spilo nil} {PGPASSWORD_SUPERUSER &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.user-pg96.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.user-pg96.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/9.6/bin\",\"pg_hba\":[\"local all all trust\",\"hostssl all +zalandos 127.0.0.1/32 pam\",\"host all all 127.0.0.1/32 md5\",\"hostssl all +zalandos ::1/128 pam\",\"host all all ::1/128 md5\",\"hostssl replication standby all md5\",\"hostnossl all all all md5\",\"hostssl all +zalandos all pam\",\"hostssl all all all md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:21Z" level=debug msg="statefulset's rolling update annotation has been set to false" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:21Z" level=debug msg="syncing pod disruption budgets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:21Z" level=debug msg="syncing roles" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:24Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:39Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:07:54Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:08:09Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:08:24Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:08:39Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:08:54Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:09:09Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:09:09Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:09:09Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T09:12:20Z" level=info msg="\"SYNC\" event has been queued" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T09:12:20Z" level=info msg="there are 1 clusters running" pkg=controller
time="2019-10-10T09:12:20Z" level=info msg="syncing of the cluster started" cluster-name=db/user-pg96 pkg=controller worker=0
time="2019-10-10T09:12:20Z" level=debug msg="team API is disabled, returning empty list of members for team \"user\"" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:20Z" level=debug msg="syncing secrets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:20Z" level=debug msg="secret \"db/foo-user.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:20Z" level=debug msg="secret \"db/kraken.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:20Z" level=debug msg="secret \"db/taras.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:20Z" level=debug msg="secret \"db/postgres.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:20Z" level=debug msg="secret \"db/standby.user-pg96.credentials\" already exists, fetching its password" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:20Z" level=debug msg="syncing services" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:20Z" level=debug msg="syncing master service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:20Z" level=debug msg="syncing replica service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:20Z" level=debug msg="No load balancer created for the replica service" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:20Z" level=debug msg="syncing persistent volumes" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:21Z" level=debug msg="skipping persistent volume \"pgdata-user-pg96-1\" corresponding to a non-running pods" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:21Z" level=debug msg="syncing statefulsets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:21Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is true" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:21Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE user-pg96 nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL version nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {KUBERNETES_LABELS application=spilo nil} {PGPASSWORD_SUPERUSER &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.user-pg96.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.user-pg96.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/9.6/bin\",\"pg_hba\":[\"local all all trust\",\"hostssl all +zalandos 127.0.0.1/32 pam\",\"host all all 127.0.0.1/32 md5\",\"hostssl all +zalandos ::1/128 pam\",\"host all all ::1/128 md5\",\"hostssl replication standby all md5\",\"hostnossl all all all md5\",\"hostssl all +zalandos all pam\",\"hostssl all all all md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:21Z" level=debug msg="statefulset's rolling update annotation has been set to false" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:21Z" level=debug msg="syncing pod disruption budgets" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:21Z" level=debug msg="syncing roles" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:24Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:39Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:12:54Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:13:09Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:13:24Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:13:39Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:13:54Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:14:09Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.104.53.216:5432: connect: no route to host" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:14:09Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=db/user-pg96 pkg=cluster
time="2019-10-10T09:14:09Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=db/user-pg96 pkg=controller worker=0
From the logs you can see that the host user-pg96.db.svc.cluster.local can not be resolved. Maybe your cluster domain differs from cluster.local like in this issue. Note, that this string can be configured, e.g. here.
Hi, I have a problem with pod after fail node on which master pod run, I configure operator to reboot quicker when fail node when is the operator container, but still don't help with db pod. operator config:
postgresql manifest:
after 20 minutes on failure:
operator-log: