Closed gnoguchi closed 9 months ago
FxKu
commented
4 years ago @gnoguchi I see that your cluster has only on instance. Did you change something in the manifest before? I remember that we had a bug on service updates, which was fixed in v1.4.0. Maybe you can consider upgrading?
gnoguchi
commented
4 years ago No, nothing has changed. Also, I've changed my operator version to v1.4.0, recreated one of postgresql instances and it looks the problems persists. The operator can't communicate with the clusters for some reason that I really don't know :(
kubectl logs postgres-operator-74658bf69b-227lr
2020/03/31 14:02:03 Fully qualified configmap name: platform/postgres-operator
2020/03/31 14:02:03 Spilo operator v1.4.0
time="2020-03-31T14:02:03Z" level=warning msg="in the operator configuration, the pod service account name postgres-operator does not match the name postgres-pod given in the account definition; using the former for consistency" pkg=controller
time="2020-03-31T14:02:03Z" level=info msg="Parse role bindings" pkg=controller
time="2020-03-31T14:02:03Z" level=info msg="successfully parsed" pkg=controller
time="2020-03-31T14:02:03Z" level=info msg="Listening to all namespaces" pkg=controller
time="2020-03-31T14:02:03Z" level=info msg="customResourceDefinition \"postgresqls.acid.zalan.do\" is already registered and will only be updated" pkg=controller
time="2020-03-31T14:02:07Z" level=warning msg="in the operator configuration, the pod service account name postgres-operator does not match the name postgres-pod given in the account definition; using the former for consistency" pkg=controller
time="2020-03-31T14:02:07Z" level=info msg="config: {\n\t\"ReadyWaitInterval\": 3000000000,\n\t\"ReadyWaitTimeout\": 30000000000,\n\t\"ResyncPeriod\": 1800000000000,\n\t\"RepairPeriod\": 300000000000,\n\t\"EnableCRDValidation\": true,\n\t\"ResourceCheckInterval\": 3000000000,\n\t\"ResourceCheckTimeout\": 600000000000,\n\t\"PodLabelWaitTimeout\": 600000000000,\n\t\"PodDeletionWaitTimeout\": 600000000000,\n\t\"SpiloFSGroup\": null,\n\t\"PodPriorityClassName\": \"\",\n\t\"ClusterDomain\": \"cluster.local\",\n\t\"SpiloPrivileged\": false,\n\t\"ClusterLabels\": {\n\t\t\"application\": \"spilo\"\n\t},\n\t\"InheritedLabels\": null,\n\t\"ClusterNameLabel\": \"version\",\n\t\"PodRoleLabel\": \"spilo-role\",\n\t\"PodToleration\": null,\n\t\"DefaultCPURequest\": \"100m\",\n\t\"DefaultMemoryRequest\": \"100Mi\",\n\t\"DefaultCPULimit\": \"1\",\n\t\"DefaultMemoryLimit\": \"500Mi\",\n\t\"MinCPULimit\": \"250m\",\n\t\"MinMemoryLimit\": \"250Mi\",\n\t\"PodEnvironmentConfigMap\": \"\",\n\t\"NodeReadinessLabel\": null,\n\t\"MaxInstances\": -1,\n\t\"MinInstances\": -1,\n\t\"ShmVolume\": true,\n\t\"SecretNameTemplate\": \"{username}.{cluster}.credentials\",\n\t\"PamRoleName\": \"zalandos\",\n\t\"PamConfiguration\": \"https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees\",\n\t\"TeamsAPIUrl\": \"https://teams.example.com/api/\",\n\t\"OAuthTokenSecretName\": \"platform/postgresql-operator\",\n\t\"InfrastructureRolesSecretName\": \"/\",\n\t\"SuperUsername\": \"postgres\",\n\t\"ReplicationUsername\": \"standby\",\n\t\"ScalyrAPIKey\": \"\",\n\t\"ScalyrImage\": \"\",\n\t\"ScalyrServerURL\": \"https://upload.eu.scalyr.com\",\n\t\"ScalyrCPURequest\": \"100m\",\n\t\"ScalyrMemoryRequest\": \"50Mi\",\n\t\"ScalyrCPULimit\": \"1\",\n\t\"ScalyrMemoryLimit\": \"500Mi\",\n\t\"LogicalBackupSchedule\": \"30 00 * * *\",\n\t\"LogicalBackupDockerImage\": \"registry.opensource.zalan.do/acid/logical-backup\",\n\t\"LogicalBackupS3Bucket\": \"\",\n\t\"LogicalBackupS3Region\": \"\",\n\t\"LogicalBackupS3Endpoint\": \"\",\n\t\"LogicalBackupS3AccessKeyID\": \"\",\n\t\"LogicalBackupS3SecretAccessKey\": \"\",\n\t\"LogicalBackupS3SSE\": \"AES256\",\n\t\"WatchedNamespace\": \"\",\n\t\"EtcdHost\": \"\",\n\t\"DockerImage\": \"registry.opensource.zalan.do/acid/spilo-12:1.6-p2\",\n\t\"Sidecars\": null,\n\t\"PodServiceAccountName\": \"postgres-operator\",\n\t\"PodServiceAccountDefinition\": \"\\n\\t\\t{\\n\\t\\t\\t\\\"apiVersion\\\": \\\"v1\\\",\\n\\t\\t\\t\\\"kind\\\": \\\"ServiceAccount\\\",\\n\\t\\t\\t\\\"metadata\\\": {\\n\\t\\t\\t\\t\\\"name\\\": \\\"postgres-pod\\\"\\n\\t\\t\\t}\\n\\t\\t}\",\n\t\"PodServiceAccountRoleBindingDefinition\": \"\\n\\t\\t{\\n\\t\\t\\t\\\"apiVersion\\\": \\\"rbac.authorization.k8s.io/v1\\\",\\n\\t\\t\\t\\\"kind\\\": \\\"RoleBinding\\\",\\n\\t\\t\\t\\\"metadata\\\": {\\n\\t\\t\\t\\t \\\"name\\\": \\\"postgres-operator\\\"\\n\\t\\t\\t},\\n\\t\\t\\t\\\"roleRef\\\": {\\n\\t\\t\\t\\t\\\"apiGroup\\\": \\\"rbac.authorization.k8s.io\\\",\\n\\t\\t\\t\\t\\\"kind\\\": \\\"ClusterRole\\\",\\n\\t\\t\\t\\t\\\"name\\\": \\\"postgres-operator\\\"\\n\\t\\t\\t},\\n\\t\\t\\t\\\"subjects\\\": [\\n\\t\\t\\t\\t{\\n\\t\\t\\t\\t\\t\\\"kind\\\": \\\"ServiceAccount\\\",\\n\\t\\t\\t\\t\\t\\\"name\\\": \\\"postgres-operator\\\"\\n\\t\\t\\t\\t}\\n\\t\\t\\t]\\n\\t\\t}\",\n\t\"MasterPodMoveTimeout\": 1200000000000,\n\t\"DbHostedZone\": \"vivareal.io\",\n\t\"AWSRegion\": \"us-east-1\",\n\t\"WALES3Bucket\": \"\",\n\t\"LogS3Bucket\": \"\",\n\t\"KubeIAMRole\": \"\",\n\t\"AdditionalSecretMount\": \"\",\n\t\"AdditionalSecretMountPath\": \"/meta/credentials\",\n\t\"DebugLogging\": true,\n\t\"EnableDBAccess\": true,\n\t\"EnableTeamsAPI\": false,\n\t\"EnableTeamSuperuser\": false,\n\t\"TeamAdminRole\": \"admin\",\n\t\"EnableAdminRoleForUsers\": true,\n\t\"EnableMasterLoadBalancer\": false,\n\t\"EnableReplicaLoadBalancer\": false,\n\t\"CustomServiceAnnotations\": null,\n\t\"CustomPodAnnotations\": null,\n\t\"EnablePodAntiAffinity\": false,\n\t\"PodAntiAffinityTopologyKey\": \"kubernetes.io/hostname\",\n\t\"EnableLoadBalancer\": null,\n\t\"MasterDNSNameFormat\": \"{cluster}.{team}.{hostedzone}\",\n\t\"ReplicaDNSNameFormat\": \"{cluster}-repl.{team}.{hostedzone}\",\n\t\"PDBNameFormat\": \"postgres-{cluster}-pdb\",\n\t\"EnablePodDisruptionBudget\": true,\n\t\"EnableInitContainers\": true,\n\t\"EnableSidecars\": true,\n\t\"Workers\": 4,\n\t\"APIPort\": 8080,\n\t\"RingLogLines\": 100,\n\t\"ClusterHistoryEntries\": 1000,\n\t\"TeamAPIRoleConfiguration\": {\n\t\t\"log_statement\": \"all\"\n\t},\n\t\"PodTerminateGracePeriod\": 300000000000,\n\t\"PodManagementPolicy\": \"ordered_ready\",\n\t\"ProtectedRoles\": [\n\t\t\"admin\"\n\t],\n\t\"PostgresSuperuserTeams\": null,\n\t\"SetMemoryRequestToLimit\": false\n}" pkg=controller
time="2020-03-31T14:02:07Z" level=debug msg="acquiring initial list of clusters" pkg=controller
time="2020-03-31T14:02:07Z" level=debug msg="added new cluster: \"ecommerce/ecommerce-smaug-db\"" pkg=controller
time="2020-03-31T14:02:07Z" level=debug msg="added new cluster: \"monitoring/platform-grafana-postgres\"" pkg=controller
time="2020-03-31T14:02:07Z" level=info msg="\"SYNC\" event has been queued" cluster-name=ecommerce/ecommerce-smaug-db pkg=controller worker=0
time="2020-03-31T14:02:07Z" level=info msg="\"SYNC\" event has been queued" cluster-name=monitoring/platform-grafana-postgres pkg=controller worker=1
time="2020-03-31T14:02:07Z" level=info msg="there are 2 clusters running and 1 are in the failed state" pkg=controller
time="2020-03-31T14:02:07Z" level=info msg="started working in background" pkg=controller
time="2020-03-31T14:02:07Z" level=info msg="listening on :8080" pkg=apiserver
time="2020-03-31T14:02:07Z" level=info msg="\"ADD\" event has been queued" cluster-name=ecommerce/ecommerce-smaug-db pkg=controller worker=0
time="2020-03-31T14:02:07Z" level=debug msg="skipping \"ADD\" event for the invalid cluster: cluster name must match {TEAM}-{NAME} format. Got cluster name 'smaug-db', team name 'ecommerce'" cluster-name=ecommerce/smaug-db pkg=controller
time="2020-03-31T14:02:07Z" level=info msg="\"ADD\" event has been queued" cluster-name=monitoring/platform-grafana-postgres pkg=controller worker=1
time="2020-03-31T14:02:07Z" level=info msg="syncing of the cluster started" cluster-name=ecommerce/ecommerce-smaug-db pkg=controller worker=0
time="2020-03-31T14:02:07Z" level=debug msg="team API is disabled, returning empty list of members for team \"ecommerce\"" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:07Z" level=debug msg="syncing secrets" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:07Z" level=info msg="syncing of the cluster started" cluster-name=monitoring/platform-grafana-postgres pkg=controller worker=1
time="2020-03-31T14:02:07Z" level=debug msg="team API is disabled, returning empty list of members for team \"platform\"" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:07Z" level=debug msg="syncing secrets" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-85-140.ec2.internal\" (aws:///us-east-1a/i-04edf0cdf1d9d5871)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-115-20.ec2.internal\" (aws:///us-east-1e/i-0a4cb8aa29a885027)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-125-61.ec2.internal\" (aws:///us-east-1d/i-002424510c9f33a02)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-125-19.ec2.internal\" (aws:///us-east-1d/i-01c160c1e39cdc7c5)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-105-44.ec2.internal\" (aws:///us-east-1b/i-0bf09298c5f689841)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-105-187.ec2.internal\" (aws:///us-east-1b/i-0474c88b26819fb45)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-125-137.ec2.internal\" (aws:///us-east-1d/i-0a6c0012fbe5e3ba4)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-105-157.ec2.internal\" (aws:///us-east-1b/i-0ebbd7eba0674ff69)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-105-28.ec2.internal\" (aws:///us-east-1b/i-0bcc78abe2f9e725b)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-85-184.ec2.internal\" (aws:///us-east-1a/i-0b0a6747a23c4b2ce)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-105-203.ec2.internal\" (aws:///us-east-1b/i-003dbd16f900847b3)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-125-213.ec2.internal\" (aws:///us-east-1d/i-08fc48b8b0dfaf2ea)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-125-88.ec2.internal\" (aws:///us-east-1d/i-04482756c6ec54612)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-85-155.ec2.internal\" (aws:///us-east-1a/i-00690f8c33310b702)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-105-185.ec2.internal\" (aws:///us-east-1b/i-08f933dd171a61e36)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-125-218.ec2.internal\" (aws:///us-east-1d/i-062b6fe3ef324e809)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-85-131.ec2.internal\" (aws:///us-east-1a/i-0139828ccc9ad9cbe)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-105-106.ec2.internal\" (aws:///us-east-1b/i-0ca3ffdd9e6abadbc)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-85-59.ec2.internal\" (aws:///us-east-1a/i-011f7fa77261d8075)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-125-231.ec2.internal\" (aws:///us-east-1d/i-064a796bbd281e313)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-125-157.ec2.internal\" (aws:///us-east-1d/i-0ad503592f23072c4)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-125-148.ec2.internal\" (aws:///us-east-1d/i-00eed893c52468062)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-85-44.ec2.internal\" (aws:///us-east-1a/i-08f1316c9d4bc866b)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-135-25.ec2.internal\" (aws:///us-east-1f/i-0bb43327eff9746eb)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-85-60.ec2.internal\" (aws:///us-east-1a/i-0852b393725a481e9)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-105-196.ec2.internal\" (aws:///us-east-1b/i-0e5f7e95c88c4bd77)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-105-9.ec2.internal\" (aws:///us-east-1b/i-0d8d31ad4db3035e9)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="new node has been added: \"/ip-10-160-85-173.ec2.internal\" (aws:///us-east-1a/i-0019869efc6e76d4c)" pkg=controller
time="2020-03-31T14:02:08Z" level=debug msg="secret \"monitoring/grafana.platform-grafana-postgres.credentials\" already exists, fetching its password" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:08Z" level=debug msg="secret \"ecommerce/smaug.ecommerce-smaug-db.credentials\" already exists, fetching its password" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:08Z" level=debug msg="secret \"monitoring/pgadmin.platform-grafana-postgres.credentials\" already exists, fetching its password" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:08Z" level=debug msg="secret \"ecommerce/postgres.ecommerce-smaug-db.credentials\" already exists, fetching its password" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:08Z" level=debug msg="secret \"monitoring/postgres.platform-grafana-postgres.credentials\" already exists, fetching its password" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:09Z" level=debug msg="secret \"ecommerce/standby.ecommerce-smaug-db.credentials\" already exists, fetching its password" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:09Z" level=debug msg="secret \"monitoring/standby.platform-grafana-postgres.credentials\" already exists, fetching its password" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:09Z" level=debug msg="syncing services" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:09Z" level=debug msg="syncing master service" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:09Z" level=debug msg="secret \"ecommerce/pgadmin.ecommerce-smaug-db.credentials\" already exists, fetching its password" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:09Z" level=debug msg="syncing services" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:09Z" level=debug msg="syncing master service" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:10Z" level=debug msg="syncing replica service" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:10Z" level=debug msg="syncing replica service" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:11Z" level=debug msg="No load balancer created for the replica service" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:11Z" level=debug msg="syncing persistent volumes" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:11Z" level=debug msg="No load balancer created for the replica service" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:11Z" level=debug msg="syncing persistent volumes" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="syncing statefulsets" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE platform-grafana-postgres nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL version nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {KUBERNETES_LABELS application=spilo nil} {PGPASSWORD_SUPERUSER &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.platform-grafana-postgres.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.platform-grafana-postgres.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/9.5/bin\",\"pg_hba\":[\"host all all 0.0.0.0/0 md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="statefulset's rolling update annotation has been set to false" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="statefulset's rolling update annotation has been set to true" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:12Z" level=info msg="statefulset \"monitoring/platform-grafana-postgres\" is not in the desired state and needs to be updated" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="metadata.annotation diff\n[\"zalando-postgres-operator-rolling-update-required\"]: \"false\" != \"true\"\n" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="spec diff between old and new statefulsets: \nTemplate.Spec.Containers[0].Resources.Limits[\"cpu\"].i.value: 3 != 1\nTemplate.Spec.Containers[0].Resources.Limits[\"cpu\"].s: \"3\" != \"1\"\nTemplate.Spec.Containers[0].Resources.Limits[\"memory\"].i.value: 1073741824 != 524288000\nTemplate.Spec.Containers[0].Resources.Limits[\"memory\"].s: \"1Gi\" != \"500Mi\"\nTemplate.Spec.Containers[0].TerminationMessagePath: \"/dev/termination-log\" != \"\"\nTemplate.Spec.Containers[0].TerminationMessagePolicy: \"File\" != \"\"\nTemplate.Spec.RestartPolicy: \"Always\" != \"\"\nTemplate.Spec.DNSPolicy: \"ClusterFirst\" != \"\"\nTemplate.Spec.DeprecatedServiceAccount: \"postgres-operator\" != \"\"\nTemplate.Spec.SchedulerName: \"default-scheduler\" != \"\"\nTemplate.Spec.Tolerations: []v1.Toleration(nil) != []v1.Toleration{}\nVolumeClaimTemplates[0].TypeMeta.Kind: \"PersistentVolumeClaim\" != \"\"\nVolumeClaimTemplates[0].TypeMeta.APIVersion: \"v1\" != \"\"\nVolumeClaimTemplates[0].Status.Phase: \"Pending\" != \"\"\nRevisionHistoryLimit: &int32(10) != nil\n" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:12Z" level=info msg="reason: \"new statefulset containers's postgres (index 0) resources don't match the current ones\"" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="updating statefulset" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="updating statefulset annotations" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="performing rolling update" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="syncing statefulsets" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE ecommerce-smaug-db nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL version nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {KUBERNETES_LABELS application=spilo nil} {PGPASSWORD_SUPERUSER &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.ecommerce-smaug-db.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.ecommerce-smaug-db.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/9.5/bin\",\"pg_hba\":[\"host all all 0.0.0.0/0 md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="statefulset's rolling update annotation has been set to false" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="statefulset's rolling update annotation has been set to true" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:12Z" level=info msg="statefulset \"ecommerce/ecommerce-smaug-db\" is not in the desired state and needs to be updated" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="metadata.annotation diff\n[\"zalando-postgres-operator-rolling-update-required\"]: \"false\" != \"true\"\n" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="spec diff between old and new statefulsets: \nTemplate.Spec.Containers[0].Resources.Limits[\"cpu\"].i.value: 3 != 1\nTemplate.Spec.Containers[0].Resources.Limits[\"cpu\"].s: \"3\" != \"1\"\nTemplate.Spec.Containers[0].Resources.Limits[\"memory\"].i.value: 1073741824 != 524288000\nTemplate.Spec.Containers[0].Resources.Limits[\"memory\"].s: \"1Gi\" != \"500Mi\"\nTemplate.Spec.Containers[0].TerminationMessagePath: \"/dev/termination-log\" != \"\"\nTemplate.Spec.Containers[0].TerminationMessagePolicy: \"File\" != \"\"\nTemplate.Spec.RestartPolicy: \"Always\" != \"\"\nTemplate.Spec.DNSPolicy: \"ClusterFirst\" != \"\"\nTemplate.Spec.DeprecatedServiceAccount: \"postgres-operator\" != \"\"\nTemplate.Spec.SchedulerName: \"default-scheduler\" != \"\"\nTemplate.Spec.Tolerations: []v1.Toleration(nil) != []v1.Toleration{}\nVolumeClaimTemplates[0].TypeMeta.Kind: \"PersistentVolumeClaim\" != \"\"\nVolumeClaimTemplates[0].TypeMeta.APIVersion: \"v1\" != \"\"\nVolumeClaimTemplates[0].Status.Phase: \"Pending\" != \"\"\nRevisionHistoryLimit: &int32(10) != nil\n" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:12Z" level=info msg="reason: \"new statefulset containers's postgres (index 0) resources don't match the current ones\"" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="updating statefulset" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="updating statefulset annotations" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="performing rolling update" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:12Z" level=info msg="there are 1 pods in the cluster to recreate" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:12Z" level=debug msg="subscribing to pod \"monitoring/platform-grafana-postgres-0\"" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:02:13Z" level=info msg="there are 1 pods in the cluster to recreate" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:13Z" level=warning msg="cannot perform switch over before re-creating the pod: no replicas" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:13Z" level=info msg="recreating old master pod \"ecommerce/ecommerce-smaug-db-0\"" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:13Z" level=debug msg="subscribing to pod \"ecommerce/ecommerce-smaug-db-0\"" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:30Z" level=info msg="pod \"ecommerce/ecommerce-smaug-db-0\" has been recreated" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:30Z" level=debug msg="unsubscribing from pod \"ecommerce/ecommerce-smaug-db-0\" events" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:30Z" level=info msg="pods have been recreated" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:30Z" level=debug msg="statefulset's rolling update annotation has been set to false" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:30Z" level=debug msg="updating statefulset annotations" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:30Z" level=debug msg="syncing pod disruption budgets" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:30Z" level=debug msg="syncing roles" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:31Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:02:46Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:03:01Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:03:16Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:03:31Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:03:46Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:04:01Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:04:16Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:04:16Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:04:16Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=ecommerce/ecommerce-smaug-db pkg=controller worker=0
time="2020-03-31T14:04:16Z" level=debug msg="cluster already exists" cluster-name=ecommerce/ecommerce-smaug-db pkg=controller worker=0
time="2020-03-31T14:12:29Z" level=debug msg="unsubscribing from pod \"monitoring/platform-grafana-postgres-0\" events" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:12:29Z" level=warning msg="error while syncing cluster state: could not sync statefulsets: could not recreate pods: could not recreate replica pod \"monitoring/platform-grafana-postgres-0\": pod label wait timeout" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:12:29Z" level=error msg="could not sync cluster: could not sync statefulsets: could not recreate pods: could not recreate replica pod \"monitoring/platform-grafana-postgres-0\": pod label wait timeout" cluster-name=monitoring/platform-grafana-postgres pkg=controller worker=1
time="2020-03-31T14:12:29Z" level=debug msg="cluster already exists" cluster-name=monitoring/platform-grafana-postgres pkg=controller worker=1
time="2020-03-31T14:32:07Z" level=info msg="\"SYNC\" event has been queued" cluster-name=ecommerce/ecommerce-smaug-db pkg=controller worker=0
time="2020-03-31T14:32:07Z" level=info msg="\"SYNC\" event has been queued" cluster-name=monitoring/platform-grafana-postgres pkg=controller worker=1
time="2020-03-31T14:32:07Z" level=info msg="there are 2 clusters running and 1 are in the failed state" pkg=controller
time="2020-03-31T14:32:07Z" level=info msg="syncing of the cluster started" cluster-name=monitoring/platform-grafana-postgres pkg=controller worker=1
time="2020-03-31T14:32:07Z" level=debug msg="team API is disabled, returning empty list of members for team \"platform\"" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:07Z" level=debug msg="syncing secrets" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:07Z" level=info msg="syncing of the cluster started" cluster-name=ecommerce/ecommerce-smaug-db pkg=controller worker=0
time="2020-03-31T14:32:07Z" level=debug msg="team API is disabled, returning empty list of members for team \"ecommerce\"" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:07Z" level=debug msg="syncing secrets" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:08Z" level=debug msg="secret \"monitoring/standby.platform-grafana-postgres.credentials\" already exists, fetching its password" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:08Z" level=debug msg="secret \"ecommerce/pgadmin.ecommerce-smaug-db.credentials\" already exists, fetching its password" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:08Z" level=debug msg="secret \"monitoring/postgres.platform-grafana-postgres.credentials\" already exists, fetching its password" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:08Z" level=debug msg="secret \"ecommerce/smaug.ecommerce-smaug-db.credentials\" already exists, fetching its password" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:08Z" level=debug msg="secret \"monitoring/grafana.platform-grafana-postgres.credentials\" already exists, fetching its password" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:08Z" level=debug msg="secret \"ecommerce/standby.ecommerce-smaug-db.credentials\" already exists, fetching its password" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:09Z" level=debug msg="secret \"monitoring/pgadmin.platform-grafana-postgres.credentials\" already exists, fetching its password" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:09Z" level=debug msg="syncing services" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:09Z" level=debug msg="syncing master service" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:09Z" level=debug msg="secret \"ecommerce/postgres.ecommerce-smaug-db.credentials\" already exists, fetching its password" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:09Z" level=debug msg="syncing services" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:09Z" level=debug msg="syncing master service" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:10Z" level=debug msg="syncing replica service" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:10Z" level=debug msg="syncing replica service" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:10Z" level=debug msg="No load balancer created for the replica service" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:10Z" level=debug msg="syncing persistent volumes" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:11Z" level=debug msg="No load balancer created for the replica service" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:11Z" level=debug msg="syncing persistent volumes" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:12Z" level=debug msg="syncing statefulsets" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:12Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is false" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:12Z" level=info msg="found a statefulset with an unfinished rolling update of the pods" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:12Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE platform-grafana-postgres nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL version nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {KUBERNETES_LABELS application=spilo nil} {PGPASSWORD_SUPERUSER &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.platform-grafana-postgres.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.platform-grafana-postgres.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/9.5/bin\",\"pg_hba\":[\"host all all 0.0.0.0/0 md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:12Z" level=debug msg="statefulset's rolling update annotation has been set to true" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:12Z" level=debug msg="performing rolling update" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:12Z" level=debug msg="syncing statefulsets" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:12Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is true" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:12Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE ecommerce-smaug-db nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL version nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {KUBERNETES_LABELS application=spilo nil} {PGPASSWORD_SUPERUSER &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.ecommerce-smaug-db.credentials,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.ecommerce-smaug-db.credentials,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/9.5/bin\",\"pg_hba\":[\"host all all 0.0.0.0/0 md5\"]},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"},\"data-checksums\",{\"encoding\":\"UTF8\"},{\"locale\":\"en_US.UTF-8\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:12Z" level=debug msg="statefulset's rolling update annotation has been set to false" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:12Z" level=debug msg="syncing pod disruption budgets" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:12Z" level=debug msg="syncing roles" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:12Z" level=info msg="there are 1 pods in the cluster to recreate" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:12Z" level=debug msg="subscribing to pod \"monitoring/platform-grafana-postgres-0\"" cluster-name=monitoring/platform-grafana-postgres pkg=cluster
time="2020-03-31T14:32:13Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:28Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:43Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:32:58Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:33:13Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:33:28Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:33:43Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:33:58Z" level=error msg="could not connect to PostgreSQL database: dial tcp 172.20.30.6:5432: connect: connection refused" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:33:58Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=ecommerce/ecommerce-smaug-db pkg=cluster
time="2020-03-31T14:33:58Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=ecommerce/ecommerce-smaug-db pkg=controller worker=0Hm, there are two different errors. For this smaug cluster, can you check if the service for the master exists. Is this maybe a cloned cluster and the credentials in the DB are out of sync with the secrets, hence connection doesn't work.
For the grafana cluster I see a pod label wait timeout. You could exec into the pod and run patronictl reinit to re-initializing this replica. The problem here, unfortunately, if this takes longer than the timeout, the operator will kill this pod again. There are some tricks for this scenario, but first try the steps mentioned.
aditya1808
commented
4 years ago @gnoguchi Is the issue resolved? if so, could you let us know what exactly the issue was?
jorgex1
commented
4 years ago Jumping in on this issue because i ended up with a similar problem. What would be a solution if the cluster was cloned and the credentials in the DB are out of sync with the secrets?
Starefossen
commented
2 years ago We also see random connection refused when trying to create new databases. Re-creating the database seams to solve the problem.
Im facing some problems with Postgres clusters. All my db clusters are presenting the status
SyncFailed.db cluster pod logs:
Postgres operator pod logs:
Can you help me, please?
Thanks in advance!