search

galaxyproject / galaxy-helm

Minimal setup required to run Galaxy under Kubernetes
MIT License
41 stars 39 forks source link

Installs stalls during PG initialization #352

Closed 23skdu closed 2 years ago

23skdu commented 2 years ago

$ helm version version.BuildInfo{Version:"v3.7.2", GitCommit:"663a896f4a815053445eec4153677ddc24a0a361", GitTreeState:"clean", GoVersion:"go1.16.10"} Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.20-eks-8c49e2", GitCommit:"8c49e2efc3cfbb7788a58025e679787daed22018", GitTreeState:"clean", BuildDate:"2021-10-17T05:13:46Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

[cloudshell-user galaxy]$ kubectl get pods NAMESPACE NAME READY STATUS RESTARTS AGE galaxy-prod galaxy-galaxy-galaxy-postgres-0 1/1 Running 0 18m galaxy-prod galaxy-galaxy-postgres-5b5fcfff54-lznw6 1/1 Running 0 18m galaxy-prod galaxy-init-db-1rntx-g4rph 0/1 Init:0/1 0 18m galaxy-prod galaxy-init-mounts-kr7mv-db2r9 0/1 Completed 0 18m galaxy-prod galaxy-job-0-c5779bcdf-hk2kc 0/1 Init:0/1 0 18m galaxy-prod galaxy-nginx-8695bc94d5-t469x 1/1 Running 0 18m galaxy-prod galaxy-web-69b6bb5fb5-bj75n 0/1 Init:0/1 0 18m galaxy-prod galaxy-workflow-7f87d9787-nblkc 0/1 Init:0/1 0 18m

seems to be waiting on the DB, and looking into that container, i find this

$ kubectl logs galaxy-galaxy-galaxy-postgres-0 -n galaxy-prod 2022-02-16 00:38:13,642 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?) 2022-02-16 00:38:13,656 - bootstrapping - INFO - Looks like your running aws 2022-02-16 00:38:13,683 - bootstrapping - INFO - Configuring bootstrap 2022-02-16 00:38:13,684 - bootstrapping - INFO - Configuring pgqd 2022-02-16 00:38:13,684 - bootstrapping - INFO - Configuring log 2022-02-16 00:38:13,684 - bootstrapping - INFO - Configuring wal-e 2022-02-16 00:38:13,684 - bootstrapping - INFO - Configuring standby-cluster 2022-02-16 00:38:13,684 - bootstrapping - INFO - Configuring certificate 2022-02-16 00:38:13,685 - bootstrapping - INFO - Generating ssl certificate 2022-02-16 00:38:13,721 - bootstrapping - INFO - Configuring pgbouncer 2022-02-16 00:38:13,721 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping 2022-02-16 00:38:13,721 - bootstrapping - INFO - Configuring crontab 2022-02-16 00:38:13,721 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability 2022-02-16 00:38:13,722 - bootstrapping - INFO - Configuring patroni 2022-02-16 00:38:13,730 - bootstrapping - INFO - Writing to file /run/postgres.yml 2022-02-16 00:38:13,731 - bootstrapping - INFO - Configuring pam-oauth2 2022-02-16 00:38:13,731 - bootstrapping - INFO - Writing to file /etc/pam.d/postgresql 2022-02-16 00:38:15,087 WARNING: Kubernetes RBAC doesn't allow GET access to the 'kubernetes' endpoint in the 'default' namespace. Disabling 'bypass_api_service'. 2022-02-16 00:38:15,109 INFO: No PostgreSQL configuration items changed, nothing to reload. 2022-02-16 00:38:15,112 INFO: Lock owner: galaxy-galaxy-galaxy-postgres-0; I am galaxy-galaxy-galaxy-postgres-0 2022-02-16 00:38:15,247 INFO: Leader key released 2022-02-16 00:38:15,305 INFO: released leader key voluntarily as data dir empty and currently leader 2022-02-16 00:38:15,306 INFO: Lock owner: None; I am galaxy-galaxy-galaxy-postgres-0 2022-02-16 00:38:15,306 INFO: waiting for leader to bootstrap

Not sure if that is why it is having issues with this CRD.

$ kubectl logs galaxy-galaxy-postgres-5b5fcfff54-lznw6 -n galaxy-prod time="2022-02-16T00:37:36Z" level=info msg="Fully qualified configmap name: galaxy-prod/galaxy-galaxy-postgres" time="2022-02-16T00:37:36Z" level=info msg="Spilo operator v1.6.3\n" time="2022-02-16T00:37:36Z" level=info msg="Parse role bindings" pkg=controller time="2022-02-16T00:37:36Z" level=info msg="successfully parsed" pkg=controller time="2022-02-16T00:37:36Z" level=info msg="Listening to all namespaces" pkg=controller time="2022-02-16T00:37:36Z" level=info msg="customResourceDefinition \"postgresqls.acid.zalan.do\" is already registered and will only be updated" pkg=controller time="2022-02-16T00:37:40Z" level=info msg="{" pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ReadyWaitInterval\": 3000000000," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ReadyWaitTimeout\": 30000000000," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ResyncPeriod\": 1800000000000," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"RepairPeriod\": 300000000000," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableCRDValidation\": true," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ResourceCheckInterval\": 3000000000," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ResourceCheckTimeout\": 600000000000," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PodLabelWaitTimeout\": 600000000000," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PodDeletionWaitTimeout\": 600000000000," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"SpiloRunAsUser\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"SpiloRunAsGroup\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"SpiloFSGroup\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PodPriorityClassName\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ClusterDomain\": \"cluster.local\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"SpiloPrivileged\": false," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"SpiloAllowPrivilegeEscalation\": true," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"AdditionalPodCapabilities\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ClusterLabels\": {" pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"application\": \"spilo\"" pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" }," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"InheritedLabels\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"InheritedAnnotations\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"DownscalerAnnotations\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ClusterNameLabel\": \"cluster-name\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"DeleteAnnotationDateKey\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"DeleteAnnotationNameKey\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PodRoleLabel\": \"spilo-role\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PodToleration\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"DefaultCPURequest\": \"100m\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"DefaultMemoryRequest\": \"100Mi\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"DefaultCPULimit\": \"1\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"DefaultMemoryLimit\": \"500Mi\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"MinCPULimit\": \"250m\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"MinMemoryLimit\": \"250Mi\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PodEnvironmentConfigMap\": \"/\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PodEnvironmentSecret\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"NodeReadinessLabel\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"MaxInstances\": -1," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"MinInstances\": -1," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ShmVolume\": true," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"SecretNameTemplate\": \"{username}.{cluster}.credentials.{tprkind}.{tprgroup}\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PamRoleName\": \"zalandos\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PamConfiguration\": \"https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"TeamsAPIUrl\": \"https://teams.example.com/api/\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"OAuthTokenSecretName\": \"galaxy-prod/postgresql-operator\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"InfrastructureRolesSecretName\": \"/\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"InfrastructureRoles\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"InfrastructureRolesDefs\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"SuperUsername\": \"postgres\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ReplicationUsername\": \"standby\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ScalyrAPIKey\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ScalyrImage\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ScalyrServerURL\": \"https://upload.eu.scalyr.com\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ScalyrCPURequest\": \"100m\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ScalyrMemoryRequest\": \"50Mi\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ScalyrCPULimit\": \"1\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ScalyrMemoryLimit\": \"500Mi\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"LogicalBackupSchedule\": \"30 00 *\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"LogicalBackupDockerImage\": \"registry.opensource.zalan.do/acid/logical-backup:v1.6.3\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"LogicalBackupProvider\": \"s3\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"LogicalBackupS3Bucket\": \"my-bucket-url\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"LogicalBackupS3Region\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"LogicalBackupS3Endpoint\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"LogicalBackupS3AccessKeyID\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"LogicalBackupS3SecretAccessKey\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"LogicalBackupS3SSE\": \"AES256\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"LogicalBackupGoogleApplicationCredentials\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"LogicalBackupJobPrefix\": \"logical-backup-\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"NumberOfInstances\": 2," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"Schema\": \"pooler\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"User\": \"pooler\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"Image\": \"registry.opensource.zalan.do/acid/pgbouncer:master-16\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"Mode\": \"transaction\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"MaxDBConnections\": 60," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ConnectionPoolerDefaultCPURequest\": \"500m\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ConnectionPoolerDefaultMemoryRequest\": \"100Mi\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ConnectionPoolerDefaultCPULimit\": \"1\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ConnectionPoolerDefaultMemoryLimit\": \"100Mi\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"WatchedNamespace\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"KubernetesUseConfigMaps\": false," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EtcdHost\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"DockerImage\": \"registry.opensource.zalan.do/acid/spilo-13:2.0-p7\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"SidecarImages\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"SidecarContainers\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PodServiceAccountName\": \"postgres-pod\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PodServiceAccountDefinition\": \"{\\"apiVersion\\":\\"v1\\",\\"kind\\":\\"ServiceAccount\\",\\"metadata\\":{\\"name\\":\\"postgres-pod\\"}}\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PodServiceAccountRoleBindingDefinition\": \"{\\"apiVersion\\":\\"rbac.authorization.k8s.io/v1\\",\\"kind\\":\\"RoleBinding\\",\\"metadata\\":{\\"name\\":\\"postgres-pod\\"},\\"roleRef\\":{\\"apiGroup\\":\\"rbac.authorization.k8s.io\\",\\"kind\\":\\"ClusterRole\\",\\"name\\":\\"postgres-pod\\"},\\"subjects\\":[{\\"kind\\":\\"ServiceAccount\\",\\"name\\":\\"postgres-pod\\"}]}\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"MasterPodMoveTimeout\": 1200000000000," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"DbHostedZone\": \"db.example.com\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"AWSRegion\": \"eu-central-1\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"WALES3Bucket\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"LogS3Bucket\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"KubeIAMRole\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"WALGSBucket\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"GCPCredentials\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"AdditionalSecretMount\": \"\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"AdditionalSecretMountPath\": \"/meta/credentials\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableEBSGp3Migration\": false," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableEBSGp3MigrationMaxSize\": 1000," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"DebugLogging\": true," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableDBAccess\": true," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableTeamsAPI\": false," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableTeamSuperuser\": false," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"TeamAdminRole\": \"admin\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"RoleDeletionSuffix\": \"_deleted\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableTeamMemberDeprecation\": false," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableAdminRoleForUsers\": true," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnablePostgresTeamCRD\": false," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnablePostgresTeamCRDSuperusers\": false," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableMasterLoadBalancer\": false," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableReplicaLoadBalancer\": false," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"CustomServiceAnnotations\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"CustomPodAnnotations\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnablePodAntiAffinity\": false," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PodAntiAffinityTopologyKey\": \"kubernetes.io/hostname\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"StorageResizeMode\": \"pvc\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableLoadBalancer\": null," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ExternalTrafficPolicy\": \"Cluster\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"MasterDNSNameFormat\": \"{cluster}.{team}.{hostedzone}\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ReplicaDNSNameFormat\": \"{cluster}-repl.{team}.{hostedzone}\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PDBNameFormat\": \"postgres-{cluster}-pdb\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnablePodDisruptionBudget\": true," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableInitContainers\": true," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableSidecars\": true," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"Workers\": 8," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"APIPort\": 8080," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"RingLogLines\": 100," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ClusterHistoryEntries\": 1000," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"TeamAPIRoleConfiguration\": {" pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"log_statement\": \"all\"" pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" }," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PodTerminateGracePeriod\": 300000000000," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PodManagementPolicy\": \"ordered_ready\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"ProtectedRoles\": [" pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"admin\"" pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" ]," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"PostgresSuperuserTeams\": [" pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"postgres_superusers\"" pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" ]," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"SetMemoryRequestToLimit\": false," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableLazySpiloUpgrade\": false," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnablePgVersionEnvVar\": true," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"EnableSpiloWalPathCompat\": false," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"MajorVersionUpgradeMode\": \"off\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"MinimalMajorVersion\": \"9.5\"," pkg=controller time="2022-02-16T00:37:40Z" level=info msg=" \"TargetMajorVersion\": \"13\"" pkg=controller time="2022-02-16T00:37:40Z" level=info msg="}" pkg=controller time="2022-02-16T00:37:40Z" level=debug msg="acquiring initial list of clusters" pkg=controller time="2022-02-16T00:37:40Z" level=debug msg="added new cluster: \"galaxy-prod/galaxy-galaxy-galaxy-postgres\"" pkg=controller time="2022-02-16T00:37:40Z" level=info msg="SYNC event has been queued" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=controller worker=0 time="2022-02-16T00:37:40Z" level=info msg="there are 1 clusters running" pkg=controller time="2022-02-16T00:37:40Z" level=info msg="started working in background" pkg=controller time="2022-02-16T00:37:40Z" level=info msg="listening on :8080" pkg=apiserver time="2022-02-16T00:37:40Z" level=info msg="ADD event has been queued" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=controller worker=0 time="2022-02-16T00:37:40Z" level=debug msg="new node has been added: /ip-10-0-2-74.us-west-2.compute.internal (aws:///us-west-2b/i-04feb62d6407c3e17)" pkg=controller time="2022-02-16T00:37:40Z" level=info msg="syncing of the cluster started" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=controller worker=0 time="2022-02-16T00:37:40Z" level=debug msg="team API is disabled" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:40Z" level=debug msg="team API is disabled" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:40Z" level=info msg="syncing secrets" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:40Z" level=debug msg="secret galaxy-prod/galaxydbuser.galaxy-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:40Z" level=debug msg="secret galaxy-prod/postgres.galaxy-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:40Z" level=debug msg="secret galaxy-prod/standby.galaxy-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:40Z" level=debug msg="syncing master service" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:40Z" level=debug msg="syncing replica service" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:41Z" level=debug msg="No load balancer created for the replica service" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:41Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:41Z" level=info msg="volume claims do not require changes" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:41Z" level=debug msg="syncing statefulsets" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:41Z" level=info msg="cluster's statefulset does not exist" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:41Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:41Z" level=debug msg="[{SCOPE galaxy-galaxy-galaxy-postgres nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.galaxy-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.galaxy-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {\"application\":\"spilo\"} nil} {SPILO_CONFIGURATION {\"postgresql\":{},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:41Z" level=debug msg="created new statefulset \"galaxy-prod/galaxy-galaxy-galaxy-postgres\", uid: \"947d6dec-841e-409c-b349-0c6f6347cb5d\"" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:37:44Z" level=debug msg="Waiting for 1 pods to become ready" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:47:41Z" level=warning msg="error while syncing cluster state: could not sync statefulsets: cluster is not ready: pod labels error: still failing after 200 retries" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster time="2022-02-16T00:47:41Z" level=error msg="could not sync cluster: could not sync statefulsets: cluster is not ready: pod labels error: still failing after 200 retries" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=controller worker=0 time="2022-02-16T00:47:41Z" level=info msg="recieved add event for already existing Postgres cluster" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=controller worker=0

23skdu commented 2 years ago

nevermind, will write my own if no one maintains this one.