search

CrunchyData / postgres-operator

Production PostgreSQL for Kubernetes, from high availability Postgres clusters to full-scale database-as-a-service.
https://access.crunchydata.com/documentation/postgres-operator/v5/
Apache License 2.0
3.92k stars 592 forks source link

Cluster creation fails #2417

Closed batistein closed 3 years ago

batistein commented 3 years ago

Overview

When creating a cluster with pgo cli, the postgres cluster is not created. See logs:

File "/usr/local/lib/python3.6/site-packages/patroni/__init__.py", line 138, in patroni_main
abstract_main(Patroni, schema)
File "/usr/local/lib/python3.6/site-packages/patroni/daemon.py", line 100, in abstract_main
controller.run()
File "/usr/local/lib/python3.6/site-packages/patroni/__init__.py", line 108, in run
super(Patroni, self).run()
File "/usr/local/lib/python3.6/site-packages/patroni/daemon.py", line 59, in run
self._run_cycle()
File "/usr/local/lib/python3.6/site-packages/patroni/__init__.py", line 111, in _run_cycle
logger.info(self.ha.run_cycle())
File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1457, in run_cycle
info = self._run_cycle()
File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1351, in _run_cycle
return self.post_bootstrap()
File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1247, in post_bootstrap
self.cancel_initialization()
File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1240, in cancel_initialization
raise PatroniFatalException('Failed to bootstrap cluster')
patroni.exceptions.PatroniFatalException: 'Failed to bootstrap cluster'

Environment

Please provide the following details:

EXPECTED

  1. Creation of postgres Cluster

ACTUAL

  1. Creation of 2 Volumes.
  2. Creation of backrest-shared-repo and failing pod 
NAME                                         READY   STATUS             RESTARTS   AGE
postgres-operator-87977ff79-bx5pn            4/4     Running            0          10m
test-6b58b5f75c-h4g77                        1/2     CrashLoopBackOff   6          9m13s
test-backrest-shared-repo-6cc6cb87c6-j8m98   1/1     Running            0          9m37s

## kubectl describe pod -n pgo test-6b58b5f75c-h4g77  
Name:         test-6b58b5f75c-h4g77
Namespace:    pgo
Priority:     0
Node:         int-a-hc-workers-base-mt-v1.0.0-sdb2d/10.0.0.6
Start Time:   Thu, 29 Apr 2021 11:43:55 +0200
Labels:       crunchy-pgha-scope=test
              crunchy-postgres-exporter=true
              deployment-name=test
              name=test
              pg-cluster=test
              pg-pod-anti-affinity=preferred
              pgo-pg-database=true
              pgo-version=4.6.2
              pgouser=admin
              pod-template-hash=6b58b5f75c
              service-name=test
              vendor=crunchydata
Annotations:  status:
                {"conn_url":"postgres://10.244.3.136:5432/postgres","api_url":"http://10.244.3.136:8009/patroni","state":"stopped","role":"uninitialized",...
Status:       Running
IP:           10.244.3.136
IPs:
  IP:           10.244.3.136
Controlled By:  ReplicaSet/test-6b58b5f75c
Containers:
  database:
    Container ID:   cri-o://35a8bdeb4ab6293677cab80dfd3102c1a096f0435325e9d360235c0a36ba75ef
    Image:          registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.2-4.6.2
    Image ID:       registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha@sha256:0c1f78a403d4ab4372f9bfbce2447f4c2de00d78690b46aaa84f3b09f32d47b9
    Ports:          5432/TCP, 8009/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 29 Apr 2021 11:54:55 +0200
      Finished:     Thu, 29 Apr 2021 11:54:56 +0200
    Ready:          False
    Restart Count:  7
    Requests:
      memory:   128Mi
    Liveness:   exec [/opt/crunchy/bin/postgres-ha/health/pgha-liveness.sh] delay=30s timeout=10s period=15s #success=1 #failure=3
    Readiness:  exec [/opt/crunchy/bin/postgres-ha/health/pgha-readiness.sh] delay=15s timeout=1s period=10s #success=1 #failure=3
    Environment:
      MODE:                               postgres
      PGHA_PG_PORT:                       5432
      PGHA_USER:                          postgres
      PGHA_INIT:                          <set to the key 'init' of config map 'test-pgha-config'>  Optional: false
      PATRONI_POSTGRESQL_DATA_DIR:        /pgdata/test
      PGBACKREST_STANZA:                  db
      PGBACKREST_REPO1_HOST:              test-backrest-shared-repo
      BACKREST_SKIP_CREATE_STANZA:        true
      PGHA_PGBACKREST:                    true
      PGBACKREST_REPO1_PATH:              /backrestrepo/test-backrest-shared-repo
      PGBACKREST_DB_PATH:                 /pgdata/test
      ENABLE_SSHD:                        true
      PGBACKREST_LOG_PATH:                /tmp
      PGBACKREST_PG1_SOCKET_PATH:         /tmp
      PGBACKREST_PG1_PORT:                5432
      PGBACKREST_REPO1_TYPE:              posix
      PGHA_PGBACKREST_LOCAL_S3_STORAGE:   false
      PGMONITOR_PASSWORD:                 <set to the key 'password' in secret 'test-exporter-secret'>  Optional: false
      PGHA_DATABASE:                      test
      PGHA_REPLICA_REINIT_ON_START_FAIL:  true
      PGHA_SYNC_REPLICATION:              false
      PGHA_TLS_ENABLED:                   false
      PGHA_TLS_ONLY:                      false
      PGHA_STANDBY:                       false
      PATRONI_KUBERNETES_NAMESPACE:       pgo (v1:metadata.namespace)
      PATRONI_KUBERNETES_SCOPE_LABEL:     crunchy-pgha-scope
      PATRONI_SCOPE:                       (v1:metadata.labels['crunchy-pgha-scope'])
      PATRONI_KUBERNETES_LABELS:          {vendor: "crunchydata"}
      PATRONI_LOG_LEVEL:                  INFO
      PGHOST:                             /tmp
    Mounts:
      /dev/shm from dshm (rw)
      /etc/pgbackrest/conf.d from pgbackrest-config (rw)
      /etc/podinfo from podinfo (rw)
      /pgconf from pgconf-volume (rw)
      /pgconf/pgreplicator from primary-volume (rw)
      /pgconf/pgsuper from root-volume (rw)
      /pgconf/pguser from user-volume (rw)
      /pgdata from pgdata (rw)
      /sshd from sshd (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sjgsb (ro)
  exporter:
    Container ID:   cri-o://0ebc2a93a2d4402e0535e7e2ebca43f20c99a7499c4e9b5eefecb478742f38dd
    Image:          registry.developers.crunchydata.com/crunchydata/crunchy-postgres-exporter:centos8-4.6.2
    Image ID:       registry.developers.crunchydata.com/crunchydata/crunchy-postgres-exporter@sha256:f2b90fac6db77686e929b6620db51c7fb6a17e1f675de57ca433b9f62d00f54e
    Port:           9187/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 29 Apr 2021 11:44:09 +0200
    Ready:          True
    Restart Count:  0
    Requests:
      memory:  24Mi
    Environment:
      EXPORTER_PG_HOST:        127.0.0.1
      EXPORTER_PG_PORT:        5432
      EXPORTER_PG_DATABASE:    postgres
      EXPORTER_PG_PARAMS:      sslmode=disable
      JOB_NAME:                test
      POSTGRES_EXPORTER_PORT:  9187
      EXPORTER_PG_USER:        <set to the key 'username' in secret 'test-exporter-secret'>  Optional: false
      EXPORTER_PG_PASSWORD:    <set to the key 'password' in secret 'test-exporter-secret'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sjgsb (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  pgdata:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  test
    ReadOnly:   false
  user-volume:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  test-app-secret
    Optional:    false
  primary-volume:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  test-primaryuser-secret
    Optional:    false
  sshd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  test-backrest-repo-config
    Optional:    false
  root-volume:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  test-postgres-secret
    Optional:    false
  report:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  64Mi
  dshm:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  pgbackrest-config:
    Type:                Projected (a volume that contains injected data from multiple sources)
    ConfigMapName:       test-config-backrest
    ConfigMapOptional:   0xc000a003a9
    SecretName:          test-config-backrest
    SecretOptionalName:  0xc000a003aa
  pgconf-volume:
    Type:               Projected (a volume that contains injected data from multiple sources)
    ConfigMapName:      test-pgha-config
    ConfigMapOptional:  0xc000a003bd
  podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      limits.cpu -> cpu_limit
      requests.cpu -> cpu_request
      limits.memory -> mem_limit
      requests.memory -> mem_request
      metadata.labels -> labels
      metadata.annotations -> annotations
  kube-api-access-sjgsb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute for 45s
                             node.kubernetes.io/unreachable:NoExecute for 45s
Events:
  Type     Reason                  Age                 From                                            Message
  ----     ------                  ----                ----                                            -------
  Normal   Scheduled               <unknown>                                                           Successfully assigned pgo/test-6b58b5f75c-h4g77 to int-a-hc-workers-base-mt-v1.0.0-sdb2d
  Normal   SuccessfulAttachVolume  11m                 attachdetach-controller                         AttachVolume.Attach succeeded for volume "pvc-9592761d-71c4-4a4c-8269-9dc23f337c54"
  Normal   Pulled                  10m                 kubelet, int-a-hc-workers-base-mt-v1.0.0-sdb2d  Successfully pulled image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.2-4.6.2" in 1.587501291s
  Normal   Pulling                 10m                 kubelet, int-a-hc-workers-base-mt-v1.0.0-sdb2d  Pulling image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres-exporter:centos8-4.6.2"
  Normal   Pulled                  10m                 kubelet, int-a-hc-workers-base-mt-v1.0.0-sdb2d  Successfully pulled image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres-exporter:centos8-4.6.2" in 1.65495558s
  Normal   Created                 10m                 kubelet, int-a-hc-workers-base-mt-v1.0.0-sdb2d  Created container exporter
  Normal   Started                 10m                 kubelet, int-a-hc-workers-base-mt-v1.0.0-sdb2d  Started container exporter
  Normal   Pulled                  10m                 kubelet, int-a-hc-workers-base-mt-v1.0.0-sdb2d  Successfully pulled image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.2-4.6.2" in 1.590705513s
  Normal   Pulled                  10m                 kubelet, int-a-hc-workers-base-mt-v1.0.0-sdb2d  Successfully pulled image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.2-4.6.2" in 1.459449915s
  Normal   Pulling                 10m (x4 over 10m)   kubelet, int-a-hc-workers-base-mt-v1.0.0-sdb2d  Pulling image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.2-4.6.2"
  Normal   Started                 10m (x4 over 10m)   kubelet, int-a-hc-workers-base-mt-v1.0.0-sdb2d  Started container database
  Normal   Created                 10m (x4 over 10m)   kubelet, int-a-hc-workers-base-mt-v1.0.0-sdb2d  Created container database
  Normal   Pulled                  10m                 kubelet, int-a-hc-workers-base-mt-v1.0.0-sdb2d  Successfully pulled image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.2-4.6.2" in 1.570792785s
  Warning  BackOff                 50s (x53 over 10m)  kubelet, int-a-hc-workers-base-mt-v1.0.0-sdb2d  Back-off restarting failed container

Logs

/opt/crunchy/bin/postgres-ha/bootstrap-postgres-ha.sh: line 198: /pgdata/test.initializing: Permission denied
Thu Apr 29 09:44:11 UTC 2021 INFO: postgres-ha pre-bootstrap starting...
Thu Apr 29 09:44:11 UTC 2021 INFO: pgBackRest auto-config disabled
Thu Apr 29 09:44:11 UTC 2021 INFO: PGHA_PGBACKREST_LOCAL_S3_STORAGE and PGHA_PGBACKREST_INITIALIZE will be ignored if provided
Thu Apr 29 09:44:11 UTC 2021 INFO: Defaults have been set for the following postgres-ha auto-configuration env vars: PGHA_DEFAULT_CONFIG, PGHA_BASE_BOOTSTRAP_CONFIG, PGHA_BASE_PG_CONFIG
Thu Apr 29 09:44:11 UTC 2021 INFO: Defaults have been set for the following postgres-ha env vars: PGHA_PATRONI_PORT
Thu Apr 29 09:44:11 UTC 2021 INFO: Defaults have been set for the following Patroni env vars: PATRONI_NAME, PATRONI_RESTAPI_LISTEN, PATRONI_RESTAPI_CONNECT_ADDRESS, PATRONI_POSTGRESQL_LISTEN, PATRONI_POSTGRESQL_CONNECT_ADDRESS
Thu Apr 29 09:44:11 UTC 2021 INFO: Setting postgres-ha configuration for database user credentials
Thu Apr 29 09:44:11 UTC 2021 INFO: Setting 'pguser' credentials using file system
Thu Apr 29 09:44:11 UTC 2021 INFO: Setting 'superuser' credentials using file system
Thu Apr 29 09:44:11 UTC 2021 INFO: Setting 'replicator' credentials using file system
Thu Apr 29 09:44:11 UTC 2021 INFO: Applying base bootstrap config to postgres-ha configuration
Thu Apr 29 09:44:11 UTC 2021 INFO: Applying base postgres config to postgres-ha configuration
Thu Apr 29 09:44:11 UTC 2021 INFO: Applying pgbackrest config to postgres-ha configuration
Thu Apr 29 09:44:11 UTC 2021 INFO: Applying standard (non-TLS) remote connection configuration to pg_hba.conf
Thu Apr 29 09:44:11 UTC 2021 INFO: Custom postgres-ha configuration file not detected
Thu Apr 29 09:44:11 UTC 2021 INFO: Finished building postgres-ha configuration file '/tmp/postgres-ha-bootstrap.yaml'
mkdir: cannot create directory ‘/pgdata/test’2021-04-29T11:44:11.535107791+02:00 : Permission denied
chmod: cannot access '/pgdata/test'2021-04-29T11:44:11.537762374+02:00 : No such file or directory
Thu Apr 29 09:44:11 UTC 2021 INFO: postgres-ha pre-bootstrap complete!  The following configuration will be utilized to initialize 
******************************
postgres-ha (PGHA) env vars:
******************************
PGHA_BASE_PG_CONFIG=true
PGHA_PATRONI_PORT=8009
PGHA_PG_PORT=5432
PGHA_PGBACKREST_LOCAL_S3_STORAGE=false
PGHA_SYNC_REPLICATION=false
PGHA_USER=postgres
PGHA_INIT=true
PGHA_DEFAULT_CONFIG=true
PGHA_REPLICA_REINIT_ON_START_FAIL=true
PGHA_PGBACKREST=true
PGHA_BASE_BOOTSTRAP_CONFIG=true
PGHA_STANDBY=false
PGHA_TLS_ONLY=false
PGHA_TLS_ENABLED=false
PGHA_DATABASE=test
******************************
Patroni env vars:
******************************
PATRONI_POSTGRESQL_CONNECT_ADDRESS=10.244.3.136:5432
PATRONI_POSTGRESQL_LISTEN=0.0.0.0:5432
PATRONI_NAME=test-6b58b5f75c-h4g77
PATRONI_SCOPE=test
PATRONI_RESTAPI_LISTEN=0.0.0.0:8009
PATRONI_POSTGRESQL_DATA_DIR=/pgdata/test
PATRONI_RESTAPI_CONNECT_ADDRESS=10.244.3.136:8009
PATRONI_LOG_LEVEL=INFO
PATRONI_KUBERNETES_LABELS={vendor: "crunchydata"}
PATRONI_KUBERNETES_SCOPE_LABEL=crunchy-pgha-scope
PATRONI_KUBERNETES_NAMESPACE=pgo
******************************
Patroni bootstrap method: initdb
******************************
Patroni configuration file:
******************************
bootstrap:
  method: initdb
  pgbackrest_init:
    command: '/opt/crunchy/bin/postgres-ha/pgbackrest/pgbackrest-create-replica.sh
      primary'
    keep_existing_recovery_conf: true
  existing_init:
    command: '/opt/crunchy/bin/postgres-ha/bootstrap/create-from-existing.sh'
    keep_existing_recovery_conf: true
  dcs:
    postgresql:
      parameters:
        jit: off
        unix_socket_directories: /tmp
        wal_level: logical
        archive_mode: on
        archive_command: 'source /opt/crunchy/bin/postgres-ha/pgbackrest/pgbackrest-set-env.sh
          && pgbackrest archive-push "%p"'
        archive_timeout: 60
        log_directory: pg_log
        shared_buffers: 128MB
        temp_buffers: 8MB
        log_min_duration_statement: 60000
        log_statement: none
        work_mem: 4MB
        max_wal_senders: 6
        shared_preload_libraries: pgaudit.so,pg_stat_statements.so,pgnodemx.so
      use_slots: false
      recovery_conf:
        restore_command: 'source /opt/crunchy/bin/postgres-ha/pgbackrest/pgbackrest-set-env.sh
          && pgbackrest archive-get %f "%p"'
      use_pg_rewind: true
  post_bootstrap: /opt/crunchy/bin/postgres-ha/bootstrap/post-bootstrap.sh
  initdb:
  - encoding: UTF8
  - data-checksums
postgresql:
  use_unix_socket: true
  pgpass: /tmp/.pgpass
  create_replica_methods:
  - pgbackrest
  - basebackup
  pgbackrest:
    command: '/opt/crunchy/bin/postgres-ha/pgbackrest/pgbackrest-create-replica.sh
      replica'
    keep_data: true
    no_params: true
  pgbackrest_standby:
    command: '/opt/crunchy/bin/postgres-ha/pgbackrest/pgbackrest-create-replica.sh
      standby'
    keep_data: true
    no_params: true
    no_master: 1
  remove_data_directory_on_rewind_failure: true
  callbacks:
    on_role_change: /opt/crunchy/bin/postgres-ha/callbacks/pgha-on-role-change.sh
  pg_hba:
  - local all postgres peer
  - host replication primaryuser 0.0.0.0/0 md5
  - host all primaryuser 0.0.0.0/0 reject
  - host all all 0.0.0.0/0 md5
Thu Apr 29 09:44:11 UTC 2021 INFO: Applying SSHD..
Thu Apr 29 09:44:11 UTC 2021 INFO: Checking for SSH Host Keys in /sshd..
Thu Apr 29 09:44:11 UTC 2021 INFO: Checking for authorized_keys in /sshd
Thu Apr 29 09:44:11 UTC 2021 INFO: Checking for sshd_config in /sshd
Thu Apr 29 09:44:11 UTC 2021 INFO: setting up .ssh directory
Thu Apr 29 09:44:11 UTC 2021 INFO: Starting SSHD..
WARNING: 'UsePAM no' is not supported in Fedora and may cause several problems.
Thu Apr 29 09:44:11 UTC 2021 INFO: Starting background process to monitor Patroni initization and restart the database if needed
Thu Apr 29 09:44:11 UTC 2021 INFO: Initializing cluster bootstrap with command: '/usr/local/bin/patroni /tmp/postgres-ha-bootstrap.yaml'
Thu Apr 29 09:44:11 UTC 2021 INFO: Running Patroni as PID 1
2021-04-29 09:44:12,001 INFO: No PostgreSQL configuration items changed, nothing to reload.
2021-04-29 09:44:12,008 INFO: Lock owner: None; I am test-6b58b5f75c-h4g77
2021-04-29 09:44:12,026 INFO: trying to bootstrap a new cluster
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf-8".
The default text search configuration will be set to "english".

Data page checksums are enabled.

creating directory /pgdata/test ... 2021-04-29T11:44:12.065710018+02:00 initdb: error: 2021-04-29T11:44:12.065806519+02:00 could not create directory "/pgdata/test": Permission denied
pg_ctl: database system initialization failed
2021-04-29 09:44:12,068 INFO: removing initialize key after failed attempt to bootstrap the cluster
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/patroni/__init__.py", line 138, in patroni_main
    abstract_main(Patroni, schema)
  File "/usr/local/lib/python3.6/site-packages/patroni/daemon.py", line 100, in abstract_main
    controller.run()
  File "/usr/local/lib/python3.6/site-packages/patroni/__init__.py", line 108, in run
    super(Patroni, self).run()
  File "/usr/local/lib/python3.6/site-packages/patroni/daemon.py", line 59, in run
    self._run_cycle()
  File "/usr/local/lib/python3.6/site-packages/patroni/__init__.py", line 111, in _run_cycle
    logger.info(self.ha.run_cycle())
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1457, in run_cycle
    info = self._run_cycle()
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1351, in _run_cycle
    return self.post_bootstrap()
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1247, in post_bootstrap
    self.cancel_initialization()
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1240, in cancel_initialization
    raise PatroniFatalException('Failed to bootstrap cluster')
patroni.exceptions.PatroniFatalException: 'Failed to bootstrap cluster'
time="2021-04-29T09:42:53Z" level=info msg="debug flag set to false"
time="2021-04-29T09:42:58Z" level=info msg="postgres-operator apiserver starts" func="main.main()" file="cmd/apiserver/main.go:111" version=4.6.2
time="2021-04-29T09:42:58Z" level=info msg="Pgo Namespace is [pgo]" func="internal/apiserver.Initialize()" file="internal/apiserver/root.go:100" version=4.6.2
time="2021-04-29T09:42:58Z" level=info msg="InstallationName is [devtest]" func="internal/apiserver.Initialize()" file="internal/apiserver/root.go:107" version=4.6.2
time="2021-04-29T09:42:58Z" level=info msg="apiserver starts" func="internal/apiserver.Initialize()" file="internal/apiserver/root.go:119" version=4.6.2
time="2021-04-29T09:42:58Z" level=info msg="loading PermMap with 56 Permissions\n" func="internal/apiserver.initializePerms()" file="internal/apiserver/perms.go:179" version=4.6.2
time="2021-04-29T09:42:58Z" level=info msg="Config: \"pgo-config\" ConfigMap found, using config files from the configmap" func="internal/config.initialize()" file="internal/config/pgoconfig.go:733" version=4.6.2
I0429 09:42:59.081754       1 request.go:655] Throttling request took 1.013788534s, request: GET:https://10.96.0.1:443/apis/cluster.x-k8s.io/v1alpha3?timeout=32s
time="2021-04-29T09:43:00Z" level=info msg="default instance memory set to [128Mi]" func="internal/config.(*PgoConfig).Validate()" file="internal/config/pgoconfig.go:380" version=4.6.2
time="2021-04-29T09:43:00Z" level=info msg="default pgbackrest repository memory set to [48Mi]" func="internal/config.(*PgoConfig).Validate()" file="internal/config/pgoconfig.go:386" version=4.6.2
time="2021-04-29T09:43:00Z" level=info msg="default pgbouncer memory set to [24Mi]" func="internal/config.(*PgoConfig).Validate()" file="internal/config/pgoconfig.go:392" version=4.6.2
time="2021-04-29T09:43:00Z" level=info msg="metrics flag is set to true" func="internal/apiserver.initConfig()" file="internal/apiserver/root.go:172" version=4.6.2
time="2021-04-29T09:43:00Z" level=info msg="badger flag is set to true" func="internal/apiserver.initConfig()" file="internal/apiserver/root.go:176" version=4.6.2
time="2021-04-29T09:43:00Z" level=info msg="BasicAuth is true" func="internal/apiserver.initConfig()" file="internal/apiserver/root.go:190" version=4.6.2
time="2021-04-29T09:43:00Z" level=info msg="Namespace operating mode is 'dynamic'" func="internal/apiserver.Initialize()" file="internal/apiserver/root.go:151" version=4.6.2
time="2021-04-29T09:43:00Z" level=info msg="pgo.tls Secret found in namespace pgo" func="internal/apiserver.WriteTLSCert()" file="internal/apiserver/root.go:418" version=4.6.2
time="2021-04-29T09:43:00Z" level=info msg="cert key data len is 554" func="internal/apiserver.WriteTLSCert()" file="internal/apiserver/root.go:419" version=4.6.2
time="2021-04-29T09:43:00Z" level=info msg="private key data len is 241" func="internal/apiserver.WriteTLSCert()" file="internal/apiserver/root.go:425" version=4.6.2
time="2021-04-29T09:43:00Z" level=info msg="listening on port 8443" func="main.main()" file="cmd/apiserver/main.go:167" version=4.6.2
time="2021-04-29T09:42:54Z" level=info msg="debug flag set to false" func="main.main()" file="cmd/postgres-operator/main.go:55" version=4.6.2
time="2021-04-29T09:42:59Z" level=info msg="CRUNCHY_DEBUG flag set to false" func="internal/operator.Initialize()" file="internal/operator/common.go:97" version=4.6.2
time="2021-04-29T09:42:59Z" level=info msg="NAMESPACE pgo,default,test" func="internal/operator.Initialize()" file="internal/operator/common.go:101" version=4.6.2
time="2021-04-29T09:42:59Z" level=info msg="InstallationName devtest" func="internal/operator.Initialize()" file="internal/operator/common.go:104" version=4.6.2
time="2021-04-29T09:42:59Z" level=info msg="Config: \"pgo-config\" ConfigMap found, using config files from the configmap" func="internal/config.initialize()" file="internal/config/pgoconfig.go:733" version=4.6.2
I0429 09:43:00.704362       1 request.go:655] Throttling request took 1.015609577s, request: GET:https://10.96.0.1:443/apis/cluster-api-provider-hcloud.capihc.com/v1alpha3?timeout=32s
time="2021-04-29T09:43:01Z" level=info msg="default instance memory set to [128Mi]" func="internal/config.(*PgoConfig).Validate()" file="internal/config/pgoconfig.go:380" version=4.6.2
time="2021-04-29T09:43:01Z" level=info msg="default pgbackrest repository memory set to [48Mi]" func="internal/config.(*PgoConfig).Validate()" file="internal/config/pgoconfig.go:386" version=4.6.2
time="2021-04-29T09:43:01Z" level=info msg="default pgbouncer memory set to [24Mi]" func="internal/config.(*PgoConfig).Validate()" file="internal/config/pgoconfig.go:392" version=4.6.2
time="2021-04-29T09:43:01Z" level=info msg="EventTCPAddress set to localhost:4150" func="internal/operator.Initialize()" file="internal/operator/common.go:154" version=4.6.2
time="2021-04-29T09:43:02Z" level=info msg="Configure install namespaces: namespace pgo will be updated to be owned by this installation" func="internal/ns.ConfigureInstallNamespaces()" file="internal/ns/nslogic.go:511" version=4.6.2
2021/04/29 09:43:02 INF    1 (localhost:4150) connecting to nsqd
time="2021-04-29T09:43:02Z" level=info msg="Configure install namespaces: namespace default will be updated to be owned by this installation" func="internal/ns.ConfigureInstallNamespaces()" file="internal/ns/nslogic.go:511" version=4.6.2
2021/04/29 09:43:02 INF    2 (localhost:4150) connecting to nsqd
time="2021-04-29T09:43:02Z" level=info msg="Configure install namespaces: namespace test will be updated to be owned by this installation" func="internal/ns.ConfigureInstallNamespaces()" file="internal/ns/nslogic.go:511" version=4.6.2
2021/04/29 09:43:02 INF    3 (localhost:4150) connecting to nsqd
I0429 09:43:02.931842       1 shared_informer.go:240] Waiting for caches to sync for namespace
I0429 09:43:03.032409       1 shared_informer.go:247] Caches are synced for namespace 
time="2021-04-29T09:43:03Z" level=info msg="PostgreSQL Operator initialized and running, waiting for signal to exit" func="main.main()" file="cmd/postgres-operator/main.go:125" version=4.6.2
I0429 09:43:03.385889       1 shared_informer.go:240] Waiting for caches to sync for test
I0429 09:43:03.586196       1 shared_informer.go:247] Caches are synced for test 
I0429 09:43:03.621055       1 shared_informer.go:240] Waiting for caches to sync for pgo
I0429 09:43:03.721380       1 shared_informer.go:247] Caches are synced for pgo 
I0429 09:43:03.745953       1 shared_informer.go:240] Waiting for caches to sync for default
I0429 09:43:03.846211       1 shared_informer.go:247] Caches are synced for default 
time="2021-04-29T09:43:27Z" level=info msg="user secret \"test-app-secret\" exists for user \"app\" for cluster \"test\"" func="internal/operator/cluster.createMissingUserSecret()" file="internal/operator/cluster/cluster.go:698" version=4.6.2
time="2021-04-29T09:43:27Z" level=info msg="created PVC =test-pgbr-repo in namespace pgo" func="internal/operator/pvc.CreatePVC()" file="internal/operator/pvc/pvc.go:128" version=4.6.2
time="2021-04-29T09:43:27Z" level=info msg="creating Pgcluster test in namespace pgo" func="internal/operator/cluster.getClusterDeploymentFields()" file="internal/operator/cluster/clusterlogic.go:264" version=4.6.2
2021/04/29 09:43:27 INF    4 (localhost:4150) connecting to nsqd
jkatz commented 3 years ago

creating directory /pgdata/test ... 2021-04-29T11:44:12.065710018+02:00 initdb: error: 2021-04-29T11:44:12.065806519+02:00 could not create directory "/pgdata/test": Permission denied pg_ctl: database system initialization failed

This appears to a configuration error at some level of your stack, likely with your storage or how you are interfacing with your storage. I'm not sure how Hetzner CSI handles provisioning and file ownership. You may need to toggle the DisableFSGroup setting, or you may need to adjust how you configured the storage configuration that handles provisioning with Hetzner CSI.

Anyway, I don't see anything at this point that indicates a bug with PGO.