search

zalando / postgres-operator

Postgres operator creates and manages PostgreSQL clusters running in Kubernetes
https://postgres-operator.readthedocs.io/
MIT License
4.22k stars 968 forks source link

Readiness probe failed: HTTP probe failed with statuscode: 503 port 8008 in POD #1974

Open Deepanshu6dec opened 2 years ago

Deepanshu6dec commented 2 years ago

Hi, Following this quick Guide, as soon as we deploy postgresql, Pod status is went to not ready state.

  Normal   Scheduled         7m42s                   default-scheduler  Successfully assigned default/acid-minimal-cluster-1 to ip-172-31-6-72.us-east-2.compute.internal
  Normal   Pulled            7m42s                   kubelet            Container image "registry.opensource.zalan.do/acid/spilo-14:2.1-p6" already present on machine
  Normal   Created           7m42s                   kubelet            Created container postgres
  Normal   Started           7m42s                   kubelet            Started container postgres
  Warning  Unhealthy         **2m52s (x13 over 6m42s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503**

Manual deployment setup on Kubernetes

First, clone the repository and change to the directory

git clone https://github.com/zalando/postgres-operator.git cd postgres-operator

apply the manifests in the following order

kubectl create -f manifests/configmap.yaml # configuration kubectl create -f manifests/operator-service-account-rbac.yaml # identity and permissions kubectl create -f manifests/postgres-operator.yaml # deployment kubectl create -f manifests/api-service.yaml # operator API to be used by UI

create a Postgres cluster

kubectl create -f manifests/minimal-postgres-manifest.yaml


\NAME                                     READY   STATUS    RESTARTS   AGE     IP             NODE                                        NOMINATED NODE   READINESS GATES
pod/acid-minimal-cluster-0               0/1     Running   0          9m18s   10.244.2.128   ip-172-31-6-72.us-east-2.compute.internal   <none>           <none>
pod/acid-minimal-cluster-1               0/1     Running   0          9m11s   10.244.2.129   ip-172-31-6-72.us-east-2.compute.internal   <none>           <none>
pod/postgres-operator-6b8c5c9656-tr6zf   1/1     Running   0          10m     10.244.2.127   ip-172-31-6-72.us-east-2.compute.internal   <none>           <none>

NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE     SELECTOR
service/acid-minimal-cluster          ClusterIP   10.96.128.61    <none>        5432/TCP   9m19s   <none>
service/acid-minimal-cluster-config   ClusterIP   None            <none>        <none>     9m16s   <none>
service/acid-minimal-cluster-repl     ClusterIP   10.96.166.233   <none>        5432/TCP   9m19s   application=spilo,cluster-name=acid-minimal-cluster,spilo-role=replica
service/kubernetes                    ClusterIP   10.96.0.1       <none>        443/TCP    10m     <none>
service/postgres-operator             ClusterIP   10.96.27.45     <none>        8080/TCP   10m     name=postgres-operator

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS          IMAGES                                                       SELECTOR
deployment.apps/postgres-operator   1/1     1            1           10m   postgres-operator   registry.opensource.zalan.do/acid/postgres-operator:v1.8.2   name=postgres-operator

NAME                                           DESIRED   CURRENT   READY   AGE   CONTAINERS          IMAGES                                                       SELECTOR
replicaset.apps/postgres-operator-6b8c5c9656   1         1         1       10m   postgres-operator   registry.opensource.zalan.do/acid/postgres-operator:v1.8.2   name=postgres-operator,pod-template-hash=6b8c5c9656

NAME                                    READY   AGE     CONTAINERS   IMAGES
statefulset.apps/acid-minimal-cluster   0/2     9m19s   postgres     registry.opensource.zalan.do/acid/spilo-14:2.1-p6

NAME                                            TEAM   VERSION   PODS   VOLUME   CPU-REQUEST   MEMORY-REQUEST   AGE     STATUS
postgresql.acid.zalan.do/acid-minimal-cluster   acid   14        2      1Gi                                     9m19s   Running
Deepanshu6dec commented 2 years ago

keep working on this issue and found after 15 mins reediness probe and there is no response

before this logs everything is working fine and probe is working fine but now getting issue on all three replica Logs:

2022-07-25 16:18:17,827 INFO: no action. I am (smsc-cluster-0), the leader with the lock
2022-07-25 16:18:26.390 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:18:27,822 INFO: no action. I am (smsc-cluster-0), the leader with the lock
2022-07-25 16:18:37,772 INFO: no action. I am (smsc-cluster-0), the leader with the lock
2022-07-25 16:18:47,823 INFO: no action. I am (smsc-cluster-0), the leader with the lock
2022-07-25 16:18:56.398 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:18:57,770 INFO: no action. I am (smsc-cluster-0), the leader with the lock
2022-07-25 16:19:07,724 WARNING: Postgresql is not running.
2022-07-25 16:19:07,724 INFO: Lock owner: smsc-cluster-0; I am smsc-cluster-0
2022-07-25 16:19:07,774 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in production
  pg_control last modified: Mon Jul 25 16:18:37 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:19:07,847 INFO: doing crash recovery in a single user mode
2022-07-25 16:19:07,895 ERROR: Crash recovery finished with code=1
2022-07-25 16:19:07,896 INFO:  stdout=
2022-07-25 16:19:07,896 INFO:  stderr=2022-07-25 16:19:07 UTC [7740]: [1-1] 62dec27b.1e3c 0     LOG:  database system was interrupted; last known up at 2022-07-25 16:18:37 UTC
2022-07-25 16:19:07 UTC [7740]: [2-1] 62dec27b.1e3c 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:19:07 UTC [7740]: [3-1] 62dec27b.1e3c 0     LOG:  redo starts at 0/C136018
2022-07-25 16:19:07 UTC [7740]: [4-1] 62dec27b.1e3c 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:19:07 UTC [7740]: [5-1] 62dec27b.1e3c 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:19:17,723 WARNING: Postgresql is not running.
2022-07-25 16:19:17,723 INFO: Lock owner: smsc-cluster-0; I am smsc-cluster-0
2022-07-25 16:19:17,774 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:19:07 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:19:17,775 INFO: Lock owner: smsc-cluster-0; I am smsc-cluster-0
2022-07-25 16:19:17,776 INFO: starting as readonly because i had the session lock
2022-07-25 16:19:17,777 INFO: closed patroni connection to the postgresql cluster
2022-07-25 16:19:18 UTC [7749]: [1-1] 62dec286.1e45 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:19:18 UTC [7749]: [2-1] 62dec286.1e45 0     LOG:  pg_stat_kcache.linux_hz is set to 1000000
2022-07-25 16:19:18,198 INFO: postmaster pid=7749
2022-07-25 16:19:18 UTC [7749]: [3-1] 62dec286.1e45 0     LOG:  could not read file "pg_stat/pg_stat_statements.stat": No such file or directory
2022-07-25 16:19:18 UTC [7749]: [4-1] 62dec286.1e45 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:19:18 UTC [7749]: [5-1] 62dec286.1e45 0     HINT:  Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - no response
2022-07-25 16:19:26.378 34 ERROR connection error: PQconnectStart
2022-07-25 16:19:26.378 34 ERROR 
2022-07-25 16:19:26.378 34 WARNING postgres: default timeout
2022-07-25 16:19:26.378 34 ERROR connection error: PQconnectStart
2022-07-25 16:19:26.378 34 ERROR 
2022-07-25 16:19:26.378 34 ERROR connection error: PQconnectStart
2022-07-25 16:19:26.378 34 ERROR 
2022-07-25 16:19:26.378 34 WARNING csmsc: default timeout
2022-07-25 16:19:26.399 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:19:27,723 INFO: Lock owner: smsc-cluster-0; I am smsc-cluster-0
2022-07-25 16:19:27,830 INFO: removed leader key after trying and failing to start postgres
2022-07-25 16:19:27,830 WARNING: Postgresql is not running.
2022-07-25 16:19:27,830 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:19:27,844 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:19:18 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:19:27,861 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:19:27,862 INFO: starting as a secondary
2022-07-25 16:19:28 UTC [7771]: [1-1] 62dec290.1e5b 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:19:28 UTC [7771]: [2-1] 62dec290.1e5b 0     LOG:  pg_stat_kcache.linux_hz is set to 1000000
2022-07-25 16:19:28,319 INFO: postmaster pid=7771
/var/run/postgresql:5432 - no response
2022-07-25 16:19:28 UTC [7771]: [3-1] 62dec290.1e5b 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:19:28 UTC [7771]: [4-1] 62dec290.1e5b 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:19:29,376 ERROR: postmaster is not running
2022-07-25 16:19:38,336 WARNING: Postgresql is not running.
2022-07-25 16:19:38,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:19:38,341 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:19:28 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:19:38,342 INFO: doing crash recovery in a single user mode
2022-07-25 16:19:38,412 ERROR: Crash recovery finished with code=1
2022-07-25 16:19:38,412 INFO:  stdout=
2022-07-25 16:19:38,412 INFO:  stderr=2022-07-25 16:19:38 UTC [7784]: [1-1] 62dec29a.1e68 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:19:28 UTC
2022-07-25 16:19:38 UTC [7784]: [2-1] 62dec29a.1e68 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:19:38 UTC [7784]: [3-1] 62dec29a.1e68 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:19:38 UTC [7784]: [4-1] 62dec29a.1e68 0     LOG:  redo starts at 0/C136018
2022-07-25 16:19:38 UTC [7784]: [5-1] 62dec29a.1e68 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:19:38 UTC [7784]: [6-1] 62dec29a.1e68 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:19:48,337 WARNING: Postgresql is not running.
2022-07-25 16:19:48,337 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:19:48,343 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:19:38 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:19:48,356 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:19:48,357 INFO: starting as a secondary
2022-07-25 16:19:48,793 INFO: postmaster pid=7795
2022-07-25 16:19:48 UTC [7795]: [1-1] 62dec2a4.1e73 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:19:48 UTC [7795]: [2-1] 62dec2a4.1e73 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
/var/run/postgresql:5432 - no response
2022-07-25 16:19:48 UTC [7795]: [3-1] 62dec2a4.1e73 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:19:48 UTC [7795]: [4-1] 62dec2a4.1e73 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:19:49,814 ERROR: postmaster is not running
2022-07-25 16:19:56.426 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:19:58,338 WARNING: Postgresql is not running.
2022-07-25 16:19:58,338 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:19:58,342 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:19:58 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:19:58,343 INFO: doing crash recovery in a single user mode
2022-07-25 16:19:58,387 ERROR: Crash recovery finished with code=1
2022-07-25 16:19:58,387 INFO:  stdout=
2022-07-25 16:19:58,387 INFO:  stderr=2022-07-25 16:19:58 UTC [7808]: [1-1] 62dec2ae.1e80 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:19:58 UTC
2022-07-25 16:19:58 UTC [7808]: [2-1] 62dec2ae.1e80 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:19:58 UTC [7808]: [3-1] 62dec2ae.1e80 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:19:58 UTC [7808]: [4-1] 62dec2ae.1e80 0     LOG:  redo starts at 0/C136018
2022-07-25 16:19:58 UTC [7808]: [5-1] 62dec2ae.1e80 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:19:58 UTC [7808]: [6-1] 62dec2ae.1e80 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:20:08,339 WARNING: Postgresql is not running.
2022-07-25 16:20:08,339 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:20:08,343 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:19:58 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:20:08,357 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:20:08,358 INFO: starting as a secondary
2022-07-25 16:20:08 UTC [7819]: [1-1] 62dec2b8.1e8b 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:20:08 UTC [7819]: [2-1] 62dec2b8.1e8b 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2022-07-25 16:20:08,815 INFO: postmaster pid=7819
2022-07-25 16:20:08 UTC [7819]: [3-1] 62dec2b8.1e8b 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:20:08 UTC [7819]: [4-1] 62dec2b8.1e8b 0     HINT:  Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - no response
2022-07-25 16:20:09,882 ERROR: postmaster is not running
2022-07-25 16:20:18,337 WARNING: Postgresql is not running.
2022-07-25 16:20:18,337 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:20:18,342 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:20:18 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:20:18,343 INFO: doing crash recovery in a single user mode
2022-07-25 16:20:18,386 ERROR: Crash recovery finished with code=1
2022-07-25 16:20:18,386 INFO:  stdout=
2022-07-25 16:20:18,386 INFO:  stderr=2022-07-25 16:20:18 UTC [7831]: [1-1] 62dec2c2.1e97 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:20:18 UTC
2022-07-25 16:20:18 UTC [7831]: [2-1] 62dec2c2.1e97 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:20:18 UTC [7831]: [3-1] 62dec2c2.1e97 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:20:18 UTC [7831]: [4-1] 62dec2c2.1e97 0     LOG:  redo starts at 0/C136018
2022-07-25 16:20:18 UTC [7831]: [5-1] 62dec2c2.1e97 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:20:18 UTC [7831]: [6-1] 62dec2c2.1e97 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:20:26.390 34 ERROR connection error: PQconnectStart
2022-07-25 16:20:26.390 34 ERROR 
2022-07-25 16:20:26.390 34 WARNING postgres: default timeout
2022-07-25 16:20:26.390 34 ERROR connection error: PQconnectStart
2022-07-25 16:20:26.390 34 ERROR 
2022-07-25 16:20:26.390 34 ERROR connection error: PQconnectStart
2022-07-25 16:20:26.390 34 ERROR 
2022-07-25 16:20:26.390 34 WARNING csmsc: default timeout
2022-07-25 16:20:26.428 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:20:28,337 WARNING: Postgresql is not running.
2022-07-25 16:20:28,337 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:20:28,346 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:20:18 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:20:28,359 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:20:28,360 INFO: starting as a secondary
2022-07-25 16:20:28 UTC [7842]: [1-1] 62dec2cc.1ea2 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:20:28 UTC [7842]: [2-1] 62dec2cc.1ea2 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2022-07-25 16:20:28,821 INFO: postmaster pid=7842
/var/run/postgresql:5432 - no response
2022-07-25 16:20:28 UTC [7842]: [3-1] 62dec2cc.1ea2 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:20:28 UTC [7842]: [4-1] 62dec2cc.1ea2 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:20:29,878 ERROR: postmaster is not running
2022-07-25 16:20:38,340 WARNING: Postgresql is not running.
2022-07-25 16:20:38,340 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:20:38,344 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:20:38 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:20:38,345 INFO: doing crash recovery in a single user mode
2022-07-25 16:20:38,389 ERROR: Crash recovery finished with code=1
2022-07-25 16:20:38,389 INFO:  stdout=
2022-07-25 16:20:38,389 INFO:  stderr=2022-07-25 16:20:38 UTC [7854]: [1-1] 62dec2d6.1eae 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:20:38 UTC
2022-07-25 16:20:38 UTC [7854]: [2-1] 62dec2d6.1eae 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:20:38 UTC [7854]: [3-1] 62dec2d6.1eae 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:20:38 UTC [7854]: [4-1] 62dec2d6.1eae 0     LOG:  redo starts at 0/C136018
2022-07-25 16:20:38 UTC [7854]: [5-1] 62dec2d6.1eae 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:20:38 UTC [7854]: [6-1] 62dec2d6.1eae 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:20:48,337 WARNING: Postgresql is not running.
2022-07-25 16:20:48,337 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:20:48,344 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:20:38 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:20:48,359 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:20:48,360 INFO: starting as a secondary
2022-07-25 16:20:48 UTC [7865]: [1-1] 62dec2e0.1eb9 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:20:48 UTC [7865]: [2-1] 62dec2e0.1eb9 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2022-07-25 16:20:48,824 INFO: postmaster pid=7865
/var/run/postgresql:5432 - no response
2022-07-25 16:20:48 UTC [7865]: [3-1] 62dec2e0.1eb9 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:20:48 UTC [7865]: [4-1] 62dec2e0.1eb9 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:20:49,874 ERROR: postmaster is not running
2022-07-25 16:20:56.434 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:20:58,339 WARNING: Postgresql is not running.
2022-07-25 16:20:58,339 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:20:58,343 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:20:58 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:20:58,344 INFO: doing crash recovery in a single user mode
2022-07-25 16:20:58,387 ERROR: Crash recovery finished with code=1
2022-07-25 16:20:58,387 INFO:  stdout=
2022-07-25 16:20:58,387 INFO:  stderr=2022-07-25 16:20:58 UTC [7877]: [1-1] 62dec2ea.1ec5 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:20:58 UTC
2022-07-25 16:20:58 UTC [7877]: [2-1] 62dec2ea.1ec5 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:20:58 UTC [7877]: [3-1] 62dec2ea.1ec5 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:20:58 UTC [7877]: [4-1] 62dec2ea.1ec5 0     LOG:  redo starts at 0/C136018
2022-07-25 16:20:58 UTC [7877]: [5-1] 62dec2ea.1ec5 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:20:58 UTC [7877]: [6-1] 62dec2ea.1ec5 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:21:08,339 WARNING: Postgresql is not running.
2022-07-25 16:21:08,339 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:21:08,344 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:20:58 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:21:08,359 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:21:08,360 INFO: starting as a secondary
2022-07-25 16:21:08,799 INFO: postmaster pid=7888
2022-07-25 16:21:08 UTC [7888]: [1-1] 62dec2f4.1ed0 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:21:08 UTC [7888]: [2-1] 62dec2f4.1ed0 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2022-07-25 16:21:08 UTC [7888]: [3-1] 62dec2f4.1ed0 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:21:08 UTC [7888]: [4-1] 62dec2f4.1ed0 0     HINT:  Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - no response
2022-07-25 16:21:09,875 ERROR: postmaster is not running
2022-07-25 16:21:18,339 WARNING: Postgresql is not running.
2022-07-25 16:21:18,339 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:21:18,344 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:21:18 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:21:18,344 INFO: doing crash recovery in a single user mode
2022-07-25 16:21:18,387 ERROR: Crash recovery finished with code=1
2022-07-25 16:21:18,387 INFO:  stdout=
2022-07-25 16:21:18,387 INFO:  stderr=2022-07-25 16:21:18 UTC [7900]: [1-1] 62dec2fe.1edc 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:21:18 UTC
2022-07-25 16:21:18 UTC [7900]: [2-1] 62dec2fe.1edc 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:21:18 UTC [7900]: [3-1] 62dec2fe.1edc 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:21:18 UTC [7900]: [4-1] 62dec2fe.1edc 0     LOG:  redo starts at 0/C136018
2022-07-25 16:21:18 UTC [7900]: [5-1] 62dec2fe.1edc 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:21:18 UTC [7900]: [6-1] 62dec2fe.1edc 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:21:26.398 34 ERROR connection error: PQconnectStart
2022-07-25 16:21:26.398 34 ERROR 
2022-07-25 16:21:26.398 34 WARNING postgres: default timeout
2022-07-25 16:21:26.398 34 ERROR connection error: PQconnectStart
2022-07-25 16:21:26.398 34 ERROR 
2022-07-25 16:21:26.398 34 ERROR connection error: PQconnectStart
2022-07-25 16:21:26.398 34 ERROR 
2022-07-25 16:21:26.398 34 WARNING csmsc: default timeout
2022-07-25 16:21:26.435 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:21:28,337 WARNING: Postgresql is not running.
2022-07-25 16:21:28,337 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:21:28,342 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:21:18 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:21:28,356 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:21:28,357 INFO: starting as a secondary
2022-07-25 16:21:28 UTC [7912]: [1-1] 62dec308.1ee8 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:21:28 UTC [7912]: [2-1] 62dec308.1ee8 0     LOG:  pg_stat_kcache.linux_hz is set to 1000000
2022-07-25 16:21:28,814 INFO: postmaster pid=7912
2022-07-25 16:21:28 UTC [7912]: [3-1] 62dec308.1ee8 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:21:28 UTC [7912]: [4-1] 62dec308.1ee8 0     HINT:  Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - no response
2022-07-25 16:21:29,832 ERROR: postmaster is not running
2022-07-25 16:21:38,336 WARNING: Postgresql is not running.
2022-07-25 16:21:38,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:21:38,342 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:21:38 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:21:38,343 INFO: doing crash recovery in a single user mode
2022-07-25 16:21:38,386 ERROR: Crash recovery finished with code=1
2022-07-25 16:21:38,386 INFO:  stdout=
2022-07-25 16:21:38,386 INFO:  stderr=2022-07-25 16:21:38 UTC [7924]: [1-1] 62dec312.1ef4 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:21:38 UTC
2022-07-25 16:21:38 UTC [7924]: [2-1] 62dec312.1ef4 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:21:38 UTC [7924]: [3-1] 62dec312.1ef4 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:21:38 UTC [7924]: [4-1] 62dec312.1ef4 0     LOG:  redo starts at 0/C136018
2022-07-25 16:21:38 UTC [7924]: [5-1] 62dec312.1ef4 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:21:38 UTC [7924]: [6-1] 62dec312.1ef4 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:21:48,337 WARNING: Postgresql is not running.
2022-07-25 16:21:48,337 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:21:48,341 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:21:38 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:21:48,355 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:21:48,356 INFO: starting as a secondary
2022-07-25 16:21:48 UTC [7935]: [1-1] 62dec31c.1eff 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:21:48 UTC [7935]: [2-1] 62dec31c.1eff 0     LOG:  pg_stat_kcache.linux_hz is set to 1000000
2022-07-25 16:21:48,810 INFO: postmaster pid=7935
2022-07-25 16:21:48 UTC [7935]: [3-1] 62dec31c.1eff 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:21:48 UTC [7935]: [4-1] 62dec31c.1eff 0     HINT:  Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - no response
2022-07-25 16:21:49,834 ERROR: postmaster is not running
2022-07-25 16:21:56.449 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:21:58,339 WARNING: Postgresql is not running.
2022-07-25 16:21:58,339 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:21:58,343 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:21:58 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:21:58,344 INFO: doing crash recovery in a single user mode
2022-07-25 16:21:58,385 ERROR: Crash recovery finished with code=1
2022-07-25 16:21:58,386 INFO:  stdout=
2022-07-25 16:21:58,386 INFO:  stderr=2022-07-25 16:21:58 UTC [7947]: [1-1] 62dec326.1f0b 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:21:58 UTC
2022-07-25 16:21:58 UTC [7947]: [2-1] 62dec326.1f0b 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:21:58 UTC [7947]: [3-1] 62dec326.1f0b 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:21:58 UTC [7947]: [4-1] 62dec326.1f0b 0     LOG:  redo starts at 0/C136018
2022-07-25 16:21:58 UTC [7947]: [5-1] 62dec326.1f0b 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:21:58 UTC [7947]: [6-1] 62dec326.1f0b 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:22:08,336 WARNING: Postgresql is not running.
2022-07-25 16:22:08,337 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:22:08,342 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:21:58 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:22:08,355 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:22:08,356 INFO: starting as a secondary
2022-07-25 16:22:08,787 INFO: postmaster pid=7958
2022-07-25 16:22:08 UTC [7958]: [1-1] 62dec330.1f16 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:22:08 UTC [7958]: [2-1] 62dec330.1f16 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
/var/run/postgresql:5432 - no response
2022-07-25 16:22:08 UTC [7958]: [3-1] 62dec330.1f16 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:22:08 UTC [7958]: [4-1] 62dec330.1f16 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:22:09,828 ERROR: postmaster is not running
2022-07-25 16:22:18,338 WARNING: Postgresql is not running.
2022-07-25 16:22:18,338 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:22:18,343 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:22:08 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:22:18,344 INFO: doing crash recovery in a single user mode
2022-07-25 16:22:18,389 ERROR: Crash recovery finished with code=1
2022-07-25 16:22:18,389 INFO:  stdout=
2022-07-25 16:22:18,390 INFO:  stderr=2022-07-25 16:22:18 UTC [7970]: [1-1] 62dec33a.1f22 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:22:08 UTC
2022-07-25 16:22:18 UTC [7970]: [2-1] 62dec33a.1f22 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:22:18 UTC [7970]: [3-1] 62dec33a.1f22 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:22:18 UTC [7970]: [4-1] 62dec33a.1f22 0     LOG:  redo starts at 0/C136018
2022-07-25 16:22:18 UTC [7970]: [5-1] 62dec33a.1f22 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:22:18 UTC [7970]: [6-1] 62dec33a.1f22 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:22:26.429 34 ERROR connection error: PQconnectStart
2022-07-25 16:22:26.429 34 ERROR 
2022-07-25 16:22:26.429 34 WARNING postgres: default timeout
2022-07-25 16:22:26.429 34 ERROR connection error: PQconnectStart
2022-07-25 16:22:26.429 34 ERROR 
2022-07-25 16:22:26.429 34 ERROR connection error: PQconnectStart
2022-07-25 16:22:26.429 34 ERROR 
2022-07-25 16:22:26.429 34 WARNING csmsc: default timeout
2022-07-25 16:22:26.451 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:22:28,336 WARNING: Postgresql is not running.
2022-07-25 16:22:28,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:22:28,340 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:22:18 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:22:28,353 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:22:28,354 INFO: starting as a secondary
2022-07-25 16:22:28 UTC [7981]: [1-1] 62dec344.1f2d 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:22:28 UTC [7981]: [2-1] 62dec344.1f2d 0     LOG:  pg_stat_kcache.linux_hz is set to 1000000
2022-07-25 16:22:28,780 INFO: postmaster pid=7981
/var/run/postgresql:5432 - no response
2022-07-25 16:22:28 UTC [7981]: [3-1] 62dec344.1f2d 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:22:28 UTC [7981]: [4-1] 62dec344.1f2d 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:22:29,797 ERROR: postmaster is not running
2022-07-25 16:22:38,336 WARNING: Postgresql is not running.
2022-07-25 16:22:38,337 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:22:38,341 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:22:38 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:22:38,342 INFO: doing crash recovery in a single user mode
2022-07-25 16:22:38,385 ERROR: Crash recovery finished with code=1
2022-07-25 16:22:38,385 INFO:  stdout=
2022-07-25 16:22:38,385 INFO:  stderr=2022-07-25 16:22:38 UTC [7991]: [1-1] 62dec34e.1f37 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:22:38 UTC
2022-07-25 16:22:38 UTC [7991]: [2-1] 62dec34e.1f37 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:22:38 UTC [7991]: [3-1] 62dec34e.1f37 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:22:38 UTC [7991]: [4-1] 62dec34e.1f37 0     LOG:  redo starts at 0/C136018
2022-07-25 16:22:38 UTC [7991]: [5-1] 62dec34e.1f37 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:22:38 UTC [7991]: [6-1] 62dec34e.1f37 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:22:48,336 WARNING: Postgresql is not running.
2022-07-25 16:22:48,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:22:48,340 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:22:39 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:22:48,353 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:22:48,354 INFO: starting as a secondary
2022-07-25 16:22:48 UTC [8003]: [1-1] 62dec358.1f43 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:22:48 UTC [8003]: [2-1] 62dec358.1f43 0     LOG:  pg_stat_kcache.linux_hz is set to 1000000
2022-07-25 16:22:48,790 INFO: postmaster pid=8003
/var/run/postgresql:5432 - no response
2022-07-25 16:22:48 UTC [8003]: [3-1] 62dec358.1f43 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:22:48 UTC [8003]: [4-1] 62dec358.1f43 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:22:49,812 ERROR: postmaster is not running
2022-07-25 16:22:56.462 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:22:58,336 WARNING: Postgresql is not running.
2022-07-25 16:22:58,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:22:58,340 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:22:58 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:22:58,341 INFO: doing crash recovery in a single user mode
2022-07-25 16:22:58,383 ERROR: Crash recovery finished with code=1
2022-07-25 16:22:58,383 INFO:  stdout=
2022-07-25 16:22:58,383 INFO:  stderr=2022-07-25 16:22:58 UTC [8013]: [1-1] 62dec362.1f4d 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:22:58 UTC
2022-07-25 16:22:58 UTC [8013]: [2-1] 62dec362.1f4d 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:22:58 UTC [8013]: [3-1] 62dec362.1f4d 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:22:58 UTC [8013]: [4-1] 62dec362.1f4d 0     LOG:  redo starts at 0/C136018
2022-07-25 16:22:58 UTC [8013]: [5-1] 62dec362.1f4d 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:22:58 UTC [8013]: [6-1] 62dec362.1f4d 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:23:08,336 WARNING: Postgresql is not running.
2022-07-25 16:23:08,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:23:08,340 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:22:59 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:23:08,353 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:23:08,354 INFO: starting as a secondary
2022-07-25 16:23:08,780 INFO: postmaster pid=8024
2022-07-25 16:23:08 UTC [8024]: [1-1] 62dec36c.1f58 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:23:08 UTC [8024]: [2-1] 62dec36c.1f58 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
/var/run/postgresql:5432 - no response
2022-07-25 16:23:08 UTC [8024]: [3-1] 62dec36c.1f58 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:23:08 UTC [8024]: [4-1] 62dec36c.1f58 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:23:09,801 ERROR: postmaster is not running
2022-07-25 16:23:18,336 WARNING: Postgresql is not running.
2022-07-25 16:23:18,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:23:18,341 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:23:18 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:23:18,341 INFO: doing crash recovery in a single user mode
2022-07-25 16:23:18,383 ERROR: Crash recovery finished with code=1
2022-07-25 16:23:18,384 INFO:  stdout=
2022-07-25 16:23:18,384 INFO:  stderr=2022-07-25 16:23:18 UTC [8034]: [1-1] 62dec376.1f62 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:23:18 UTC
2022-07-25 16:23:18 UTC [8034]: [2-1] 62dec376.1f62 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:23:18 UTC [8034]: [3-1] 62dec376.1f62 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:23:18 UTC [8034]: [4-1] 62dec376.1f62 0     LOG:  redo starts at 0/C136018
2022-07-25 16:23:18 UTC [8034]: [5-1] 62dec376.1f62 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:23:18 UTC [8034]: [6-1] 62dec376.1f62 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:23:26.460 34 ERROR connection error: PQconnectStart
2022-07-25 16:23:26.461 34 ERROR 
2022-07-25 16:23:26.461 34 WARNING postgres: default timeout
2022-07-25 16:23:26.461 34 ERROR connection error: PQconnectStart
2022-07-25 16:23:26.461 34 ERROR 
2022-07-25 16:23:26.461 34 ERROR connection error: PQconnectStart
2022-07-25 16:23:26.461 34 ERROR 
2022-07-25 16:23:26.461 34 WARNING csmsc: default timeout
2022-07-25 16:23:26.463 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:23:28,336 WARNING: Postgresql is not running.
2022-07-25 16:23:28,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:23:28,340 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:23:19 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:23:28,353 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:23:28,354 INFO: starting as a secondary
2022-07-25 16:23:28 UTC [8045]: [1-1] 62dec380.1f6d 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:23:28 UTC [8045]: [2-1] 62dec380.1f6d 0     LOG:  pg_stat_kcache.linux_hz is set to 1000000
2022-07-25 16:23:28,786 INFO: postmaster pid=8045
/var/run/postgresql:5432 - no response
2022-07-25 16:23:28 UTC [8045]: [3-1] 62dec380.1f6d 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:23:28 UTC [8045]: [4-1] 62dec380.1f6d 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:23:29,799 ERROR: postmaster is not running
2022-07-25 16:23:38,336 WARNING: Postgresql is not running.
2022-07-25 16:23:38,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:23:38,340 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:23:38 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:23:38,341 INFO: doing crash recovery in a single user mode
2022-07-25 16:23:38,384 ERROR: Crash recovery finished with code=1
2022-07-25 16:23:38,385 INFO:  stdout=
2022-07-25 16:23:38,385 INFO:  stderr=2022-07-25 16:23:38 UTC [8055]: [1-1] 62dec38a.1f77 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:23:38 UTC
2022-07-25 16:23:38 UTC [8055]: [2-1] 62dec38a.1f77 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:23:38 UTC [8055]: [3-1] 62dec38a.1f77 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:23:38 UTC [8055]: [4-1] 62dec38a.1f77 0     LOG:  redo starts at 0/C136018
2022-07-25 16:23:38 UTC [8055]: [5-1] 62dec38a.1f77 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:23:38 UTC [8055]: [6-1] 62dec38a.1f77 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:23:48,336 WARNING: Postgresql is not running.
2022-07-25 16:23:48,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:23:48,340 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:23:39 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:23:48,353 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:23:48,354 INFO: starting as a secondary
2022-07-25 16:23:48,783 INFO: postmaster pid=8066
2022-07-25 16:23:48 UTC [8066]: [1-1] 62dec394.1f82 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:23:48 UTC [8066]: [2-1] 62dec394.1f82 0     LOG:  pg_stat_kcache.linux_hz is set to 1000000
/var/run/postgresql:5432 - no response
2022-07-25 16:23:48 UTC [8066]: [3-1] 62dec394.1f82 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:23:48 UTC [8066]: [4-1] 62dec394.1f82 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:23:49,797 ERROR: postmaster is not running
2022-07-25 16:23:56.493 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:23:58,336 WARNING: Postgresql is not running.
2022-07-25 16:23:58,337 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:23:58,341 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:23:58 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:23:58,341 INFO: doing crash recovery in a single user mode
2022-07-25 16:23:58,383 ERROR: Crash recovery finished with code=1
2022-07-25 16:23:58,384 INFO:  stdout=
2022-07-25 16:23:58,384 INFO:  stderr=2022-07-25 16:23:58 UTC [8077]: [1-1] 62dec39e.1f8d 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:23:58 UTC
2022-07-25 16:23:58 UTC [8077]: [2-1] 62dec39e.1f8d 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:23:58 UTC [8077]: [3-1] 62dec39e.1f8d 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:23:58 UTC [8077]: [4-1] 62dec39e.1f8d 0     LOG:  redo starts at 0/C136018
2022-07-25 16:23:58 UTC [8077]: [5-1] 62dec39e.1f8d 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:23:58 UTC [8077]: [6-1] 62dec39e.1f8d 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:24:08,336 WARNING: Postgresql is not running.
2022-07-25 16:24:08,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:24:08,340 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:23:59 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:24:08,354 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:24:08,355 INFO: starting as a secondary
2022-07-25 16:24:08 UTC [8088]: [1-1] 62dec3a8.1f98 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:24:08 UTC [8088]: [2-1] 62dec3a8.1f98 0     LOG:  pg_stat_kcache.linux_hz is set to 1000000
2022-07-25 16:24:08 UTC [8088]: [3-1] 62dec3a8.1f98 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:24:08 UTC [8088]: [4-1] 62dec3a8.1f98 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:24:08,811 INFO: postmaster pid=8088
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - no response
2022-07-25 16:24:18,336 WARNING: Postgresql is not running.
2022-07-25 16:24:18,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:24:18,340 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:24:18 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:24:18,341 INFO: doing crash recovery in a single user mode
2022-07-25 16:24:18,384 ERROR: Crash recovery finished with code=1
2022-07-25 16:24:18,384 INFO:  stdout=
2022-07-25 16:24:18,384 INFO:  stderr=2022-07-25 16:24:18 UTC [8102]: [1-1] 62dec3b2.1fa6 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:24:18 UTC
2022-07-25 16:24:18 UTC [8102]: [2-1] 62dec3b2.1fa6 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:24:18 UTC [8102]: [3-1] 62dec3b2.1fa6 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:24:18 UTC [8102]: [4-1] 62dec3b2.1fa6 0     LOG:  redo starts at 0/C136018
2022-07-25 16:24:18 UTC [8102]: [5-1] 62dec3b2.1fa6 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:24:18 UTC [8102]: [6-1] 62dec3b2.1fa6 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:24:26.482 34 ERROR connection error: PQconnectStart
2022-07-25 16:24:26.482 34 ERROR 
2022-07-25 16:24:26.482 34 WARNING postgres: default timeout
2022-07-25 16:24:26.482 34 ERROR connection error: PQconnectStart
2022-07-25 16:24:26.482 34 ERROR 
2022-07-25 16:24:26.483 34 ERROR connection error: PQconnectStart
2022-07-25 16:24:26.483 34 ERROR 
2022-07-25 16:24:26.483 34 WARNING csmsc: default timeout
2022-07-25 16:24:26.495 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:24:28,336 WARNING: Postgresql is not running.
2022-07-25 16:24:28,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:24:28,340 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:24:19 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:24:28,353 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:24:28,354 INFO: starting as a secondary
2022-07-25 16:24:28 UTC [8113]: [1-1] 62dec3bc.1fb1 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:24:28 UTC [8113]: [2-1] 62dec3bc.1fb1 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2022-07-25 16:24:28,777 INFO: postmaster pid=8113
/var/run/postgresql:5432 - no response
2022-07-25 16:24:28 UTC [8113]: [3-1] 62dec3bc.1fb1 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:24:28 UTC [8113]: [4-1] 62dec3bc.1fb1 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:24:29,793 ERROR: postmaster is not running
2022-07-25 16:24:38,336 WARNING: Postgresql is not running.
2022-07-25 16:24:38,337 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:24:38,341 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:24:38 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:24:38,341 INFO: doing crash recovery in a single user mode
2022-07-25 16:24:38,383 ERROR: Crash recovery finished with code=1
2022-07-25 16:24:38,383 INFO:  stdout=
2022-07-25 16:24:38,383 INFO:  stderr=2022-07-25 16:24:38 UTC [8123]: [1-1] 62dec3c6.1fbb 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:24:38 UTC
2022-07-25 16:24:38 UTC [8123]: [2-1] 62dec3c6.1fbb 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:24:38 UTC [8123]: [3-1] 62dec3c6.1fbb 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:24:38 UTC [8123]: [4-1] 62dec3c6.1fbb 0     LOG:  redo starts at 0/C136018
2022-07-25 16:24:38 UTC [8123]: [5-1] 62dec3c6.1fbb 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:24:38 UTC [8123]: [6-1] 62dec3c6.1fbb 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:24:48,336 WARNING: Postgresql is not running.
2022-07-25 16:24:48,337 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:24:48,341 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:24:39 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:24:48,355 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:24:48,356 INFO: starting as a secondary
2022-07-25 16:24:48,775 INFO: postmaster pid=8134
2022-07-25 16:24:48 UTC [8134]: [1-1] 62dec3d0.1fc6 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:24:48 UTC [8134]: [2-1] 62dec3d0.1fc6 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
/var/run/postgresql:5432 - no response
2022-07-25 16:24:48 UTC [8134]: [3-1] 62dec3d0.1fc6 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:24:48 UTC [8134]: [4-1] 62dec3d0.1fc6 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:24:49,794 ERROR: postmaster is not running
2022-07-25 16:24:56.502 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:24:58,336 WARNING: Postgresql is not running.
2022-07-25 16:24:58,337 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:24:58,341 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:24:58 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:24:58,341 INFO: doing crash recovery in a single user mode
2022-07-25 16:24:58,383 ERROR: Crash recovery finished with code=1
2022-07-25 16:24:58,383 INFO:  stdout=
2022-07-25 16:24:58,383 INFO:  stderr=2022-07-25 16:24:58 UTC [8144]: [1-1] 62dec3da.1fd0 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:24:58 UTC
2022-07-25 16:24:58 UTC [8144]: [2-1] 62dec3da.1fd0 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:24:58 UTC [8144]: [3-1] 62dec3da.1fd0 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:24:58 UTC [8144]: [4-1] 62dec3da.1fd0 0     LOG:  redo starts at 0/C136018
2022-07-25 16:24:58 UTC [8144]: [5-1] 62dec3da.1fd0 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:24:58 UTC [8144]: [6-1] 62dec3da.1fd0 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:25:08,337 WARNING: Postgresql is not running.
2022-07-25 16:25:08,337 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:25:08,341 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:24:59 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:25:08,356 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:25:08,356 INFO: starting as a secondary
2022-07-25 16:25:08,788 INFO: postmaster pid=8155
2022-07-25 16:25:08 UTC [8155]: [1-1] 62dec3e4.1fdb 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:25:08 UTC [8155]: [2-1] 62dec3e4.1fdb 0     LOG:  pg_stat_kcache.linux_hz is set to 1000000
/var/run/postgresql:5432 - no response
2022-07-25 16:25:08 UTC [8155]: [3-1] 62dec3e4.1fdb 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:25:08 UTC [8155]: [4-1] 62dec3e4.1fdb 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:25:09,811 ERROR: postmaster is not running
2022-07-25 16:25:18,336 WARNING: Postgresql is not running.
2022-07-25 16:25:18,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:25:18,340 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:25:18 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:25:18,341 INFO: doing crash recovery in a single user mode
2022-07-25 16:25:18,384 ERROR: Crash recovery finished with code=1
2022-07-25 16:25:18,384 INFO:  stdout=
2022-07-25 16:25:18,384 INFO:  stderr=2022-07-25 16:25:18 UTC [8166]: [1-1] 62dec3ee.1fe6 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:25:18 UTC
2022-07-25 16:25:18 UTC [8166]: [2-1] 62dec3ee.1fe6 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:25:18 UTC [8166]: [3-1] 62dec3ee.1fe6 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:25:18 UTC [8166]: [4-1] 62dec3ee.1fe6 0     LOG:  redo starts at 0/C136018
2022-07-25 16:25:18 UTC [8166]: [5-1] 62dec3ee.1fe6 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:25:18 UTC [8166]: [6-1] 62dec3ee.1fe6 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:25:26.493 34 ERROR connection error: PQconnectStart
2022-07-25 16:25:26.493 34 ERROR 
2022-07-25 16:25:26.493 34 WARNING postgres: default timeout
2022-07-25 16:25:26.493 34 ERROR connection error: PQconnectStart
2022-07-25 16:25:26.493 34 ERROR 
2022-07-25 16:25:26.493 34 ERROR connection error: PQconnectStart
2022-07-25 16:25:26.493 34 ERROR 
2022-07-25 16:25:26.493 34 WARNING csmsc: default timeout
2022-07-25 16:25:26.503 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:25:28,336 WARNING: Postgresql is not running.
2022-07-25 16:25:28,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:25:28,341 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:25:19 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:25:28,355 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:25:28,356 INFO: starting as a secondary
2022-07-25 16:25:28 UTC [8177]: [1-1] 62dec3f8.1ff1 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:25:28 UTC [8177]: [2-1] 62dec3f8.1ff1 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2022-07-25 16:25:28,792 INFO: postmaster pid=8177
2022-07-25 16:25:28 UTC [8177]: [3-1] 62dec3f8.1ff1 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:25:28 UTC [8177]: [4-1] 62dec3f8.1ff1 0     HINT:  Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - no response
2022-07-25 16:25:29,874 ERROR: postmaster is not running
2022-07-25 16:25:38,336 WARNING: Postgresql is not running.
2022-07-25 16:25:38,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:25:38,340 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:25:38 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:25:38,341 INFO: doing crash recovery in a single user mode
2022-07-25 16:25:38,384 ERROR: Crash recovery finished with code=1
2022-07-25 16:25:38,384 INFO:  stdout=
2022-07-25 16:25:38,384 INFO:  stderr=2022-07-25 16:25:38 UTC [8187]: [1-1] 62dec402.1ffb 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:25:38 UTC
2022-07-25 16:25:38 UTC [8187]: [2-1] 62dec402.1ffb 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:25:38 UTC [8187]: [3-1] 62dec402.1ffb 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:25:38 UTC [8187]: [4-1] 62dec402.1ffb 0     LOG:  redo starts at 0/C136018
2022-07-25 16:25:38 UTC [8187]: [5-1] 62dec402.1ffb 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:25:38 UTC [8187]: [6-1] 62dec402.1ffb 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:25:48,336 WARNING: Postgresql is not running.
2022-07-25 16:25:48,337 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:25:48,341 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:25:39 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:25:48,354 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:25:48,355 INFO: starting as a secondary
2022-07-25 16:25:48,789 INFO: postmaster pid=8198
2022-07-25 16:25:48 UTC [8198]: [1-1] 62dec40c.2006 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:25:48 UTC [8198]: [2-1] 62dec40c.2006 0     LOG:  pg_stat_kcache.linux_hz is set to 1000000
/var/run/postgresql:5432 - no response
2022-07-25 16:25:48 UTC [8198]: [3-1] 62dec40c.2006 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:25:48 UTC [8198]: [4-1] 62dec40c.2006 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:25:49,803 ERROR: postmaster is not running
2022-07-25 16:25:56.527 34 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-25 16:25:58,336 WARNING: Postgresql is not running.
2022-07-25 16:25:58,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:25:58,340 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:25:58 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:25:58,341 INFO: doing crash recovery in a single user mode
2022-07-25 16:25:58,383 ERROR: Crash recovery finished with code=1
2022-07-25 16:25:58,384 INFO:  stdout=
2022-07-25 16:25:58,384 INFO:  stderr=2022-07-25 16:25:58 UTC [8208]: [1-1] 62dec416.2010 0     LOG:  database system was interrupted while in recovery at 2022-07-25 16:25:58 UTC
2022-07-25 16:25:58 UTC [8208]: [2-1] 62dec416.2010 0     HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-07-25 16:25:58 UTC [8208]: [3-1] 62dec416.2010 0     LOG:  database system was not properly shut down; automatic recovery in progress
2022-07-25 16:25:58 UTC [8208]: [4-1] 62dec416.2010 0     LOG:  redo starts at 0/C136018
2022-07-25 16:25:58 UTC [8208]: [5-1] 62dec416.2010 0     FATAL:  mismatching overwritten LSN 0/C135C38 -> 0/C135758
2022-07-25 16:25:58 UTC [8208]: [6-1] 62dec416.2010 0     CONTEXT:  WAL redo at 0/C136018 for XLOG/OVERWRITE_CONTRECORD: lsn 0/C135C38; time 2022-07-25 16:18:06.830395+00

2022-07-25 16:26:08,336 WARNING: Postgresql is not running.
2022-07-25 16:26:08,336 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:26:08,341 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7124342229348196421
  Database cluster state: in crash recovery
  pg_control last modified: Mon Jul 25 16:25:59 2022
  Latest checkpoint location: 0/C26AFB8
  Latest checkpoint's REDO location: 0/C135758
  Latest checkpoint's REDO WAL file: 0000002F000000000000000C
  Latest checkpoint's TimeLineID: 47
  Latest checkpoint's PrevTimeLineID: 47
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:3698
  Latest checkpoint's NextOID: 81920
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 3698
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Mon Jul 25 16:18:06 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 8d4b1e53f039b5d2f26f3912fd76ad1904431e4e7803d20e34f5e9039306a98a

2022-07-25 16:26:08,355 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-25 16:26:08,356 INFO: starting as a secondary
2022-07-25 16:26:08 UTC [8219]: [1-1] 62dec420.201b 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-25 16:26:08 UTC [8219]: [2-1] 62dec420.201b 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2022-07-25 16:26:08,797 INFO: postmaster pid=8219
/var/run/postgresql:5432 - no response
2022-07-25 16:26:08 UTC [8219]: [3-1] 62dec420.201b 0     LOG:  redirecting log output to logging collector process
2022-07-25 16:26:08 UTC [8219]: [4-1] 62dec420.201b 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-25 16:26:09,816 ERROR: postmaster is not running
postgres-operator-69ffbd9fd4-hdb5m   1/1     Running   0          28m   10.244.2.135   ip-172-31-6-72.us-east-2.compute.internal   <none>           <none>
smsc-cluster-0                       0/1     Running   0          25m   10.244.2.136   ip-172-31-6-72.us-east-2.compute.internal   <none>           <none>
smsc-cluster-1                       0/1     Running   0          24m   10.244.2.137   ip-172-31-6-72.us-east-2.compute.internal   <none>           <none>
smsc-cluster-2                       0/1     Running   0          24m   10.244.2.138   ip-172-31-6-72.us-east-2.compute.internal   <none>           <none>

Please assist

Deepanshudang commented 2 years ago

@FxKu

Can you please assit on this issue

FxKu commented 2 years ago

Could be various reasons. Are you using the latest version? Check Postgres logs inside the pod if you can find something.

Deepanshu6dec commented 2 years ago

@FxKu Yes using the latest release, registry.opensource.zalan.do/acid/spilo-14:2.1-p6

pod/smsc-cluster-0 1/1 Running
pod/smsc-cluster-1 1/1 Running

most of the logs are similar and sometime its throwing this error

pg_ctl: could not send promote signal (PID: 12719): No such process pg_ctl: could not send promote signal (PID: 12719): No such process

2022-07-27 14:41:53,820 INFO: promoted self to leader because I had the session lock pg_ctl: PID file "/home/postgres/pgdata/pgroot/data/postmaster.pid" does not exist Is server running? 2022-07-27 14:41:53,828 INFO: Lock owner: smsc-cluster-0; I am smsc-cluster-0 2022-07-27 14:41:53,875 INFO: promoted self to leader because I had the session lock pg_ctl: PID file "/home/postgres/pgdata/pgroot/data/postmaster.pid" does not exist Is server running?

2022-07-27 14:28:47,668 INFO: Lock owner: smsc-cluster-0; I am smsc-cluster-0 2022-07-27 14:28:47,720 INFO: starting as readonly because i had the session lock 2022-07-27 14:28:47,721 INFO: closed patroni connection to the postgresql cluster 2022-07-27 14:28:48 UTC [13302]: [1-1] 62e14ba0.33f6 0 LOG: Auto detecting pg_stat_kcache.linux_hz parameter... 2022-07-27 14:28:48 UTC [13302]: [2-1] 62e14ba0.33f6 0 LOG: pg_stat_kcache.linux_hz is set to 1000000 2022-07-27 14:28:48,139 INFO: postmaster pid=13302 2022-07-27 14:28:48 UTC [13302]: [3-1] 62e14ba0.33f6 0 LOG: redirecting log output to logging collector process 2022-07-27 14:28:48 UTC [13302]: [4-1] 62e14ba0.33f6 0 HINT: Future log output will appear in directory "../pg_log". /var/run/postgresql:5432 - no response /var/run/postgresql:5432 - accepting connections /var/run/postgresql:5432 - accepting connections 2022-07-27 14:28:49,256 INFO: Lock owner: smsc-cluster-0; I am smsc-cluster-0 2022-07-27 14:28:49,257 INFO: establishing a new patroni connection to the postgres cluster 2022-07-27 14:28:49,377 INFO: promoted self to leader because I had the session lock

This is the error i have seen apart from that info logs is available

After some time Pod status went to not ready state and redness probe error there 

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  5m44s                default-scheduler  0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
  Normal   Scheduled         5m42s                default-scheduler  Successfully assigned default/smsc-cluster-1 to ip-172-31-6-72.us-east-2.compute.internal
  Normal   Pulling           5m41s                kubelet            Pulling image "busybox"
  Normal   Pulled            5m41s                kubelet            Successfully pulled image "busybox" in 234.842724ms
  Normal   Created           5m41s                kubelet            Created container date
  Normal   Started           5m41s                kubelet            Started container date
  Normal   Pulled            5m40s                kubelet            Container image "registry.opensource.zalan.do/acid/spilo-14:2.1-p6" already present on machine
  Normal   Created           5m40s                kubelet            Created container postgres
  Normal   Started           5m40s                kubelet            Started container postgres
  Warning  Unhealthy         32s (x3 over 4m42s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503

pod/smsc-cluster-0 0/1 Running
pod/smsc-cluster-1 0/1 Running

Deepanshu6dec commented 2 years ago

@FxKu I'm keep watching the state of Postgresql cluster and not its went failed state. Please find the logs

kubectl logs smsc-cluster-0 

2022-07-27 16:22:16,749 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-27 16:22:16,749 INFO: starting as a secondary
2022-07-27 16:22:17 UTC [13059]: [1-1] 62e16639.3303 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-27 16:22:17 UTC [13059]: [2-1] 62e16639.3303 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2022-07-27 16:22:17,171 INFO: postmaster pid=13059
/var/run/postgresql:5432 - no response
2022-07-27 16:22:17 UTC [13059]: [3-1] 62e16639.3303 0     LOG:  redirecting log output to logging collector process
2022-07-27 16:22:17 UTC [13059]: [4-1] 62e16639.3303 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-27 16:22:18,235 ERROR: postmaster is not running
2022-07-27 16:22:23.635 35 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-27 16:22:26,732 WARNING: Postgresql is not running.
2022-07-27 16:22:26,732 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-27 16:22:26,736 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7125055893847375941
  Database cluster state: in production
  pg_control last modified: Wed Jul 27 15:58:55 2022
  Latest checkpoint location: 0/3D001198
  Latest checkpoint's REDO location: 0/3D001198
  Latest checkpoint's REDO WAL file: 000001A3000000000000003D
  Latest checkpoint's TimeLineID: 419
  Latest checkpoint's PrevTimeLineID: 419
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:8008
  Latest checkpoint's NextOID: 532480
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 0
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Wed Jul 27 15:58:55 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 65b5cfd5369725735558015d63fd41ae49599572e79cef8dc1fd918076fb95a4

2022-07-27 16:22:26,736 INFO: doing crash recovery in a single user mode
2022-07-27 16:22:26,823 ERROR: Crash recovery finished with code=-6
2022-07-27 16:22:26,823 INFO:  stdout=
2022-07-27 16:22:26,823 INFO:  stderr=2022-07-27 16:22:26 UTC [13069]: [1-1] 62e16642.330d 0     LOG:  database system was interrupted; last known up at 2022-07-27 15:58:55 UTC
2022-07-27 16:22:26 UTC [13069]: [2-1] 62e16642.330d 0     LOG:  invalid xl_info in primary checkpoint record
2022-07-27 16:22:26 UTC [13069]: [3-1] 62e16642.330d 0     PANIC:  could not locate a valid checkpoint record

2022-07-27 16:22:36,733 WARNING: Postgresql is not running.
2022-07-27 16:22:36,733 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-27 16:22:36,736 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7125055893847375941
  Database cluster state: in production
  pg_control last modified: Wed Jul 27 15:58:55 2022
  Latest checkpoint location: 0/3D001198
  Latest checkpoint's REDO location: 0/3D001198
  Latest checkpoint's REDO WAL file: 000001A3000000000000003D
  Latest checkpoint's TimeLineID: 419
  Latest checkpoint's PrevTimeLineID: 419
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:8008
  Latest checkpoint's NextOID: 532480
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 0
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Wed Jul 27 15:58:55 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 65b5cfd5369725735558015d63fd41ae49599572e79cef8dc1fd918076fb95a4

2022-07-27 16:22:36,749 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-27 16:22:36,750 INFO: starting as a secondary
2022-07-27 16:22:37,166 INFO: postmaster pid=13080
2022-07-27 16:22:37 UTC [13080]: [1-1] 62e1664d.3318 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-27 16:22:37 UTC [13080]: [2-1] 62e1664d.3318 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
/var/run/postgresql:5432 - no response
2022-07-27 16:22:37 UTC [13080]: [3-1] 62e1664d.3318 0     LOG:  redirecting log output to logging collector process
2022-07-27 16:22:37 UTC [13080]: [4-1] 62e1664d.3318 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-27 16:22:38,234 ERROR: postmaster is not running
2022-07-27 16:22:46,732 WARNING: Postgresql is not running.
2022-07-27 16:22:46,732 INFO: Lock owner: None; I am smsc-cluster-0
2022-07-27 16:22:46,735 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7125055893847375941
  Database cluster state: in production
  pg_control last modified: Wed Jul 27 15:58:55 2022
  Latest checkpoint location: 0/3D001198
  Latest checkpoint's REDO location: 0/3D001198
  Latest checkpoint's REDO WAL file: 000001A3000000000000003D
  Latest checkpoint's TimeLineID: 419
  Latest checkpoint's PrevTimeLineID: 419
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:8008
  Latest checkpoint's NextOID: 532480
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 0
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Wed Jul 27 15:58:55 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 65b5cfd5369725735558015d63fd41ae49599572e79cef8dc1fd918076fb95a4

2022-07-27 16:22:46,736 INFO: doing crash recovery in a single user mode
2022-07-27 16:22:46,826 ERROR: Crash recovery finished with code=-6
2022-07-27 16:22:46,826 INFO:  stdout=
2022-07-27 16:22:46,826 INFO:  stderr=2022-07-27 16:22:46 UTC [13090]: [1-1] 62e16656.3322 0     LOG:  database system was interrupted; last known up at 2022-07-27 15:58:55 UTC
2022-07-27 16:22:46 UTC [13090]: [2-1] 62e16656.3322 0     LOG:  invalid xl_info in primary checkpoint record
2022-07-27 16:22:46 UTC [13090]: [3-1] 62e16656.3322 0     PANIC:  could not locate a valid checkpoint record

2022-07-27 16:22:53.635 35 ERROR connection error: PQconnectStart
2022-07-27 16:22:53.635 35 ERROR 
2022-07-27 16:22:53.635 35 WARNING csmsc: default timeout
2022-07-27 16:22:53.635 35 ERROR connection error: PQconnectStart
2022-07-27 16:22:53.635 35 ERROR 
2022-07-27 16:22:53.635 35 WARNING postgres: default timeout
2022-07-27 16:22:53.636 35 ERROR connection error: PQconnectStart
2022-07-27 16:22:53.636 35 ERROR 
2022-07-27 16:22:53.636 35 LOG {ticks: 0, maint: 0, retry: 0}

kubectl logs smsc-cluster-1

2022-07-27 16:23:56,786 INFO: Lock owner: None; I am smsc-cluster-1
2022-07-27 16:23:56,787 INFO: starting as a secondary
2022-07-27 16:23:57,172 INFO: postmaster pid=21799
2022-07-27 16:23:57 UTC [21799]: [1-1] 62e1669d.5527 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2022-07-27 16:23:57 UTC [21799]: [2-1] 62e1669d.5527 0     LOG:  pg_stat_kcache.linux_hz is set to 333333
/var/run/postgresql:5432 - no response
2022-07-27 16:23:57 UTC [21799]: [3-1] 62e1669d.5527 0     LOG:  redirecting log output to logging collector process
2022-07-27 16:23:57 UTC [21799]: [4-1] 62e1669d.5527 0     HINT:  Future log output will appear in directory "../pg_log".
2022-07-27 16:23:58,235 ERROR: postmaster is not running
2022-07-27 16:24:03.724 27 ERROR connection error: PQconnectStart
2022-07-27 16:24:03.725 27 ERROR 
2022-07-27 16:24:03.725 27 WARNING csmsc: default timeout
2022-07-27 16:24:03.725 27 ERROR connection error: PQconnectStart
2022-07-27 16:24:03.725 27 ERROR 
2022-07-27 16:24:03.725 27 ERROR connection error: PQconnectStart
2022-07-27 16:24:03.725 27 ERROR 
2022-07-27 16:24:03.725 27 WARNING postgres: default timeout
2022-07-27 16:24:03.756 27 LOG {ticks: 0, maint: 0, retry: 0}
2022-07-27 16:24:06,770 WARNING: Postgresql is not running.
2022-07-27 16:24:06,770 INFO: Lock owner: None; I am smsc-cluster-1
2022-07-27 16:24:06,776 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202107181
  Database system identifier: 7125055893847375941
  Database cluster state: in production
  pg_control last modified: Wed Jul 27 15:58:55 2022
  Latest checkpoint location: 0/3D001198
  Latest checkpoint's REDO location: 0/3D001198
  Latest checkpoint's REDO WAL file: 000001A3000000000000003D
  Latest checkpoint's TimeLineID: 419
  Latest checkpoint's PrevTimeLineID: 419
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:8008
  Latest checkpoint's NextOID: 532480
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 727
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 0
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Wed Jul 27 15:58:55 2022
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 65b5cfd5369725735558015d63fd41ae49599572e79cef8dc1fd918076fb95a4

2022-07-27 16:24:06,776 INFO: doing crash recovery in a single user mode
2022-07-27 16:24:06,847 ERROR: Crash recovery finished with code=-6
2022-07-27 16:24:06,847 INFO:  stdout=
2022-07-27 16:24:06,847 INFO:  stderr=2022-07-27 16:24:06 UTC [21809]: [1-1] 62e166a6.5531 0     LOG:  database system was interrupted; last known up at 2022-07-27 15:58:55 UTC
2022-07-27 16:24:06 UTC [21809]: [2-1] 62e166a6.5531 0     LOG:  invalid xl_info in primary checkpoint record
2022-07-27 16:24:06 UTC [21809]: [3-1] 62e166a6.5531 0     PANIC:  could not locate a valid checkpoint record

k describe postgresql.acid.zalan.do/smsc-cluster

Events:
  Type     Reason  Age   From               Message
  ----     ------  ----  ----               -------
  Warning  Sync    10m   postgres-operator  could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries
mpdevul commented 1 year ago

was this ever fixed, in my case the readiness probes work when updating the check with /patroni instead of /readiness.

although if I do curl of localhost:8008/readiness inside the pod it gives the same output as localhost:8008/patroni