CrunchyData / postgres-operator

Production PostgreSQL for Kubernetes, from high availability Postgres clusters to full-scale database-as-a-service.
https://access.crunchydata.com/documentation/postgres-operator/v5/
Apache License 2.0
3.91k stars 587 forks source link

remote-0 process on 'rmgl-backrest-shared-repo' terminated unexpectedly [255]: ssh: connect to host rmgl-backrest-shared-repo port 2022: Connection timed out #2096

Closed Bhagi33 closed 3 years ago

Bhagi33 commented 3 years ago

I have upgraded postgresql crunchy cluster from 12.4 to 12.5 version. After upgrade only primay has created and running in read-only transactions mode, observed there is an issue found in log. log info has been listed down.

2020-12-03 10:46:40.326 UTC [12117] LOG: could not receive data from client: Connection reset by peer 2020-12-03 10:47:43.987 UTC [12504] LOG: could not receive data from client: Connection reset by peer 2020-12-03 10:49:56.991 UTC [13290] LOG: could not receive data from client: Connection reset by peer 2020-12-03 10:51:57.278 UTC [259] LOG: received fast shutdown request 2020-12-03 10:51:57.282 UTC [259] LOG: aborting any active transactions 2020-12-03 10:51:57.282 UTC [273] FATAL: terminating connection due to administrator command 2020-12-03 10:51:57.283 UTC [259] LOG: background worker "logical replication launcher" (PID 269) exited with exit code 1 2020-12-03 10:51:57.287 UTC [263] LOG: shutting down 2020-12-03 10:51:57.390 UTC [259] LOG: database system is shut down 2020-12-03 10:53:16.517 UTC [174] LOG: database system was shut down at 2020-12-03 10:51:57 UTC 2020-12-03 10:53:16.850 UTC [181] FATAL: the database system is starting up 2020-12-03 10:53:16.990 UTC [174] LOG: entering standby mode 2020-12-03 10:53:17.308 UTC [174] LOG: consistent recovery state reached at 0/110000A0 2020-12-03 10:53:17.308 UTC [174] LOG: invalid record length at 0/110000A0: wanted 24, got 0 2020-12-03 10:53:17.309 UTC [171] LOG: database system is ready to accept read only connections 2020-12-03 10:55:04.054 UTC [630] ERROR: cannot execute CREATE DATABASE in a read-only transaction 2020-12-03 10:55:04.054 UTC [630] STATEMENT: create database bhagi; ERROR: [125]: remote-0 process on 'rmgl-backrest-shared-repo' terminated unexpectedly [255]: ssh: connect to host rmgl-backrest-shared-repo port 2022: Connection timed out ######################## bash-4.2$ psql psql (12.5) Type "help" for help.

postgres=# create database bb; ERROR: cannot execute CREATE DATABASE in a read-only transaction postgres=#

################ kubectl get po -n pg-cron-12-ver NAME READY STATUS RESTARTS AGE rmgl-766759b8d7-9d9dd 1/1 Running 0 13m rmgl-backrest-shared-repo-64685df94d-9ht4s 1/1 Running 0 13m

Note: there is no replicas added after upgrade.

Additional context Add any other context about the problem here.

jkatz commented 3 years ago

What do the logs on the rmgl-backrest-shared-repo Pod show?