Closed PGPAWAN closed 1 year ago
I have followed all the steps manually to test the upgrade and current running postgres service failed to stop.
Checked cluster compatible
postgres@pg-testdb-0:/tmp$ time /usr/lib/postgresql/15/bin/pg_upgrade --old-bindir /usr/lib/postgresql/14/bin --new-bindir /usr/lib/postgresql/15/bin -p 5432 -P 5433 --old-datadir /home/postgres/pgdata/pgroot/data --new-datadir /home/postgres/pgdata/pgroot/data1 --link --check
Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for system-defined composite types in user tables ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for presence of required libraries ok
Checking database user is the install user ok
Checking for prepared transactions ok
Checking for new cluster tablespace directories ok
*Clusters are compatible*
real 0m0.609s
user 0m0.060s
sys 0m0.067s
postgres@pg-testdb-0:/tmp$
Failed to stop postgresql service.
postgres@pg-testdb-0:/tmp$ patronictl list
+ Cluster: pg-testdb ------+--------+---------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-----------------+------------+--------+---------+----+-----------+
| pg-testdb-0 | 172.17.0.5 | Leader | running | 7 | |
+-----------------+------------+--------+---------+----+-----------+
Maintenance mode: on
postgres@pg-testdb-0:/tmp$ pg_ctl -D /home/postgres/pgdata/pgroot/data stop
waiting for server to shut down............................................................... failed
pg_ctl: server does not shut down
postgres log
2023-05-18 07:04:53.427 UTC,"postgres","postgres",415,"[local]",6465ce15.19f,4,"idle",2023-05-18 07:04:53 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.002 user=postgres database=postgres host=[local]",,,,,,,,,"pgq ticker","client backend",,0
2023-05-18 07:04:53.429 UTC,"postgres","testdb",416,"[local]",6465ce15.1a0,4,"idle",2023-05-18 07:04:53 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.003 user=postgres database=testdb host=[local]",,,,,,,,,"pgq ticker","client backend",,0
2023-05-18 07:04:57.418 UTC,,,63,,6465ccea.3f,10,,2023-05-18 06:59:54 UTC,,0,LOG,00000,"received SIGHUP, reloading configuration files",,,,,,,,,"","postmaster",,0
2023-05-18 07:04:58.423 UTC,,,119,,6465cced.77,3,,2023-05-18 06:59:57 UTC,,0,LOG,00000,"checkpoint starting: time",,,,,,,,,"","checkpointer",,0
2023-05-18 07:05:01.163 UTC,,,63,,6465ccea.3f,11,,2023-05-18 06:59:54 UTC,,0,LOG,00000,"received fast shutdown request",,,,,,,,,"","postmaster",,0
2023-05-18 07:05:01.166 UTC,,,63,,6465ccea.3f,12,,2023-05-18 06:59:54 UTC,,0,LOG,00000,"aborting any active transactions",,,,,,,,,"","postmaster",,0
2023-05-18 07:05:01.166 UTC,,,210,,6465ccee.d2,2,,2023-05-18 06:59:58 UTC,4/0,0,LOG,00000,"pg_cron scheduler shutting down",,,,,,,,,"","pg_cron launcher",,-5280970146734079043
2023-05-18 07:05:01.169 UTC,"postgres","postgres",138,"[local]",6465cced.8a,4,"idle",2023-05-18 06:59:57 UTC,2/0,0,FATAL,57P01,"terminating connection due to administrator command",,,,,,,,,"Patroni","client backend",,-1474624185022020383
2023-05-18 07:05:01.169 UTC,"postgres","postgres",138,"[local]",6465cced.8a,5,"idle",2023-05-18 06:59:57 UTC,,0,LOG,00000,"disconnection: session time: 0:05:04.006 user=postgres database=postgres host=[local]",,,,,,,,,"Patroni","client backend",,0
2023-05-18 07:05:01.173 UTC,,,63,,6465ccea.3f,13,,2023-05-18 06:59:54 UTC,,0,LOG,00000,"background worker ""logical replication launcher"" (PID 212) exited with exit code 1",,,,,,,,,"","postmaster",,0
2023-05-18 07:05:01.180 UTC,,,63,,6465ccea.3f,14,,2023-05-18 06:59:54 UTC,,0,LOG,00000,"background worker ""bg_mon"" (PID 67) exited with exit code 1",,,,,,,,,"","postmaster",,0
2023-05-18 07:05:01.238 UTC,,,119,,6465cced.77,4,,2023-05-18 06:59:57 UTC,,0,LOG,00000,"checkpoint complete: wrote 491 buffers (1.5%); 0 WAL file(s) added, 0 removed, 0 recycled; write=2.774 s, sync=0.037 s, total=2.816 s; sync files=165, longest=0.014 s, average=0.001 s; distance=1879 kB, estimate=1879 kB",,,,,,,,,"","checkpointer",,0
2023-05-18 07:05:01.238 UTC,,,119,,6465cced.77,5,,2023-05-18 06:59:57 UTC,,0,LOG,00000,"shutting down",,,,,,,,,"","checkpointer",,0
2023-05-18 07:05:01.255 UTC,,,119,,6465cced.77,6,,2023-05-18 06:59:57 UTC,,0,LOG,00000,"checkpoint starting: shutdown immediate",,,,,,,,,"","checkpointer",,0
2023-05-18 07:05:01.285 UTC,,,119,,6465cced.77,7,,2023-05-18 06:59:57 UTC,,0,LOG,00000,"checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0
.001 s, total=0.031 s; sync files=0, longest=0.000 s, average=0.000 s; distance=14504 kB, estimate=14504 kB",,,,,,,,,"","checkpointer",,0
2023-05-18 07:05:08.547 UTC,,,426,"[local]",6465ce24.1aa,1,"",2023-05-18 07:05:08 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2023-05-18 07:05:08.547 UTC,"postgres","postgres",426,"[local]",6465ce24.1aa,2,"",2023-05-18 07:05:08 UTC,,0,FATAL,57P03,"the database system is shutting down",,,,,,,,,"","client backend",,0
2023-05-18 07:05:08.557 UTC,,,428,"[local]",6465ce24.1ac,1,"",2023-05-18 07:05:08 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2023-05-18 07:05:08.557 UTC,"postgres","postgres",428,"[local]",6465ce24.1ac,2,"",2023-05-18 07:05:08 UTC,,0,FATAL,57P03,"the database system is shutting down",,,,,,,,,"","client backend",,0
2023-05-18 07:05:17.857 UTC,,,438,"[local]",6465ce2d.1b6,1,"",2023-05-18 07:05:17 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2023-05-18 07:05:17.857 UTC,"postgres","postgres",438,"[local]",6465ce2d.1b6,2,"",2023-05-18 07:05:17 UTC,,0,FATAL,57P03,"the database system is shutting down",,,,,,,,,"","client backend",,0
2023-05-18 07:05:17.865 UTC,,,439,"[local]",6465ce2d.1b7,1,"",2023-05-18 07:05:17 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
2023-05-18 07:05:17.866 UTC,"postgres","postgres",439,"[local]",6465ce2d.1b7,2,"",2023-05-18 07:05:17 UTC,,0,FATAL,57P03,"the database system is shutting down",,,,,,,,,"","client backend",,0
2023-05-18 07:05:18.630 UTC,,,440,"[local]",6465ce2e.1b8,1,"",2023-05-18 07:05:18 UTC,,0,LOG,00000,"connection received: host=[local]",,,,,,,,,"","not initialized",,0
There are two things that possibly prevents the instance from a clear shut down
Thanks its working. it was because of the acrhive_command. Since we have disable the archiving in lower environment.
We have identified below issue while doing Spilo upgrade to upgrade PostgreSQL 14 to PostgreSQL 15.
During the upgrade patroni cluster failed to shutdown with below error. upgrade command on pod it is getting stuck on doing clean shutdown phase.
Request you please check and help me fix it.
Note: we are running only 1 Pod as master in DEV env..
Upgrade log
Patroni log
Operator log
Images Used
registry.opensource.zalan.do/acid/postgres-operator:v1.9.0 registry.opensource.zalan.do/acid/spilo-14:2.1-p6 ghcr.io/zalando/spilo-15:2.1-p9
Where do you run it - cloud or metal? Kubernetes or OpenShift?
VMware Virtual Machine
Kubernetes version
v.21.