Closed ray73864 closed 8 months ago
Application Name:tt-rss Pod Name:tt-rss-cnpg-main-rw-cdb5b888f-qmjf2 Container Name:pgbouncer
2023-05-01 00:17:06.761348+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"Starting CloudNativePG PgBouncer Instance Manager","version":"1.19.0","build":{"Version":"1.19.0","Commit":"d9bf88dd","Date":"2023-02-14"}}
2023-05-01 00:17:06.819445+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/pgbouncer.ini"}
2023-05-01 00:17:06.822372+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/pg_hba.conf"}
2023-05-01 00:17:06.825059+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/server-tls/ca.crt"}
2023-05-01 00:17:06.827623+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/client-ca/ca.crt"}
2023-05-01 00:17:06.830298+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/server-tls/tls.crt"}
2023-05-01 00:17:06.832973+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/server-tls/tls.key"}
2023-05-01 00:17:06.835518+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/authUser/tls.crt"}
2023-05-01 00:17:06.838181+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/authUser/tls.key"}
2023-05-01 00:17:06.846080+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"record","pipe":"stderr","record":{"timestamp":"2023-05-01 00:17:06.845 UTC","pid":"20","level":"LOG","msg":"kernel file descriptor limit: 1048576 (hard: 1048576); max_client_conn: 1000, max expected fd use: 1012"}}
2023-05-01 00:17:06.846276+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"record","pipe":"stderr","record":{"timestamp":"2023-05-01 00:17:06.846 UTC","pid":"20","level":"LOG","msg":"listening on 0.0.0.0:5432"}}
2023-05-01 00:17:06.846290+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"record","pipe":"stderr","record":{"timestamp":"2023-05-01 00:17:06.846 UTC","pid":"20","level":"LOG","msg":"listening on [::]:5432"}}
2023-05-01 00:17:06.846341+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"record","pipe":"stderr","record":{"timestamp":"2023-05-01 00:17:06.846 UTC","pid":"20","level":"LOG","msg":"listening on unix:/controller/run/.s.PGSQL.5432"}}
2023-05-01 00:17:06.846357+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"record","pipe":"stderr","record":{"timestamp":"2023-05-01 00:17:06.846 UTC","pid":"20","level":"LOG","msg":"process up: PgBouncer 1.18.0, libevent 2.1.8-stable (epoll), adns: udns 0.4, tls: OpenSSL 1.1.1n 15 Mar 2022"}}
Application Name:tt-rss Pod Name:tt-rss-cnpg-main-rw-cdb5b888f-qmjf2 Container Name:bootstrap-controller
2023-05-01 00:16:15.101275+00:00{"level":"info","ts":"2023-05-01T00:16:15Z","msg":"Installing the manager executable","destination":"/controller/manager","version":"1.19.0","build":{"Version":"1.19.0","Commit":"d9bf88dd","Date":"2023-02-14"}}
2023-05-01 00:16:17.135261+00:00{"level":"info","ts":"2023-05-01T00:16:17Z","msg":"Setting 0750 permissions"}
2023-05-01 00:16:17.135283+00:00{"level":"info","ts":"2023-05-01T00:16:17Z","msg":"Bootstrap completed"}
Application Name:tt-rss Pod Name:tt-rss-cnpg-main-rw-cdb5b888f-z5mhz Container Name:pgbouncer
2023-05-01 00:17:06.761523+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"Starting CloudNativePG PgBouncer Instance Manager","version":"1.19.0","build":{"Version":"1.19.0","Commit":"d9bf88dd","Date":"2023-02-14"}}
2023-05-01 00:17:06.819432+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/server-tls/tls.crt"}
2023-05-01 00:17:06.822363+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/server-tls/tls.key"}
2023-05-01 00:17:06.825052+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/authUser/tls.crt"}
2023-05-01 00:17:06.827616+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/authUser/tls.key"}
2023-05-01 00:17:06.830291+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/pgbouncer.ini"}
2023-05-01 00:17:06.832967+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/pg_hba.conf"}
2023-05-01 00:17:06.835517+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/server-tls/ca.crt"}
2023-05-01 00:17:06.838170+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"updated configuration file","name":"/controller/configs/client-ca/ca.crt"}
2023-05-01 00:17:06.846158+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"record","pipe":"stderr","record":{"timestamp":"2023-05-01 00:17:06.845 UTC","pid":"19","level":"LOG","msg":"kernel file descriptor limit: 1048576 (hard: 1048576); max_client_conn: 1000, max expected fd use: 1012"}}
2023-05-01 00:17:06.846282+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"record","pipe":"stderr","record":{"timestamp":"2023-05-01 00:17:06.846 UTC","pid":"19","level":"LOG","msg":"listening on 0.0.0.0:5432"}}
2023-05-01 00:17:06.846294+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"record","pipe":"stderr","record":{"timestamp":"2023-05-01 00:17:06.846 UTC","pid":"19","level":"LOG","msg":"listening on [::]:5432"}}
2023-05-01 00:17:06.846342+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"record","pipe":"stderr","record":{"timestamp":"2023-05-01 00:17:06.846 UTC","pid":"19","level":"LOG","msg":"listening on unix:/controller/run/.s.PGSQL.5432"}}
2023-05-01 00:17:06.846361+00:00{"level":"info","ts":"2023-05-01T00:17:06Z","msg":"record","pipe":"stderr","record":{"timestamp":"2023-05-01 00:17:06.846 UTC","pid":"19","level":"LOG","msg":"process up: PgBouncer 1.18.0, libevent 2.1.8-stable (epoll), adns: udns 0.4, tls: OpenSSL 1.1.1n 15 Mar 2022"}}
Application Name:tt-rss Pod Name:tt-rss-cnpg-main-rw-cdb5b888f-z5mhz Container Name:bootstrap-controller
2023-05-01 00:16:15.095543+00:00{"level":"info","ts":"2023-05-01T00:16:15Z","msg":"Installing the manager executable","destination":"/controller/manager","version":"1.19.0","build":{"Version":"1.19.0","Commit":"d9bf88dd","Date":"2023-02-14"}}
2023-05-01 00:16:17.133944+00:00{"level":"info","ts":"2023-05-01T00:16:17Z","msg":"Setting 0750 permissions"}
2023-05-01 00:16:17.133981+00:00{"level":"info","ts":"2023-05-01T00:16:17Z","msg":"Bootstrap completed"}
Application Name:tt-rss Pod Name:tt-rss-cnpg-main-1 Container Name:postgres
2023-05-01 00:17:50.228297+00:00{"level":"info","ts":"2023-05-01T00:17:50Z","logger":"setup","msg":"Starting CloudNativePG Instance Manager","logging_pod":"tt-rss-cnpg-main-1","version":"1.19.0","build":{"Version":"1.19.0","Commit":"d9bf88dd","Date":"2023-02-14"}}
2023-05-01 00:17:51.537379+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"setup","msg":"starting controller-runtime manager","logging_pod":"tt-rss-cnpg-main-1"}
2023-05-01 00:17:51.537688+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Starting EventSource","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","source":"kind source: *v1.Cluster"}
2023-05-01 00:17:51.537701+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Starting Controller","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster"}
2023-05-01 00:17:51.537962+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Starting webserver","logging_pod":"tt-rss-cnpg-main-1","address":":9187"}
2023-05-01 00:17:51.537976+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Starting webserver","logging_pod":"tt-rss-cnpg-main-1","address":"localhost:8010"}
2023-05-01 00:17:51.537985+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Starting webserver","logging_pod":"tt-rss-cnpg-main-1","address":":8000"}
2023-05-01 00:17:51.604693+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Instance status probe failing","logging_pod":"tt-rss-cnpg-main-1","err":"failed to connect to `host=/controller/run user=postgres database=postgres`: dial error (dial unix /controller/run/.s.PGSQL.5432: connect: no such file or directory)"}
2023-05-01 00:17:51.638682+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Starting workers","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","worker count":1}
2023-05-01 00:17:51.639056+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Ignore minSyncReplicas to enforce self-healing","logging_pod":"tt-rss-cnpg-main-1","syncReplicas":-1,"minSyncReplicas":0,"maxSyncReplicas":0}
2023-05-01 00:17:51.693748+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Refreshed configuration file","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"89aa9c6e-5a42-4a71-9d2d-c4c31da0bacb","uuid":"9c56d8bb-e7b5-11ed-ad8b-b6ea5f78a982","logging_pod":"tt-rss-cnpg-main-1","filename":"/controller/certificates/server.crt","secret":"tt-rss-cnpg-main-server"}
2023-05-01 00:17:51.696512+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Refreshed configuration file","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"89aa9c6e-5a42-4a71-9d2d-c4c31da0bacb","uuid":"9c56d8bb-e7b5-11ed-ad8b-b6ea5f78a982","logging_pod":"tt-rss-cnpg-main-1","filename":"/controller/certificates/server.key","secret":"tt-rss-cnpg-main-server"}
2023-05-01 00:17:51.700530+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Refreshed configuration file","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"89aa9c6e-5a42-4a71-9d2d-c4c31da0bacb","uuid":"9c56d8bb-e7b5-11ed-ad8b-b6ea5f78a982","logging_pod":"tt-rss-cnpg-main-1","filename":"/controller/certificates/streaming_replica.crt","secret":"tt-rss-cnpg-main-replication"}
2023-05-01 00:17:51.703328+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Refreshed configuration file","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"89aa9c6e-5a42-4a71-9d2d-c4c31da0bacb","uuid":"9c56d8bb-e7b5-11ed-ad8b-b6ea5f78a982","logging_pod":"tt-rss-cnpg-main-1","filename":"/controller/certificates/streaming_replica.key","secret":"tt-rss-cnpg-main-replication"}
2023-05-01 00:17:51.707113+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Refreshed configuration file","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"89aa9c6e-5a42-4a71-9d2d-c4c31da0bacb","uuid":"9c56d8bb-e7b5-11ed-ad8b-b6ea5f78a982","logging_pod":"tt-rss-cnpg-main-1","filename":"/controller/certificates/client-ca.crt","secret":"tt-rss-cnpg-main-ca"}
2023-05-01 00:17:51.710948+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Refreshed configuration file","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"89aa9c6e-5a42-4a71-9d2d-c4c31da0bacb","uuid":"9c56d8bb-e7b5-11ed-ad8b-b6ea5f78a982","logging_pod":"tt-rss-cnpg-main-1","filename":"/controller/certificates/server-ca.crt","secret":"tt-rss-cnpg-main-ca"}
2023-05-01 00:17:51.714091+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Installed configuration file","logging_pod":"tt-rss-cnpg-main-1","pgdata":"/var/lib/postgresql/data/pgdata","filename":"pg_hba.conf"}
2023-05-01 00:17:51.714119+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Ignore minSyncReplicas to enforce self-healing","logging_pod":"tt-rss-cnpg-main-1","syncReplicas":-1,"minSyncReplicas":0,"maxSyncReplicas":0}
2023-05-01 00:17:51.716999+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Installed configuration file","logging_pod":"tt-rss-cnpg-main-1","pgdata":"/var/lib/postgresql/data/pgdata","filename":"custom.conf"}
2023-05-01 00:17:51.717086+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Cluster status","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"89aa9c6e-5a42-4a71-9d2d-c4c31da0bacb","uuid":"9c56d8bb-e7b5-11ed-ad8b-b6ea5f78a982","logging_pod":"tt-rss-cnpg-main-1","currentPrimary":"","targetPrimary":"tt-rss-cnpg-main-1"}
2023-05-01 00:17:51.717101+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"First primary instance bootstrap, marking myself as primary","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"89aa9c6e-5a42-4a71-9d2d-c4c31da0bacb","uuid":"9c56d8bb-e7b5-11ed-ad8b-b6ea5f78a982","logging_pod":"tt-rss-cnpg-main-1","currentPrimary":"","targetPrimary":"tt-rss-cnpg-main-1"}
2023-05-01 00:17:51.723519+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Extracting pg_controldata information","logging_pod":"tt-rss-cnpg-main-1","reason":"postmaster start up"}
2023-05-01 00:17:51.724662+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"pg_controldata","msg":"pg_control version number: 1300\nCatalog version number: 202209061\nDatabase system identifier: 7228001436425232409\nDatabase cluster state: shut down\npg_control last modified: Mon 01 May 2023 12:17:09 AM UTC\nLatest checkpoint location: 0/191C8C8\nLatest checkpoint's REDO location: 0/191C8C8\nLatest checkpoint's REDO WAL file: 000000010000000000000001\nLatest checkpoint's TimeLineID: 1\nLatest checkpoint's PrevTimeLineID: 1\nLatest checkpoint's full_page_writes: on\nLatest checkpoint's NextXID: 0:726\nLatest checkpoint's NextOID: 16386\nLatest checkpoint's NextMultiXactId: 1\nLatest checkpoint's NextMultiOffset: 0\nLatest checkpoint's oldestXID: 716\nLatest checkpoint's oldestXID's DB: 1\nLatest checkpoint's oldestActiveXID: 0\nLatest checkpoint's oldestMultiXid: 1\nLatest checkpoint's oldestMulti's DB: 1\nLatest checkpoint's oldestCommitTsXid:0\nLatest checkpoint's newestCommitTsXid:0\nTime of latest checkpoint: Mon 01 May 2023 12:17:08 AM UTC\nFake LSN counter for unlogged rels: 0/3E8\nMinimum recovery ending location: 0/0\nMin recovery ending loc's timeline: 0\nBackup start location: 0/0\nBackup end location: 0/0\nEnd-of-backup record required: no\nwal_level setting: replica\nwal_log_hints setting: off\nmax_connections setting: 100\nmax_worker_processes setting: 8\nmax_wal_senders setting: 10\nmax_prepared_xacts setting: 0\nmax_locks_per_xact setting: 64\ntrack_commit_timestamp setting: off\nMaximum data alignment: 8\nDatabase block size: 8192\nBlocks per segment of large relation: 131072\nWAL block size: 8192\nBytes per WAL segment: 16777216\nMaximum length of identifiers: 64\nMaximum columns in an index: 32\nMaximum size of a TOAST chunk: 1996\nSize of a large-object chunk: 2048\nDate/time type storage: 64-bit integers\nFloat8 argument passing: by value\nData page checksum version: 0\nMock authentication nonce: a2242f73221a20076d18df5af9595eb833ace467bd909e72b8c21dd4db20060e\n","pipe":"stdout","logging_pod":"tt-rss-cnpg-main-1"}
2023-05-01 00:17:51.724689+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"The PID file content is wrong, deleting it and assuming it's stale","file":"/var/lib/postgresql/data/pgdata/postmaster.pid","logging_pod":"tt-rss-cnpg-main-1","err":"file does not exist","pidFileContents":""}
2023-05-01 00:17:51.725189+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"DB not available, will retry","logging_pod":"tt-rss-cnpg-main-1","err":"failed to connect to `host=/controller/run user=postgres database=postgres`: dial error (dial unix /controller/run/.s.PGSQL.5432: connect: no such file or directory)"}
2023-05-01 00:17:51.744034+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Instance is still down, will retry in 1 second","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"89aa9c6e-5a42-4a71-9d2d-c4c31da0bacb","uuid":"9c56d8bb-e7b5-11ed-ad8b-b6ea5f78a982","logging_pod":"tt-rss-cnpg-main-1"}
2023-05-01 00:17:51.744114+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Ignore minSyncReplicas to enforce self-healing","logging_pod":"tt-rss-cnpg-main-1","syncReplicas":-1,"minSyncReplicas":0,"maxSyncReplicas":0}
2023-05-01 00:17:51.744858+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"postgres","msg":"2023-05-01 00:17:51.744 UTC [28] LOG: redirecting log output to logging collector process","pipe":"stderr","logging_pod":"tt-rss-cnpg-main-1"}
2023-05-01 00:17:51.744881+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"postgres","msg":"2023-05-01 00:17:51.744 UTC [28] HINT: Future log output will appear in directory \"/controller/log\".","pipe":"stderr","logging_pod":"tt-rss-cnpg-main-1"}
2023-05-01 00:17:51.745109+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:17:51.744 UTC","process_id":"28","session_id":"644f052f.1c","session_line_num":"1","session_start_time":"2023-05-01 00:17:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"ending log output to stderr","hint":"Future log output will go to log destination \"csvlog\".","backend_type":"postmaster","query_id":"0"}}
2023-05-01 00:17:51.745123+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:17:51.744 UTC","process_id":"28","session_id":"644f052f.1c","session_line_num":"2","session_start_time":"2023-05-01 00:17:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"starting PostgreSQL 15.2 (Debian 15.2-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit","backend_type":"postmaster","query_id":"0"}}
2023-05-01 00:17:51.745133+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:17:51.744 UTC","process_id":"28","session_id":"644f052f.1c","session_line_num":"3","session_start_time":"2023-05-01 00:17:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"listening on IPv4 address \"0.0.0.0\", port 5432","backend_type":"postmaster","query_id":"0"}}
2023-05-01 00:17:51.745140+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:17:51.744 UTC","process_id":"28","session_id":"644f052f.1c","session_line_num":"4","session_start_time":"2023-05-01 00:17:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"listening on IPv6 address \"::\", port 5432","backend_type":"postmaster","query_id":"0"}}
2023-05-01 00:17:51.745146+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"postgres","msg":"2023-05-01 00:17:51.744 UTC [28] LOG: ending log output to stderr","source":"/controller/log/postgres","logging_pod":"tt-rss-cnpg-main-1"}
2023-05-01 00:17:51.745152+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"postgres","msg":"2023-05-01 00:17:51.744 UTC [28] HINT: Future log output will go to log destination \"csvlog\".","source":"/controller/log/postgres","logging_pod":"tt-rss-cnpg-main-1"}
2023-05-01 00:17:51.748917+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"Ignore minSyncReplicas to enforce self-healing","logging_pod":"tt-rss-cnpg-main-1","syncReplicas":-1,"minSyncReplicas":0,"maxSyncReplicas":0}
2023-05-01 00:17:51.750799+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:17:51.750 UTC","process_id":"28","session_id":"644f052f.1c","session_line_num":"5","session_start_time":"2023-05-01 00:17:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"listening on Unix socket \"/controller/run/.s.PGSQL.5432\"","backend_type":"postmaster","query_id":"0"}}
2023-05-01 00:17:51.757610+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:17:51.757 UTC","process_id":"33","session_id":"644f052f.21","session_line_num":"1","session_start_time":"2023-05-01 00:17:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"database system was shut down at 2023-05-01 00:17:09 UTC","backend_type":"startup","query_id":"0"}}
2023-05-01 00:17:51.764738+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:17:51.764 UTC","user_name":"postgres","database_name":"postgres","process_id":"34","connection_from":"[local]","session_id":"644f052f.22","session_line_num":"1","session_start_time":"2023-05-01 00:17:51 UTC","transaction_id":"0","error_severity":"FATAL","sql_state_code":"57P03","message":"the database system is starting up","backend_type":"client backend","query_id":"0"}}
2023-05-01 00:17:51.765497+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","msg":"DB not available, will retry","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"4baf6899-bf4f-4eb6-86bb-b15f7a813896","uuid":"9c66e9fa-e7b5-11ed-ad8b-b6ea5f78a982","logging_pod":"tt-rss-cnpg-main-1","err":"failed to connect to `host=/controller/run user=postgres database=postgres`: server error (FATAL: the database system is starting up (SQLSTATE 57P03))"}
2023-05-01 00:17:51.766084+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:17:51.765 UTC","user_name":"postgres","database_name":"postgres","process_id":"35","connection_from":"[local]","session_id":"644f052f.23","session_line_num":"1","session_start_time":"2023-05-01 00:17:51 UTC","transaction_id":"0","error_severity":"FATAL","sql_state_code":"57P03","message":"the database system is starting up","backend_type":"client backend","query_id":"0"}}
2023-05-01 00:17:51.768405+00:00{"level":"info","ts":"2023-05-01T00:17:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:17:51.768 UTC","process_id":"28","session_id":"644f052f.1c","session_line_num":"6","session_start_time":"2023-05-01 00:17:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"database system is ready to accept connections","backend_type":"postmaster","query_id":"0"}}
2023-05-01 00:17:52.271432+00:00{"level":"info","ts":"2023-05-01T00:17:52Z","msg":"Readiness probe failing","logging_pod":"tt-rss-cnpg-main-1","err":"instance is not ready yet"}
2023-05-01 00:17:52.745081+00:00{"level":"info","ts":"2023-05-01T00:17:52Z","msg":"Ignore minSyncReplicas to enforce self-healing","logging_pod":"tt-rss-cnpg-main-1","syncReplicas":-1,"minSyncReplicas":0,"maxSyncReplicas":0}
2023-05-01 00:17:52.753554+00:00{"level":"info","ts":"2023-05-01T00:17:52Z","msg":"Ignore minSyncReplicas to enforce self-healing","logging_pod":"tt-rss-cnpg-main-1","syncReplicas":-1,"minSyncReplicas":0,"maxSyncReplicas":0}
2023-05-01 00:17:53.076979+00:00{"level":"info","ts":"2023-05-01T00:17:53Z","msg":"Ignore minSyncReplicas to enforce self-healing","logging_pod":"tt-rss-cnpg-main-1","syncReplicas":-1,"minSyncReplicas":0,"maxSyncReplicas":0}
2023-05-01 00:17:53.081576+00:00{"level":"info","ts":"2023-05-01T00:17:53Z","msg":"Ignore minSyncReplicas to enforce self-healing","logging_pod":"tt-rss-cnpg-main-1","syncReplicas":-1,"minSyncReplicas":0,"maxSyncReplicas":0}
2023-05-01 00:18:07.901081+00:00{"level":"info","ts":"2023-05-01T00:18:07Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:18:07.900 UTC","user_name":"tt-rss","database_name":"tt-rss","process_id":"182","connection_from":"172.16.3.59:33570","session_id":"644f053f.b6","session_line_num":"1","command_tag":"SELECT","session_start_time":"2023-05-01 00:18:07 UTC","virtual_transaction_id":"3/248","transaction_id":"0","error_severity":"ERROR","sql_state_code":"42P01","message":"relation \"ttrss_version\" does not exist","query":"select * from ttrss_version","query_pos":"15","application_name":"psql","backend_type":"client backend","query_id":"0"}}
2023-05-01 00:18:07.981958+00:00{"level":"info","ts":"2023-05-01T00:18:07Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:18:07.981 UTC","user_name":"tt-rss","database_name":"tt-rss","process_id":"184","connection_from":"172.16.3.59:33586","session_id":"644f053f.b8","session_line_num":"1","command_tag":"PARSE","session_start_time":"2023-05-01 00:18:07 UTC","virtual_transaction_id":"3/263","transaction_id":"0","error_severity":"ERROR","sql_state_code":"42P01","message":"relation \"ttrss_version\" does not exist","query":"SELECT * FROM ttrss_version","query_pos":"15","backend_type":"client backend","query_id":"0"}}
2023-05-01 00:18:29.480838+00:00{"level":"info","ts":"2023-05-01T00:18:29Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:18:29.480 UTC","process_id":"31","session_id":"644f052f.1f","session_line_num":"1","session_start_time":"2023-05-01 00:17:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"checkpoint starting: force wait","backend_type":"checkpointer","query_id":"0"}}
2023-05-01 00:18:29.525425+00:00{"level":"info","ts":"2023-05-01T00:18:29Z","logger":"wal-archive","msg":"Backup not configured, skip WAL archiving","logging_pod":"tt-rss-cnpg-main-1","walName":"pg_wal/000000010000000000000001","currentPrimary":"tt-rss-cnpg-main-1","targetPrimary":"tt-rss-cnpg-main-1"}
2023-05-01 00:19:02.524157+00:00{"level":"info","ts":"2023-05-01T00:19:02Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:19:02.524 UTC","process_id":"31","session_id":"644f052f.1f","session_line_num":"2","session_start_time":"2023-05-01 00:17:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"checkpoint complete: wrote 332 buffers (2.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=32.957 s, sync=0.067 s, total=33.044 s; sync files=152, longest=0.006 s, average=0.001 s; distance=7053 kB, estimate=7053 kB","backend_type":"checkpointer","query_id":"0"}}
2023-05-01 00:19:03.224471+00:00{"level":"info","ts":"2023-05-01T00:19:03Z","logger":"wal-archive","msg":"Backup not configured, skip WAL archiving","logging_pod":"tt-rss-cnpg-main-1","walName":"pg_wal/000000010000000000000002","currentPrimary":"tt-rss-cnpg-main-1","targetPrimary":"tt-rss-cnpg-main-1"}
2023-05-01 00:19:03.656956+00:00{"level":"info","ts":"2023-05-01T00:19:03Z","logger":"wal-archive","msg":"Backup not configured, skip WAL archiving","logging_pod":"tt-rss-cnpg-main-1","walName":"pg_wal/000000010000000000000002.00000028.backup","currentPrimary":"tt-rss-cnpg-main-1","targetPrimary":"tt-rss-cnpg-main-1"}
2023-05-01 00:23:29.660369+00:00{"level":"info","ts":"2023-05-01T00:23:29Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:23:29.657 UTC","process_id":"31","session_id":"644f052f.1f","session_line_num":"3","session_start_time":"2023-05-01 00:17:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"checkpoint starting: time","backend_type":"checkpointer","query_id":"0"}}
2023-05-01 00:23:34.218213+00:00{"level":"info","ts":"2023-05-01T00:23:34Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-1","record":{"log_time":"2023-05-01 00:23:34.217 UTC","process_id":"31","session_id":"644f052f.1f","session_line_num":"4","session_start_time":"2023-05-01 00:17:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"checkpoint complete: wrote 46 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.508 s, sync=0.036 s, total=4.561 s; sync files=14, longest=0.010 s, average=0.003 s; distance=16411 kB, estimate=16411 kB","backend_type":"checkpointer","query_id":"0"}}
2023-05-01 00:24:04.005718+00:00{"level":"info","ts":"2023-05-01T00:24:04Z","logger":"wal-archive","msg":"Backup not configured, skip WAL archiving","logging_pod":"tt-rss-cnpg-main-1","walName":"pg_wal/000000010000000000000003","currentPrimary":"tt-rss-cnpg-main-1","targetPrimary":"tt-rss-cnpg-main-1"}
Application Name:tt-rss Pod Name:tt-rss-cnpg-main-1 Container Name:bootstrap-controller
2023-05-01 00:17:34.158060+00:00{"level":"info","ts":"2023-05-01T00:17:34Z","msg":"Installing the manager executable","destination":"/controller/manager","version":"1.19.0","build":{"Version":"1.19.0","Commit":"d9bf88dd","Date":"2023-02-14"}}
2023-05-01 00:17:35.150391+00:00{"level":"info","ts":"2023-05-01T00:17:35Z","msg":"Setting 0750 permissions"}
2023-05-01 00:17:35.150544+00:00{"level":"info","ts":"2023-05-01T00:17:35Z","msg":"Bootstrap completed"}
Application Name:tt-rss Pod Name:tt-rss-77444564bc-j5vsp Container Name:tt-rss-system-cnpg-wait
2023-05-01 00:16:30.676288+00:00Executing DB waits...
2023-05-01 00:16:30.676311+00:00Detected RW pooler, testing RW pooler availability...
2023-05-01 00:16:30.676316+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:16:30.702139+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:16:35.704367+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:16:35.720461+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:16:40.720972+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:16:40.738165+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:16:45.738978+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:16:45.780375+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:16:50.781077+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:16:50.797145+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:16:55.797653+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:16:55.813368+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:17:00.813963+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:17:00.830236+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:17:05.831087+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:17:05.846591+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:17:10.847095+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:17:10.863888+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:17:15.864496+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:17:15.880790+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:17:20.881371+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:17:20.897314+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:17:25.898020+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:17:25.915000+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:17:30.915655+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:17:30.932700+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:17:35.933623+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:17:35.949335+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:17:40.949965+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:17:40.965896+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:17:45.966487+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:17:45.983810+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:17:50.984587+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:17:51.001755+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:17:56.002386+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:17:56.019274+00:00tt-rss-cnpg-main-rw:5432 - no response
2023-05-01 00:18:01.019792+00:00Testing database on url: tt-rss-cnpg-main-rw
2023-05-01 00:18:01.048422+00:00tt-rss-cnpg-main-rw:5432 - accepting connections
Application Name:tt-rss Pod Name:tt-rss-cnpg-main-2 Container Name:postgres
2023-05-01 00:19:50.244692+00:00{"level":"info","ts":"2023-05-01T00:19:50Z","logger":"setup","msg":"Starting CloudNativePG Instance Manager","logging_pod":"tt-rss-cnpg-main-2","version":"1.19.0","build":{"Version":"1.19.0","Commit":"d9bf88dd","Date":"2023-02-14"}}
2023-05-01 00:19:51.554723+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"setup","msg":"starting controller-runtime manager","logging_pod":"tt-rss-cnpg-main-2"}
2023-05-01 00:19:51.555242+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Starting EventSource","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","source":"kind source: *v1.Cluster"}
2023-05-01 00:19:51.555271+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Starting Controller","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster"}
2023-05-01 00:19:51.555290+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Starting webserver","logging_pod":"tt-rss-cnpg-main-2","address":":9187"}
2023-05-01 00:19:51.555300+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Starting webserver","logging_pod":"tt-rss-cnpg-main-2","address":":8000"}
2023-05-01 00:19:51.555311+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Starting webserver","logging_pod":"tt-rss-cnpg-main-2","address":"localhost:8010"}
2023-05-01 00:19:51.655707+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Starting workers","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","worker count":1}
2023-05-01 00:19:51.656038+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Readiness probe failing","logging_pod":"tt-rss-cnpg-main-2","err":"instance is not ready yet"}
2023-05-01 00:19:51.709152+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Refreshed configuration file","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"ae292db0-2e0d-4901-b118-edd93bb27b1d","uuid":"e3e00060-e7b5-11ed-817e-3eb70d9f8da1","logging_pod":"tt-rss-cnpg-main-2","filename":"/controller/certificates/server.crt","secret":"tt-rss-cnpg-main-server"}
2023-05-01 00:19:51.711982+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Refreshed configuration file","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"ae292db0-2e0d-4901-b118-edd93bb27b1d","uuid":"e3e00060-e7b5-11ed-817e-3eb70d9f8da1","logging_pod":"tt-rss-cnpg-main-2","filename":"/controller/certificates/server.key","secret":"tt-rss-cnpg-main-server"}
2023-05-01 00:19:51.716168+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Refreshed configuration file","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"ae292db0-2e0d-4901-b118-edd93bb27b1d","uuid":"e3e00060-e7b5-11ed-817e-3eb70d9f8da1","logging_pod":"tt-rss-cnpg-main-2","filename":"/controller/certificates/streaming_replica.crt","secret":"tt-rss-cnpg-main-replication"}
2023-05-01 00:19:51.719009+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Refreshed configuration file","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"ae292db0-2e0d-4901-b118-edd93bb27b1d","uuid":"e3e00060-e7b5-11ed-817e-3eb70d9f8da1","logging_pod":"tt-rss-cnpg-main-2","filename":"/controller/certificates/streaming_replica.key","secret":"tt-rss-cnpg-main-replication"}
2023-05-01 00:19:51.723278+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Refreshed configuration file","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"ae292db0-2e0d-4901-b118-edd93bb27b1d","uuid":"e3e00060-e7b5-11ed-817e-3eb70d9f8da1","logging_pod":"tt-rss-cnpg-main-2","filename":"/controller/certificates/client-ca.crt","secret":"tt-rss-cnpg-main-ca"}
2023-05-01 00:19:51.727672+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Refreshed configuration file","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"ae292db0-2e0d-4901-b118-edd93bb27b1d","uuid":"e3e00060-e7b5-11ed-817e-3eb70d9f8da1","logging_pod":"tt-rss-cnpg-main-2","filename":"/controller/certificates/server-ca.crt","secret":"tt-rss-cnpg-main-ca"}
2023-05-01 00:19:51.730790+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Updated replication settings in postgresql.auto.conf file","logging_pod":"tt-rss-cnpg-main-2"}
2023-05-01 00:19:51.733901+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Extracting pg_controldata information","logging_pod":"tt-rss-cnpg-main-2","reason":"postmaster start up"}
2023-05-01 00:19:51.735243+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"pg_controldata","msg":"pg_control version number: 1300\nCatalog version number: 202209061\nDatabase system identifier: 7228001436425232409\nDatabase cluster state: in production\npg_control last modified: Mon 01 May 2023 12:19:02 AM UTC\nLatest checkpoint location: 0/2053B90\nLatest checkpoint's REDO location: 0/2000028\nLatest checkpoint's REDO WAL file: 000000010000000000000002\nLatest checkpoint's TimeLineID: 1\nLatest checkpoint's PrevTimeLineID: 1\nLatest checkpoint's full_page_writes: on\nLatest checkpoint's NextXID: 0:743\nLatest checkpoint's NextOID: 24578\nLatest checkpoint's NextMultiXactId: 1\nLatest checkpoint's NextMultiOffset: 0\nLatest checkpoint's oldestXID: 716\nLatest checkpoint's oldestXID's DB: 1\nLatest checkpoint's oldestActiveXID: 743\nLatest checkpoint's oldestMultiXid: 1\nLatest checkpoint's oldestMulti's DB: 1\nLatest checkpoint's oldestCommitTsXid:0\nLatest checkpoint's newestCommitTsXid:0\nTime of latest checkpoint: Mon 01 May 2023 12:18:29 AM UTC\nFake LSN counter for unlogged rels: 0/3E8\nMinimum recovery ending location: 0/0\nMin recovery ending loc's timeline: 0\nBackup start location: 0/0\nBackup end location: 0/0\nEnd-of-backup record required: no\nwal_level setting: logical\nwal_log_hints setting: on\nmax_connections setting: 100\nmax_worker_processes setting: 32\nmax_wal_senders setting: 10\nmax_prepared_xacts setting: 0\nmax_locks_per_xact setting: 64\ntrack_commit_timestamp setting: off\nMaximum data alignment: 8\nDatabase block size: 8192\nBlocks per segment of large relation: 131072\nWAL block size: 8192\nBytes per WAL segment: 16777216\nMaximum length of identifiers: 64\nMaximum columns in an index: 32\nMaximum size of a TOAST chunk: 1996\nSize of a large-object chunk: 2048\nDate/time type storage: 64-bit integers\nFloat8 argument passing: by value\nData page checksum version: 0\nMock authentication nonce: a2242f73221a20076d18df5af9595eb833ace467bd909e72b8c21dd4db20060e\n","pipe":"stdout","logging_pod":"tt-rss-cnpg-main-2"}
2023-05-01 00:19:51.735315+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"The PID file content is wrong, deleting it and assuming it's stale","file":"/var/lib/postgresql/data/pgdata/postmaster.pid","logging_pod":"tt-rss-cnpg-main-2","err":"file does not exist","pidFileContents":""}
2023-05-01 00:19:51.750970+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"postgres","msg":"2023-05-01 00:19:51.750 UTC [24] LOG: redirecting log output to logging collector process","pipe":"stderr","logging_pod":"tt-rss-cnpg-main-2"}
2023-05-01 00:19:51.750990+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"postgres","msg":"2023-05-01 00:19:51.750 UTC [24] HINT: Future log output will appear in directory \"/controller/log\".","pipe":"stderr","logging_pod":"tt-rss-cnpg-main-2"}
2023-05-01 00:19:51.751272+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-2","record":{"log_time":"2023-05-01 00:19:51.750 UTC","process_id":"24","session_id":"644f05a7.18","session_line_num":"1","session_start_time":"2023-05-01 00:19:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"ending log output to stderr","hint":"Future log output will go to log destination \"csvlog\".","backend_type":"postmaster","query_id":"0"}}
2023-05-01 00:19:51.751288+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-2","record":{"log_time":"2023-05-01 00:19:51.750 UTC","process_id":"24","session_id":"644f05a7.18","session_line_num":"2","session_start_time":"2023-05-01 00:19:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"starting PostgreSQL 15.2 (Debian 15.2-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit","backend_type":"postmaster","query_id":"0"}}
2023-05-01 00:19:51.751297+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-2","record":{"log_time":"2023-05-01 00:19:51.751 UTC","process_id":"24","session_id":"644f05a7.18","session_line_num":"3","session_start_time":"2023-05-01 00:19:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"listening on IPv4 address \"0.0.0.0\", port 5432","backend_type":"postmaster","query_id":"0"}}
2023-05-01 00:19:51.751308+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-2","record":{"log_time":"2023-05-01 00:19:51.751 UTC","process_id":"24","session_id":"644f05a7.18","session_line_num":"4","session_start_time":"2023-05-01 00:19:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"listening on IPv6 address \"::\", port 5432","backend_type":"postmaster","query_id":"0"}}
2023-05-01 00:19:51.751317+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"postgres","msg":"2023-05-01 00:19:51.750 UTC [24] LOG: ending log output to stderr","source":"/controller/log/postgres","logging_pod":"tt-rss-cnpg-main-2"}
2023-05-01 00:19:51.751324+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"postgres","msg":"2023-05-01 00:19:51.750 UTC [24] HINT: Future log output will go to log destination \"csvlog\".","source":"/controller/log/postgres","logging_pod":"tt-rss-cnpg-main-2"}
2023-05-01 00:19:51.751907+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","msg":"Instance is still down, will retry in 1 second","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"tt-rss-cnpg-main","namespace":"ix-tt-rss"},"namespace":"ix-tt-rss","name":"tt-rss-cnpg-main","reconcileID":"ae292db0-2e0d-4901-b118-edd93bb27b1d","uuid":"e3e00060-e7b5-11ed-817e-3eb70d9f8da1","logging_pod":"tt-rss-cnpg-main-2"}
2023-05-01 00:19:51.756437+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-2","record":{"log_time":"2023-05-01 00:19:51.756 UTC","process_id":"24","session_id":"644f05a7.18","session_line_num":"5","session_start_time":"2023-05-01 00:19:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"listening on Unix socket \"/controller/run/.s.PGSQL.5432\"","backend_type":"postmaster","query_id":"0"}}
2023-05-01 00:19:51.765897+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-2","record":{"log_time":"2023-05-01 00:19:51.765 UTC","process_id":"28","session_id":"644f05a7.1c","session_line_num":"1","session_start_time":"2023-05-01 00:19:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"database system was interrupted; last known up at 2023-05-01 00:19:02 UTC","backend_type":"startup","query_id":"0"}}
2023-05-01 00:19:51.795379+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"wal-restore","msg":"tried restoring WALs, but no backup was configured","logging_pod":"tt-rss-cnpg-main-2"}
2023-05-01 00:19:51.896348+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-2","record":{"log_time":"2023-05-01 00:19:51.896 UTC","process_id":"28","session_id":"644f05a7.1c","session_line_num":"2","session_start_time":"2023-05-01 00:19:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"entering standby mode","backend_type":"startup","query_id":"0"}}
2023-05-01 00:19:51.910023+00:00{"level":"info","ts":"2023-05-01T00:19:51Z","logger":"wal-restore","msg":"tried restoring WALs, but no backup was configured","logging_pod":"tt-rss-cnpg-main-2"}
2023-05-01 00:19:52.018429+00:00{"level":"info","ts":"2023-05-01T00:19:52Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-2","record":{"log_time":"2023-05-01 00:19:52.018 UTC","process_id":"28","session_id":"644f05a7.1c","session_line_num":"3","session_start_time":"2023-05-01 00:19:51 UTC","virtual_transaction_id":"1/0","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"redo starts at 0/2000028","backend_type":"startup","query_id":"0"}}
2023-05-01 00:19:52.033764+00:00{"level":"info","ts":"2023-05-01T00:19:52Z","logger":"wal-restore","msg":"tried restoring WALs, but no backup was configured","logging_pod":"tt-rss-cnpg-main-2"}
2023-05-01 00:19:52.137961+00:00{"level":"info","ts":"2023-05-01T00:19:52Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-2","record":{"log_time":"2023-05-01 00:19:52.137 UTC","process_id":"28","session_id":"644f05a7.1c","session_line_num":"4","session_start_time":"2023-05-01 00:19:51 UTC","virtual_transaction_id":"1/0","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"consistent recovery state reached at 0/2053C30","backend_type":"startup","query_id":"0"}}
2023-05-01 00:19:52.137990+00:00{"level":"info","ts":"2023-05-01T00:19:52Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-2","record":{"log_time":"2023-05-01 00:19:52.137 UTC","process_id":"24","session_id":"644f05a7.18","session_line_num":"6","session_start_time":"2023-05-01 00:19:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"database system is ready to accept read-only connections","backend_type":"postmaster","query_id":"0"}}
2023-05-01 00:19:52.151780+00:00{"level":"info","ts":"2023-05-01T00:19:52Z","logger":"wal-restore","msg":"tried restoring WALs, but no backup was configured","logging_pod":"tt-rss-cnpg-main-2"}
2023-05-01 00:19:52.257230+00:00{"level":"info","ts":"2023-05-01T00:19:52Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-2","record":{"log_time":"2023-05-01 00:19:52.257 UTC","process_id":"94","session_id":"644f05a8.5e","session_line_num":"1","session_start_time":"2023-05-01 00:19:52 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"started streaming WAL from primary at 0/3000000 on timeline 1","backend_type":"walreceiver","query_id":"0"}}
2023-05-01 00:24:51.863730+00:00{"level":"info","ts":"2023-05-01T00:24:51Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-2","record":{"log_time":"2023-05-01 00:24:51.861 UTC","process_id":"26","session_id":"644f05a7.1a","session_line_num":"1","session_start_time":"2023-05-01 00:19:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"restartpoint starting: time","backend_type":"checkpointer","query_id":"0"}}
2023-05-01 00:24:56.429649+00:00{"level":"info","ts":"2023-05-01T00:24:56Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-2","record":{"log_time":"2023-05-01 00:24:56.429 UTC","process_id":"26","session_id":"644f05a7.1a","session_line_num":"2","session_start_time":"2023-05-01 00:19:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"restartpoint complete: wrote 47 buffers (0.3%); 1 WAL file(s) added, 0 removed, 0 recycled; write=4.513 s, sync=0.029 s, total=4.568 s; sync files=14, longest=0.006 s, average=0.003 s; distance=16411 kB, estimate=16411 kB","backend_type":"checkpointer","query_id":"0"}}
2023-05-01 00:24:56.429686+00:00{"level":"info","ts":"2023-05-01T00:24:56Z","logger":"postgres","msg":"record","logging_pod":"tt-rss-cnpg-main-2","record":{"log_time":"2023-05-01 00:24:56.429 UTC","process_id":"26","session_id":"644f05a7.1a","session_line_num":"3","session_start_time":"2023-05-01 00:19:51 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"recovery restart point at 0/3006F88","detail":"Last completed transaction was at log time 2023-05-01 00:18:41.88056+00.","backend_type":"checkpointer","query_id":"0"}}
Application Name:tt-rss Pod Name:tt-rss-cnpg-main-2 Container Name:bootstrap-controller
2023-05-01 00:19:34.879384+00:00{"level":"info","ts":"2023-05-01T00:19:34Z","msg":"Installing the manager executable","destination":"/controller/manager","version":"1.19.0","build":{"Version":"1.19.0","Commit":"d9bf88dd","Date":"2023-02-14"}}
2023-05-01 00:19:35.737699+00:00{"level":"info","ts":"2023-05-01T00:19:35Z","msg":"Setting 0750 permissions"}
2023-05-01 00:19:35.737724+00:00{"level":"info","ts":"2023-05-01T00:19:35Z","msg":"Bootstrap completed"}
I believe I'm seeing the same issue.
If I open a shell in the main tt-rss app and run mkdir /var/run/php && service php7.4-fpm restart
php-fpm appears to work.
But then I run into a separate error about the database:
I'm probably experiencing the same issue.
Something fishy seems to be going on with the databases in general. Because when I tried to stop tt-rss both database pods got stuck in the Terminating-State.
Nonetheless, I found a temporary fix the issue that appears after creating the php-7.4.sock
file and restarting the php7.4-fpm
service as described by @ctag.
I "fixed" it by creating both the /run/php/php7.4-fpm.sock
file as well as the /var/run/postgresql/.s.PGSQL.5432
file and setting the ownership on both to www-data:www-data
. Then I restarted the php7.4-fpm
service by executing the command for starting it in the /entrypoint.sh
file.
Here are the commands:
# create the php7.4-fpm socket
mkdir /run/php;
touch /run/php/php7.4-fpm.sock;
chown www-data:www-data /run/php -R;
# create the DB connection socket
mkdir /var/run/postgresql;
touch /var/run/postgresql/.s.PGSQL.5432;
chown www-data:www-data /var/run/postgresql -R;
# start the php7.4-fpm service via the same command as in the entrypoint.sh script
/usr/sbin/php-fpm7.4;
Clearly something is going wrong when the containers are created, but at least there is a temporary fix until the actual issue is found and fixed.
Edit: Turns out that you have to apply this fix after each update manually as it affects files that get reset on container recreation.
By now this container is heavily outdated, so this will likely not get any attention untill that issue is resolved. But feel free to PR a fix.
App Name
tt-rss
SCALE Version
22.02.3
App Version
2.09113_11.0.0
Application Events
Application Logs
Application Name:tt-rss Pod Name:tt-rss-77444564bc-j5vsp Container Name:tt-rss
2023-05-01 00:18:08.389832+00:00[01-May-2023 08:18:08] ERROR: unable to bind listening socket for address '/run/php/php7.4-fpm.sock': No such file or directory (2) 2023-05-01 00:18:08.389855+00:00[01-May-2023 08:18:08] ERROR: FPM initialization failed