pgcat cannot start if one of the replica servers is not reachable

Description Pgcat fails to start when one of the servers (replica) is not reachable.

To Reproduce Steps to reproduce the behavior:

Add two servers in the configuration of your pool, one reachable and one unreachable:

[pools."testdb".shards.0]
servers = [
[ "aurora-primary-db", 5432, "primary" ],
[ "aurora-replica-db", 5432, "replica" ],
]
database = "testdb"

Check that it is not working and the logs are similar to:

2024-09-17T11:01:33.329262Z  INFO ThreadId(01) pgcat: Welcome to PgCat! Meow. (Version 1.1.2-dev4)
2024-09-17T11:01:33.333039Z  INFO ThreadId(01) pgcat: Running on 0.0.0.0:6432
2024-09-17T11:01:33.333061Z  INFO ThreadId(01) pgcat::config: Config path: pgcat.toml
2024-09-17T11:01:33.333065Z  INFO ThreadId(01) pgcat::config: Ban time: 60s
2024-09-17T11:01:33.333068Z  INFO ThreadId(01) pgcat::config: Idle client in transaction timeout: 0ms
2024-09-17T11:01:33.333071Z  INFO ThreadId(01) pgcat::config: Worker threads: 4
2024-09-17T11:01:33.333074Z  INFO ThreadId(01) pgcat::config: Healthcheck timeout: 1000ms
2024-09-17T11:01:33.333077Z  INFO ThreadId(01) pgcat::config: Connection timeout: 5000ms
2024-09-17T11:01:33.333081Z  INFO ThreadId(01) pgcat::config: Idle timeout: 30000ms
2024-09-17T11:01:33.333084Z  INFO ThreadId(01) pgcat::config: Log client connections: false
2024-09-17T11:01:33.333088Z  INFO ThreadId(01) pgcat::config: Log client disconnections: false
2024-09-17T11:01:33.333091Z  INFO ThreadId(01) pgcat::config: Shutdown timeout: 60000ms
2024-09-17T11:01:33.333094Z  INFO ThreadId(01) pgcat::config: Healthcheck delay: 30000ms
2024-09-17T11:01:33.333097Z  INFO ThreadId(01) pgcat::config: Default max server lifetime: 86400000ms
2024-09-17T11:01:33.333100Z  INFO ThreadId(01) pgcat::config: Sever round robin: true
2024-09-17T11:01:33.333104Z  INFO ThreadId(01) pgcat::config: TLS support is disabled
2024-09-17T11:01:33.333107Z  INFO ThreadId(01) pgcat::config: Server TLS enabled: true
2024-09-17T11:01:33.333110Z  INFO ThreadId(01) pgcat::config: Server TLS certificate verification: false
2024-09-17T11:01:33.333113Z  INFO ThreadId(01) pgcat::config: Plugins: not configured
2024-09-17T11:01:33.333117Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Maximum user connections: 10000
2024-09-17T11:01:33.333121Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Default pool mode: transaction
2024-09-17T11:01:33.333124Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Load Balancing mode: Random
2024-09-17T11:01:33.333127Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Connection timeout: 5000ms
2024-09-17T11:01:33.333131Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Idle timeout: 30000ms
2024-09-17T11:01:33.333135Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Sharding function: pg_bigint_hash
2024-09-17T11:01:33.333138Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Primary reads: true
2024-09-17T11:01:33.333142Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Query router: true
2024-09-17T11:01:33.333145Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Query parser max length: None
2024-09-17T11:01:33.333149Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Infer role from query: true
2024-09-17T11:01:33.333152Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Number of shards: 1
2024-09-17T11:01:33.333156Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Number of users: 1
2024-09-17T11:01:33.333160Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Max server lifetime: default
2024-09-17T11:01:33.333163Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Cleanup server connections: true
2024-09-17T11:01:33.333167Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Log client parameter status changes: false
2024-09-17T11:01:33.333171Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Prepared statements server cache size: 500
2024-09-17T11:01:33.333177Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Plugins: not configured
2024-09-17T11:01:33.333184Z  INFO ThreadId(01) pgcat::config: [pool: testdb][user: testdb] Pool size: 10000
2024-09-17T11:01:33.333190Z  INFO ThreadId(01) pgcat::config: [pool: testdb][user: testdb] Minimum pool size: 3
2024-09-17T11:01:33.333195Z  INFO ThreadId(01) pgcat::config: [pool: testdb][user: testdb] Statement timeout: 0
2024-09-17T11:01:33.333200Z  INFO ThreadId(01) pgcat::config: [pool: testdb][user: testdb] Pool mode: transaction
2024-09-17T11:01:33.333206Z  INFO ThreadId(01) pgcat::config: [pool: testdb][user: testdb] Max server lifetime: 60000ms
2024-09-17T11:01:33.333212Z  INFO ThreadId(01) pgcat::config: [pool: testdb][user: testdb] Connection timeout: not set
2024-09-17T11:01:33.333218Z  INFO ThreadId(01) pgcat::config: [pool: testdb][user: testdb] Idle timeout: not set
2024-09-17T11:01:33.333468Z  INFO ThreadId(01) pgcat::pool: [pool: testdb][user: testdb] creating new pool
2024-09-17T11:01:33.333550Z  INFO ThreadId(01) pgcat::pool: Creating a new server connection Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T11:01:33.333735Z  INFO ThreadId(01) pgcat::pool: Creating a new server connection Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T11:01:33.333967Z  INFO ThreadId(01) pgcat::pool: Creating a new server connection Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T11:01:33.401033Z  INFO ThreadId(01) pgcat::pool: Creating a new server connection Address { id: 1, host: "aurora-replica-db", port: 5432, shard: 0, database: "testdb", role: Replica, replica_number: 0, address_index: 1, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T11:01:33.401092Z  INFO ThreadId(01) pgcat::pool: Creating a new server connection Address { id: 1, host: "aurora-replica-db", port: 5432, shard: 0, database: "testdb", role: Replica, replica_number: 0, address_index: 1, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T11:01:33.401122Z  INFO ThreadId(01) pgcat::pool: Creating a new server connection Address { id: 1, host: "aurora-replica-db", port: 5432, shard: 0, database: "testdb", role: Replica, replica_number: 0, address_index: 1, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T11:02:33.333992Z  INFO ThreadId(04) pgcat::server: Server connection closed Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }, session duration: 0d 00:00:59.942
2024-09-17T11:02:33.334177Z  INFO ThreadId(04) pgcat::server: Server connection closed Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }, session duration: 0d 00:00:59.942
2024-09-17T11:02:33.334266Z  INFO ThreadId(04) pgcat::server: Server connection closed Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }, session duration: 0d 00:00:59.933
2024-09-17T11:02:33.334419Z  INFO ThreadId(04) pgcat::pool: Creating a new server connection Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T11:02:33.334535Z  INFO ThreadId(04) pgcat::pool: Creating a new server connection Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T11:02:33.334592Z  INFO ThreadId(04) pgcat::pool: Creating a new server connection Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T11:03:33.334083Z  INFO ThreadId(04) pgcat::server: Server connection closed Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }, session duration: 0d 00:00:59.943
2024-09-17T11:03:33.334224Z  INFO ThreadId(04) pgcat::server: Server connection closed Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }, session duration: 0d 00:00:59.942
2024-09-17T11:03:33.334322Z  INFO ThreadId(04) pgcat::server: Server connection closed Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }, session duration: 0d 00:00:59.942
2024-09-17T11:03:33.334438Z  INFO ThreadId(04) pgcat::pool: Creating a new server connection Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T11:03:33.334536Z  INFO ThreadId(04) pgcat::pool: Creating a new server connection Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T11:03:33.334620Z  INFO ThreadId(04) pgcat::pool: Creating a new server connection Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T11:03:48.455949Z ERROR ThreadId(01) pgcat::server: Could not connect to server: Connection timed out (os error 110)
2024-09-17T11:03:48.456079Z  INFO ThreadId(01) pgcat::server: Server connection closed Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }, session duration: 0d 00:00:15.055
2024-09-17T11:03:48.456250Z  INFO ThreadId(01) pgcat::server: Server connection closed Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }, session duration: 0d 00:00:15.055
2024-09-17T11:03:48.456374Z  INFO ThreadId(01) pgcat::server: Server connection closed Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "root", pool_name: "root", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }, session duration: 0d 00:00:15.054
2024-09-17T11:03:48.456501Z ERROR ThreadId(01) pgcat: Pool error: SocketError("Could not connect to server: Connection timed out (os error 110)")

Remove the unreachable server in the configuration:

[pools."testdb".shards.0]
servers = [
[ "aurora-primary-db", 5432, "primary" ],
]

Check that it is working fine. These are the expected logs:

2024-09-17T14:34:08.661921Z  INFO ThreadId(01) pgcat: Welcome to PgCat! Meow. (Version 1.1.2-dev4)
2024-09-17T14:34:08.665669Z  INFO ThreadId(01) pgcat: Running on 0.0.0.0:6432
2024-09-17T14:34:08.665689Z  INFO ThreadId(01) pgcat::config: Config path: pgcat.toml
2024-09-17T14:34:08.665693Z  INFO ThreadId(01) pgcat::config: Ban time: 60s
2024-09-17T14:34:08.665697Z  INFO ThreadId(01) pgcat::config: Idle client in transaction timeout: 0ms
2024-09-17T14:34:08.665700Z  INFO ThreadId(01) pgcat::config: Worker threads: 4
2024-09-17T14:34:08.665703Z  INFO ThreadId(01) pgcat::config: Healthcheck timeout: 1000ms
2024-09-17T14:34:08.665706Z  INFO ThreadId(01) pgcat::config: Connection timeout: 5000ms
2024-09-17T14:34:08.665709Z  INFO ThreadId(01) pgcat::config: Idle timeout: 30000ms
2024-09-17T14:34:08.665712Z  INFO ThreadId(01) pgcat::config: Log client connections: false
2024-09-17T14:34:08.665716Z  INFO ThreadId(01) pgcat::config: Log client disconnections: false
2024-09-17T14:34:08.665719Z  INFO ThreadId(01) pgcat::config: Shutdown timeout: 60000ms
2024-09-17T14:34:08.665722Z  INFO ThreadId(01) pgcat::config: Healthcheck delay: 30000ms
2024-09-17T14:34:08.665725Z  INFO ThreadId(01) pgcat::config: Default max server lifetime: 86400000ms
2024-09-17T14:34:08.665732Z  INFO ThreadId(01) pgcat::config: Sever round robin: true
2024-09-17T14:34:08.665736Z  INFO ThreadId(01) pgcat::config: TLS support is disabled
2024-09-17T14:34:08.665740Z  INFO ThreadId(01) pgcat::config: Server TLS enabled: true
2024-09-17T14:34:08.665743Z  INFO ThreadId(01) pgcat::config: Server TLS certificate verification: false
2024-09-17T14:34:08.665746Z  INFO ThreadId(01) pgcat::config: Plugins: not configured
2024-09-17T14:34:08.665750Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Maximum user connections: 10000
2024-09-17T14:34:08.665754Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Default pool mode: transaction
2024-09-17T14:34:08.665758Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Load Balancing mode: Random
2024-09-17T14:34:08.665761Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Connection timeout: 5000ms
2024-09-17T14:34:08.665765Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Idle timeout: 30000ms
2024-09-17T14:34:08.665769Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Sharding function: pg_bigint_hash
2024-09-17T14:34:08.665772Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Primary reads: true
2024-09-17T14:34:08.665775Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Query router: true
2024-09-17T14:34:08.665779Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Query parser max length: None
2024-09-17T14:34:08.665782Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Infer role from query: true
2024-09-17T14:34:08.665786Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Number of shards: 1
2024-09-17T14:34:08.665789Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Number of users: 1
2024-09-17T14:34:08.665793Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Max server lifetime: default
2024-09-17T14:34:08.665796Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Cleanup server connections: true
2024-09-17T14:34:08.665802Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Log client parameter status changes: false
2024-09-17T14:34:08.665806Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Prepared statements server cache size: 500
2024-09-17T14:34:08.665810Z  INFO ThreadId(01) pgcat::config: [pool: testdb] Plugins: not configured
2024-09-17T14:34:08.665813Z  INFO ThreadId(01) pgcat::config: [pool: testdb][user: testdb] Pool size: 10000
2024-09-17T14:34:08.665817Z  INFO ThreadId(01) pgcat::config: [pool: testdb][user: testdb] Minimum pool size: 3
2024-09-17T14:34:08.665824Z  INFO ThreadId(01) pgcat::config: [pool: testdb][user: testdb] Statement timeout: 0
2024-09-17T14:34:08.665830Z  INFO ThreadId(01) pgcat::config: [pool: testdb][user: testdb] Pool mode: transaction
2024-09-17T14:34:08.665836Z  INFO ThreadId(01) pgcat::config: [pool: testdb][user: testdb] Max server lifetime: 60000ms
2024-09-17T14:34:08.665841Z  INFO ThreadId(01) pgcat::config: [pool: testdb][user: testdb] Connection timeout: not set
2024-09-17T14:34:08.665848Z  INFO ThreadId(01) pgcat::config: [pool: testdb][user: testdb] Idle timeout: not set
2024-09-17T14:34:08.666211Z  INFO ThreadId(01) pgcat::pool: [pool: testdb][user: testdb] creating new pool
2024-09-17T14:34:08.666316Z  INFO ThreadId(01) pgcat::pool: Creating a new server connection Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "testdb", pool_name: "testdb", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T14:34:08.666707Z  INFO ThreadId(01) pgcat::pool: Creating a new server connection Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "testdb", pool_name: "testdb", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T14:34:08.666830Z  INFO ThreadId(01) pgcat::pool: Creating a new server connection Address { id: 0, host: "aurora-primary-db", port: 5432, shard: 0, database: "testdb", role: Primary, replica_number: 0, address_index: 0, username: "testdb", pool_name: "testdb", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false }, error_count: 0 }
2024-09-17T14:34:08.730908Z  INFO ThreadId(01) pgcat: Config autoreloader: disabled
2024-09-17T14:34:08.730967Z  INFO ThreadId(01) pgcat: Waiting for clients
2024-09-17T14:34:08.731368Z  INFO ThreadId(04) pgcat::stats: Events reporter started

Expected behavior I would expect that it works even if it can't reach one of the replicas. And if the primary host is not reachable, that it fails over to the replica when it becomes the writer.

Configuration This is the config.toml file:

[general]
host = "0.0.0.0"
port = 6432
enable_prometheus_exporter = false
prometheus_exporter_port = 9930
connect_timeout = 5000
idle_timeout = 30000
server_lifetime = 86400000
idle_client_in_transaction_timeout = 0
healthcheck_timeout = 1000
healthcheck_delay = 30000
shutdown_timeout = 60000
ban_time = 60
log_client_connections = false
log_client_disconnections = false
tcp_keepalives_idle = 5
tcp_keepalives_count = 5
tcp_keepalives_interval = 5
admin_username = "root"
admin_password = "rootpass"
server_tls = true

##
## pool for testdb
##
[pools."testdb"]
pool_mode = "transaction"
load_balancing_mode = "random"
default_role = "any"
prepared_statements_cache_size = 500
query_parser_enabled = true
query_parser_read_write_splitting = true
primary_reads_enabled = true
sharding_function = "pg_bigint_hash"

## pool testdb user "testdb"
##
[pools."testdb".users.0]
username = "testdb"
password = "testdbpass"
pool_size = 10000
statement_timeout = 0
min_pool_size = 3
server_lifetime = 60000
server_username = "testdb"
server_password = "testdbserverpass"

## pool testdb database testdb
##
[pools."testdb".shards.0]
servers = [
    [ "aurora-primary-db", 5432, "primary" ],
    [ "aurora-replica-db", 5432, "replica" ],
]
database = "testdb"

Additional context I am accessing Aurora Postgres RDS Global Database. I am running pgcat on EKS. I changed the auth method to md5 instead of scram. And it works when I only specify one host (the primary one). It does not start when the secondary is not reachable (which is my case at the moment).

postgresml / pgcat

pgcat cannot start if one of the replica servers is not reachable #823