Today we had a production outage of a couple minutes, suspectedly because our stolon cluster did not failover correctly.
I manually fixed the problem by restarting a keeper, but another keeper is still in an unhealthy state from the cluster's perspective, while the keeper itself seems to think that it is fine.
stolonctl status shows keeper node_6 as HEALTHY = false, PG HEALTHY = false, and PG LISTENADDRESS = (unknown):
Click to expand full `stolonctl status` output
```
# stolonctl status
=== Active sentinels ===
ID LEADER
778771af false
7c12d47f false
bfe3a503 true
d5cd2ad2 false
=== Active proxies ===
ID
26803fc5
295e41dc
62ae83e9
651751d3
c431e17d
c939976b
e0d69a43
=== Keepers ===
UID HEALTHY PG LISTENADDRESS PG HEALTHY PG WANTEDGENERATION PG CURRENTGENERATION
node_4 true 10.0.0.4:5432 true 2 2
node_5 true 10.0.0.5:5432 true 5 5
node_6 false (unknown) false 2 0
=== Cluster Info ===
Master Keeper: node_5
===== Keepers/DB tree =====
node_5 (master)
├─node_6
└─node_4
```
For reading convenience, Keepers formatted nicely:
What happened:
Today we had a production outage of a couple minutes, suspectedly because our stolon cluster did not failover correctly.
I manually fixed the problem by restarting a keeper, but another keeper is still in an unhealthy state from the cluster's perspective, while the keeper itself seems to think that it is fine.
stolonctl status
shows keepernode_6
asHEALTHY = false
,PG HEALTHY = false
, andPG LISTENADDRESS = (unknown)
:Click to expand full `stolonctl status` output
``` # stolonctl status === Active sentinels === ID LEADER 778771af false 7c12d47f false bfe3a503 true d5cd2ad2 false === Active proxies === ID 26803fc5 295e41dc 62ae83e9 651751d3 c431e17d c939976b e0d69a43 === Keepers === UID HEALTHY PG LISTENADDRESS PG HEALTHY PG WANTEDGENERATION PG CURRENTGENERATION node_4 true 10.0.0.4:5432 true 2 2 node_5 true 10.0.0.5:5432 true 5 5 node_6 false (unknown) false 2 0 === Cluster Info === Master Keeper: node_5 ===== Keepers/DB tree ===== node_5 (master) ├─node_6 └─node_4 ```For reading convenience,
Keepers
formatted nicely:Click to expand full `stolonctl spec` output
```json { "sleepInterval": "2s", "requestTimeout": "4s", "failInterval": "8s", "synchronousReplication": true, "minSynchronousStandbys": 1, "maxSynchronousStandbys": 2, "usePgrewind": true, "initMode": "new", "pgParameters": { "datestyle": "iso, mdy", "default_text_search_config": "pg_catalog.english", "dynamic_shared_memory_type": "posix", "lc_messages": "en_US.UTF-8", "lc_monetary": "en_US.UTF-8", "lc_numeric": "en_US.UTF-8", "lc_time": "en_US.UTF-8", "log_min_duration_statement": "10", "log_timezone": "UTC", "logging_collector": "on", "max_connections": "100", "max_wal_size": "1GB", "min_wal_size": "80MB", "random_page_cost": "1.1", "shared_buffers": "128MB", "timezone": "UTC", "wal_level": "replica", "work_mem": "65536" } } ```Click to expand full `stolonctl clusterdata read | jq .` output
```json { "formatVersion": 1, "changeTime": "2023-07-26T12:41:50.048697008Z", "cluster": { "uid": "735e8856", "generation": 1, "changeTime": "2023-07-26T11:41:07.029419632Z", "spec": { "sleepInterval": "2s", "requestTimeout": "4s", "failInterval": "8s", "synchronousReplication": true, "minSynchronousStandbys": 1, "maxSynchronousStandbys": 2, "additionalWalSenders": null, "additionalMasterReplicationSlots": null, "usePgrewind": true, "initMode": "new", "pgParameters": { "datestyle": "iso, mdy", "default_text_search_config": "pg_catalog.english", "dynamic_shared_memory_type": "posix", "lc_messages": "en_US.UTF-8", "lc_monetary": "en_US.UTF-8", "lc_numeric": "en_US.UTF-8", "lc_time": "en_US.UTF-8", "log_min_duration_statement": "10", "log_timezone": "UTC", "logging_collector": "on", "max_connections": "100", "max_wal_size": "1GB", "min_wal_size": "80MB", "random_page_cost": "1.1", "shared_buffers": "128MB", "timezone": "UTC", "wal_level": "replica", "work_mem": "65536" }, "pgHBA": null, "automaticPgRestart": null }, "status": { "phase": "normal", "master": "3397fb23" } }, "keepers": { "node_4": { "uid": "node_4", "generation": 1, "changeTime": "2023-07-26T12:41:50.048761009Z", "spec": {}, "status": { "healthy": true, "lastHealthyTime": "2023-07-26T12:41:50.046055293Z", "bootUUID": "e687f779-361c-4213-a5f4-157f352ed63a", "postgresBinaryVersion": { "Maj": 14, "Min": 1 }, "canBeMaster": true, "canBeSynchronousReplica": true } }, "node_5": { "uid": "node_5", "generation": 1, "changeTime": "2023-07-26T12:41:50.048755779Z", "spec": {}, "status": { "healthy": true, "lastHealthyTime": "2023-07-26T12:41:50.046055443Z", "bootUUID": "9b604f18-c48a-42f4-a774-cd49795e0e89", "postgresBinaryVersion": { "Maj": 14, "Min": 1 }, "canBeMaster": true, "canBeSynchronousReplica": true } }, "node_6": { "uid": "node_6", "generation": 1, "changeTime": "2023-07-24T14:10:46.921750869Z", "spec": {}, "status": { "lastHealthyTime": "2023-07-24T14:10:44.902643198Z", "bootUUID": "2c4a6ad5-1ed8-4a55-aa25-0c30cb2a4acc", "postgresBinaryVersion": { "Maj": 14, "Min": 1 }, "canBeMaster": true, "canBeSynchronousReplica": true } } }, "dbs": { "32e06e90": { "uid": "32e06e90", "generation": 2, "changeTime": "2023-07-26T11:41:11.057249694Z", "spec": { "keeperUID": "node_6", "requestTimeout": "4s", "maxStandbys": 20, "usePgrewind": true, "additionalWalSenders": 5, "additionalReplicationSlots": null, "initMode": "resync", "pgParameters": { "datestyle": "iso, mdy", "default_text_search_config": "pg_catalog.english", "dynamic_shared_memory_type": "posix", "lc_messages": "en_US.UTF-8", "lc_monetary": "en_US.UTF-8", "lc_numeric": "en_US.UTF-8", "lc_time": "en_US.UTF-8", "log_min_duration_statement": "10", "log_timezone": "UTC", "logging_collector": "on", "max_connections": "100", "max_wal_size": "1GB", "min_wal_size": "80MB", "random_page_cost": "1.1", "shared_buffers": "128MB", "timezone": "UTC", "wal_level": "replica", "work_mem": "65536" }, "pgHBA": null, "role": "standby", "followConfig": { "type": "internal", "dbuid": "3397fb23" }, "followers": [], "synchronousStandbys": null, "externalSynchronousStandbys": null }, "status": { "synchronousStandbys": null } }, "3397fb23": { "uid": "3397fb23", "generation": 5, "changeTime": "2023-07-26T12:41:50.048753905Z", "spec": { "keeperUID": "node_5", "requestTimeout": "4s", "maxStandbys": 20, "synchronousReplication": true, "usePgrewind": true, "additionalWalSenders": 5, "additionalReplicationSlots": null, "initMode": "none", "pgParameters": { "datestyle": "iso, mdy", "default_text_search_config": "pg_catalog.english", "dynamic_shared_memory_type": "posix", "lc_messages": "en_US.UTF-8", "lc_monetary": "en_US.UTF-8", "lc_numeric": "en_US.UTF-8", "lc_time": "en_US.UTF-8", "log_min_duration_statement": "10", "log_timezone": "UTC", "logging_collector": "on", "max_connections": "100", "max_wal_size": "1GB", "min_wal_size": "80MB", "random_page_cost": "1.1", "shared_buffers": "128MB", "timezone": "UTC", "wal_level": "replica", "work_mem": "65536" }, "pgHBA": null, "role": "master", "followers": [ "32e06e90", "c6cb2f35" ], "synchronousStandbys": [ "c6cb2f35" ], "externalSynchronousStandbys": [] }, "status": { "healthy": true, "currentGeneration": 5, "listenAddress": "10.0.0.5", "port": "5432", "systemdID": "7191260299357525396", "timelineID": 31, "xLogPos": 411870652720, "timelinesHistory": [ { "timelineID": 29, "switchPoint": 365271653264, "reason": "no recovery target specified" }, { "timelineID": 30, "switchPoint": 409000508648, "reason": "no recovery target specified" } ], "pgParameters": { "datestyle": "iso, mdy", "default_text_search_config": "pg_catalog.english", "dynamic_shared_memory_type": "posix", "lc_messages": "en_US.UTF-8", "lc_monetary": "en_US.UTF-8", "lc_numeric": "en_US.UTF-8", "lc_time": "en_US.UTF-8", "log_min_duration_statement": "10", "log_timezone": "UTC", "logging_collector": "on", "max_connections": "100", "max_wal_size": "1GB", "min_wal_size": "80MB", "random_page_cost": "1.1", "shared_buffers": "128MB", "timezone": "UTC", "wal_level": "replica", "work_mem": "65536" }, "synchronousStandbys": [ "c6cb2f35" ], "olderWalFile": "0000001F0000005F000000D1" } }, "c6cb2f35": { "uid": "c6cb2f35", "generation": 2, "changeTime": "2023-07-26T12:41:50.048725131Z", "spec": { "keeperUID": "node_4", "requestTimeout": "4s", "maxStandbys": 20, "usePgrewind": true, "additionalWalSenders": 5, "additionalReplicationSlots": null, "initMode": "none", "pgParameters": { "datestyle": "iso, mdy", "default_text_search_config": "pg_catalog.english", "dynamic_shared_memory_type": "posix", "lc_messages": "en_US.UTF-8", "lc_monetary": "en_US.UTF-8", "lc_numeric": "en_US.UTF-8", "lc_time": "en_US.UTF-8", "log_min_duration_statement": "10", "log_timezone": "UTC", "logging_collector": "on", "max_connections": "100", "max_wal_size": "1GB", "min_wal_size": "80MB", "random_page_cost": "1.1", "shared_buffers": "128MB", "timezone": "UTC", "wal_level": "replica", "work_mem": "65536" }, "pgHBA": null, "role": "standby", "followConfig": { "type": "internal", "dbuid": "3397fb23" }, "followers": [], "synchronousStandbys": null, "externalSynchronousStandbys": null }, "status": { "healthy": true, "currentGeneration": 2, "listenAddress": "10.0.0.4", "port": "5432", "systemdID": "7191260299357525396", "timelineID": 31, "xLogPos": 411870652048, "timelinesHistory": [ { "timelineID": 29, "switchPoint": 365271653264, "reason": "no recovery target specified" }, { "timelineID": 30, "switchPoint": 409000508648, "reason": "no recovery target specified" } ], "pgParameters": { "datestyle": "iso, mdy", "default_text_search_config": "pg_catalog.english", "dynamic_shared_memory_type": "posix", "lc_messages": "en_US.UTF-8", "lc_monetary": "en_US.UTF-8", "lc_numeric": "en_US.UTF-8", "lc_time": "en_US.UTF-8", "log_min_duration_statement": "10", "log_timezone": "UTC", "logging_collector": "on", "max_connections": "100", "max_wal_size": "1GB", "min_wal_size": "80MB", "random_page_cost": "1.1", "shared_buffers": "128MB", "timezone": "UTC", "wal_level": "replica", "work_mem": "65536" }, "synchronousStandbys": null, "olderWalFile": "0000001F0000005F000000C5" } } }, "proxy": { "generation": 430, "changeTime": "2023-07-26T11:41:13.06941116Z", "spec": { "masterDbUid": "3397fb23", "enabledProxies": [ "26803fc5", "295e41dc", "62ae83e9", "651751d3", "c431e17d", "c939976b", "e0d69a43" ] }, "status": {} } } ```Relevant here is
The
stolon-keeper
seems to be running fine according tosystemctl status stolon-keeper.service
:Its logs show some earlier errors while
pg_rewind
was copying data over, but no indication that anything failed permanently:Relevant sections:
What you expected to happen:
stolonctl status
finds that the keeper is up and working, orHow to reproduce it (as minimally and precisely as possible):
Unclear.
Stolon had been running uninterrupted for 2 months until this happened.
Environment:
master
commit 4bb4107523c2db09fa711c1d96ddfe33bacf405c