search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.65k stars 589 forks source link

Can't decommission node, cluster stuck in reconciliation loop? #7355

Open nivekuil opened 1 year ago

nivekuil commented 1 year ago

Version & Environment

Redpanda version: (use rpk version):

v22.2.7 (rev 96ed1cc)

What went wrong?

Try replacing node, going from 3->4->3 nodes. 3->4 works and the cluster is functional but can't decom any nodes, it's always stuck in draining.

What should have happened instead?

decommission works

How to reproduce the issue?

not sure, decom/recom nodes randomly in chaos test?

Additional information.

Logs are silent except for the last added node (node-id 3)

Nov 17 11:45:57 netcup-1 example.pod.redpanda_redpanda[3099]: INFO  2022-11-17 11:45:57,744 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false
Nov 17 11:45:57 netcup-1 example.pod.redpanda_redpanda[3099]: INFO  2022-11-17 11:45:57,745 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}
Nov 17 11:46:02 netcup-1 example.pod.redpanda_redpanda[3099]: INFO  2022-11-17 11:46:02,769 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false
Nov 17 11:46:02 netcup-1 example.pod.redpanda_redpanda[3099]: INFO  2022-11-17 11:46:02,770 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}
Nov 17 11:46:07 netcup-1 example.pod.redpanda_redpanda[3099]: INFO  2022-11-17 11:46:07,795 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false
Nov 17 11:46:07 netcup-1 example.pod.redpanda_redpanda[3099]: INFO  2022-11-17 11:46:07,795 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}
Nov 17 11:46:12 netcup-1 example.pod.redpanda_redpanda[3099]: INFO  2022-11-17 11:46:12,837 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false
Nov 17 11:46:12 netcup-1 example.pod.redpanda_redpanda[3099]: INFO  2022-11-17 11:46:12,838 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}
Nov 17 11:46:17 netcup-1 example.pod.redpanda_redpanda[3099]: INFO  2022-11-17 11:46:17,862 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false
Nov 17 11:46:17 netcup-1 example.pod.redpanda_redpanda[3099]: INFO  2022-11-17 11:46:17,862 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}
Nov 17 11:46:22 netcup-1 example.pod.redpanda_redpanda[3099]: INFO  2022-11-17 11:46:22,887 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false
Nov 17 11:46:22 netcup-1 example.pod.redpanda_redpanda[3099]: INFO  2022-11-17 11:46:22,887 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}
Nov 17 11:46:27 netcup-1 example.pod.redpanda_redpanda[3099]: INFO  2022-11-17 11:46:27,912 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false
Nov 17 11:46:27 netcup-1 example.pod.redpanda_redpanda[3099]: INFO  2022-11-17 11:46:27,912 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}

from slack thread https://redpandacommunity.slack.com/archives/C01AJDUT88N/p1667848435753559

JIRA Link: CORE-1089

emaxerrno commented 1 year ago

cc: @mattschumpert and @piyushredpanda

piyushredpanda commented 1 year ago

@nivekuil : do you have any logs you could share with us and relevant details like the amount of data you had on the node that was decommissioned? The status will remain draining until we've successfully moved the data (and of course leadership) off that node. So it is tough to know if this is a bug until we know more.

piyushredpanda commented 1 year ago

Ah, but this is suspect.

calculated reallocations: {}

cc: @dotnwat

nivekuil commented 1 year ago

those are all the logs, it's not much data. I think you have a PR for this already? https://github.com/redpanda-data/redpanda/pull/7389

mmaslankaprv commented 1 year ago

@nivekuil , would it be possible to run redpanda with trace logs enabled, so we can better understand what is happening. Here is the command: 

rpk redpanda admin config log-level set cluster -l trace --expiry-seconds 180
mmaslankaprv commented 1 year ago

also another useful piece of information would be output of: 

 http://<redpanda-ip>:9644/v1/brokers
nivekuil commented 1 year ago

From the leader (where the reallocation logs are, I omitted "updated partition:" trace logs)

2022-11-21T00:43:37-08:00   DEBUG 2022-11-21 08:43:37,708 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:36-08:00   INFO  2022-11-21 08:43:36,744 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}

2022-11-21T00:43:36-08:00   INFO  2022-11-21 08:43:36,743 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false

2022-11-21T00:43:34-08:00   DEBUG 2022-11-21 08:43:34,707 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:31-08:00   INFO  2022-11-21 08:43:31,742 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}

2022-11-21T00:43:31-08:00   INFO  2022-11-21 08:43:31,742 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false

2022-11-21T00:43:31-08:00   DEBUG 2022-11-21 08:43:31,706 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:30-08:00   TRACE 2022-11-21 08:43:30,039 [shard 0] cluster - health_monitor_backend.cc:304 - skipping metadata refresh request current metadata age: 0 ms

2022-11-21T00:43:30-08:00   TRACE 2022-11-21 08:43:30,039 [shard 0] cluster - health_monitor_backend.cc:304 - skipping metadata refresh request current metadata age: 0 ms

2022-11-21T00:43:30-08:00   DEBUG 2022-11-21 08:43:30,038 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 3: {id: 3, disks: {{path: /var/lib/redpanda/data, free: 283.466GiB, total: 319.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19197747}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 161353048, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:43:30-08:00   DEBUG 2022-11-21 08:43:30,038 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 3 health report: {id: 3, disks: {{path: /var/lib/redpanda/data, free: 283.466GiB, total: 319.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19198637}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 161363095, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:43:30-08:00   DEBUG 2022-11-21 08:43:30,038 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 2: {id: 2, disks: {{path: /var/lib/redpanda/data, free: 145.924GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331247}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19197747}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 319209634, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:43:30-08:00   DEBUG 2022-11-21 08:43:30,038 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 2 health report: {id: 2, disks: {{path: /var/lib/redpanda/data, free: 145.924GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331247}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19198637}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 319219664, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:43:30-08:00   DEBUG 2022-11-21 08:43:30,038 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 1: {id: 1, disks: {{path: /var/lib/redpanda/data, free: 150.644GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 360919524, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:43:30-08:00   DEBUG 2022-11-21 08:43:30,037 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 1 health report: {id: 1, disks: {{path: /var/lib/redpanda/data, free: 150.644GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 360929557, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:43:30-08:00   DEBUG 2022-11-21 08:43:30,037 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 0: {id: 0, disks: {{path: /var/lib/redpanda/data, free: 173.221GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19197747}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 122420150, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:43:30-08:00   DEBUG 2022-11-21 08:43:30,037 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 0 health report: {id: 0, disks: {{path: /var/lib/redpanda/data, free: 173.221GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19198637}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 122430182, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:43:30-08:00   DEBUG 2022-11-21 08:43:30,015 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:29-08:00   DEBUG 2022-11-21 08:43:29,992 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:29-08:00   DEBUG 2022-11-21 08:43:29,991 [shard 0] cluster - health_monitor_backend.cc:620 - collecting health report with filter: {include_partitions: true, ntp_filters: {}}

2022-11-21T00:43:29-08:00   DEBUG 2022-11-21 08:43:29,991 [shard 0] cluster - health_monitor_backend.cc:567 - collecting cluster health statistics

2022-11-21T00:43:29-08:00   DEBUG 2022-11-21 08:43:29,991 [shard 0] cluster - health_monitor_backend.cc:312 - refreshing health cache, leader id: {0} self: 0

2022-11-21T00:43:29-08:00   DEBUG 2022-11-21 08:43:29,991 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:28-08:00   DEBUG 2022-11-21 08:43:28,704 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:28-08:00   DEBUG 2022-11-21 08:43:28,055 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: false, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:26-08:00   INFO  2022-11-21 08:43:26,741 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}

2022-11-21T00:43:26-08:00   INFO  2022-11-21 08:43:26,741 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false

2022-11-21T00:43:25-08:00   DEBUG 2022-11-21 08:43:25,704 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:22-08:00   DEBUG 2022-11-21 08:43:22,697 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:21-08:00   INFO  2022-11-21 08:43:21,740 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}

2022-11-21T00:43:21-08:00   INFO  2022-11-21 08:43:21,740 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false

2022-11-21T00:43:19-08:00   TRACE 2022-11-21 08:43:19,990 [shard 0] cluster - health_monitor_backend.cc:304 - skipping metadata refresh request current metadata age: 0 ms

2022-11-21T00:43:19-08:00   TRACE 2022-11-21 08:43:19,990 [shard 0] cluster - health_monitor_backend.cc:304 - skipping metadata refresh request current metadata age: 0 ms

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,990 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 3: {id: 3, disks: {{path: /var/lib/redpanda/data, free: 283.466GiB, total: 319.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19196857}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 161343016, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,989 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 3 health report: {id: 3, disks: {{path: /var/lib/redpanda/data, free: 283.466GiB, total: 319.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19197747}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 161353048, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,989 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 2: {id: 2, disks: {{path: /var/lib/redpanda/data, free: 145.924GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331247}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19196857}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 319199603, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,989 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 2 health report: {id: 2, disks: {{path: /var/lib/redpanda/data, free: 145.924GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331247}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19197747}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 319209634, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,989 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 1: {id: 1, disks: {{path: /var/lib/redpanda/data, free: 150.644GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 360909495, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,989 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 1 health report: {id: 1, disks: {{path: /var/lib/redpanda/data, free: 150.644GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 360919524, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,989 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 0: {id: 0, disks: {{path: /var/lib/redpanda/data, free: 173.221GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19196857}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 122410121, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,989 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 0 health report: {id: 0, disks: {{path: /var/lib/redpanda/data, free: 173.221GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19197747}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 122420150, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,983 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,960 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,960 [shard 0] cluster - health_monitor_backend.cc:620 - collecting health report with filter: {include_partitions: true, ntp_filters: {}}

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,960 [shard 0] cluster - health_monitor_backend.cc:567 - collecting cluster health statistics

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,959 [shard 0] cluster - health_monitor_backend.cc:312 - refreshing health cache, leader id: {0} self: 0

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,959 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:19-08:00   DEBUG 2022-11-21 08:43:19,696 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:17-08:00   DEBUG 2022-11-21 08:43:17,810 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: false, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:16-08:00   DEBUG 2022-11-21 08:43:16,935 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: false, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:16-08:00   INFO  2022-11-21 08:43:16,739 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}

2022-11-21T00:43:16-08:00   INFO  2022-11-21 08:43:16,739 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false

2022-11-21T00:43:16-08:00   DEBUG 2022-11-21 08:43:16,691 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:13-08:00   DEBUG 2022-11-21 08:43:13,690 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:11-08:00   INFO  2022-11-21 08:43:11,738 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}

2022-11-21T00:43:11-08:00   INFO  2022-11-21 08:43:11,738 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false

2022-11-21T00:43:10-08:00   DEBUG 2022-11-21 08:43:10,687 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:09-08:00   TRACE 2022-11-21 08:43:09,958 [shard 0] cluster - health_monitor_backend.cc:304 - skipping metadata refresh request current metadata age: 0 ms

2022-11-21T00:43:09-08:00   TRACE 2022-11-21 08:43:09,958 [shard 0] cluster - health_monitor_backend.cc:304 - skipping metadata refresh request current metadata age: 0 ms

2022-11-21T00:43:09-08:00   DEBUG 2022-11-21 08:43:09,958 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 3: {id: 3, disks: {{path: /var/lib/redpanda/data, free: 283.466GiB, total: 319.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19195967}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 161332988, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:43:09-08:00   DEBUG 2022-11-21 08:43:09,958 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 3 health report: {id: 3, disks: {{path: /var/lib/redpanda/data, free: 283.466GiB, total: 319.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19196857}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 161343016, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:43:09-08:00   DEBUG 2022-11-21 08:43:09,958 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 2: {id: 2, disks: {{path: /var/lib/redpanda/data, free: 145.924GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331247}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19195967}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 319189575, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:43:09-08:00   DEBUG 2022-11-21 08:43:09,958 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 2 health report: {id: 2, disks: {{path: /var/lib/redpanda/data, free: 145.924GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331247}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19196857}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 319199603, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:43:09-08:00   DEBUG 2022-11-21 08:43:09,958 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 1: {id: 1, disks: {{path: /var/lib/redpanda/data, free: 150.644GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 360899467, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:43:09-08:00   DEBUG 2022-11-21 08:43:09,958 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 1 health report: {id: 1, disks: {{path: /var/lib/redpanda/data, free: 150.644GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 360909495, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:43:09-08:00   DEBUG 2022-11-21 08:43:09,958 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 0: {id: 0, disks: {{path: /var/lib/redpanda/data, free: 173.221GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19195967}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 122400093, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:43:09-08:00   DEBUG 2022-11-21 08:43:09,958 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 0 health report: {id: 0, disks: {{path: /var/lib/redpanda/data, free: 173.221GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19196857}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 122410121, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:43:09-08:00   DEBUG 2022-11-21 08:43:09,955 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:09-08:00   DEBUG 2022-11-21 08:43:09,931 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:09-08:00   DEBUG 2022-11-21 08:43:09,931 [shard 0] cluster - health_monitor_backend.cc:620 - collecting health report with filter: {include_partitions: true, ntp_filters: {}}

2022-11-21T00:43:09-08:00   DEBUG 2022-11-21 08:43:09,931 [shard 0] cluster - health_monitor_backend.cc:567 - collecting cluster health statistics

2022-11-21T00:43:09-08:00   DEBUG 2022-11-21 08:43:09,931 [shard 0] cluster - health_monitor_backend.cc:312 - refreshing health cache, leader id: {0} self: 0

2022-11-21T00:43:09-08:00   DEBUG 2022-11-21 08:43:09,931 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:07-08:00   DEBUG 2022-11-21 08:43:07,680 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:06-08:00   INFO  2022-11-21 08:43:06,738 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}

2022-11-21T00:43:06-08:00   INFO  2022-11-21 08:43:06,737 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false

2022-11-21T00:43:04-08:00   DEBUG 2022-11-21 08:43:04,678 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:43:01-08:00   INFO  2022-11-21 08:43:01,736 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}

2022-11-21T00:43:01-08:00   INFO  2022-11-21 08:43:01,736 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false

2022-11-21T00:43:01-08:00   DEBUG 2022-11-21 08:43:01,676 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:42:59-08:00   TRACE 2022-11-21 08:42:59,930 [shard 0] cluster - health_monitor_backend.cc:304 - skipping metadata refresh request current metadata age: 0 ms

2022-11-21T00:42:59-08:00   TRACE 2022-11-21 08:42:59,930 [shard 0] cluster - health_monitor_backend.cc:304 - skipping metadata refresh request current metadata age: 0 ms

2022-11-21T00:42:59-08:00   DEBUG 2022-11-21 08:42:59,929 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 3: {id: 3, disks: {{path: /var/lib/redpanda/data, free: 283.466GiB, total: 319.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19195077}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 161322960, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:42:59-08:00   DEBUG 2022-11-21 08:42:59,929 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 3 health report: {id: 3, disks: {{path: /var/lib/redpanda/data, free: 283.466GiB, total: 319.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19195967}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 161332988, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:42:59-08:00   DEBUG 2022-11-21 08:42:59,929 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 2: {id: 2, disks: {{path: /var/lib/redpanda/data, free: 145.924GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331247}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19195077}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 319179547, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:42:59-08:00   DEBUG 2022-11-21 08:42:59,929 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 2 health report: {id: 2, disks: {{path: /var/lib/redpanda/data, free: 145.924GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331247}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19195967}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 319189575, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:42:59-08:00   DEBUG 2022-11-21 08:42:59,929 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 1: {id: 1, disks: {{path: /var/lib/redpanda/data, free: 150.644GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 360889439, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:42:59-08:00   DEBUG 2022-11-21 08:42:59,929 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 1 health report: {id: 1, disks: {{path: /var/lib/redpanda/data, free: 150.644GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 360899467, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:42:59-08:00   DEBUG 2022-11-21 08:42:59,929 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 0: {id: 0, disks: {{path: /var/lib/redpanda/data, free: 173.221GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19195077}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 122390064, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:42:59-08:00   DEBUG 2022-11-21 08:42:59,929 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 0 health report: {id: 0, disks: {{path: /var/lib/redpanda/data, free: 173.221GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19195967}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 122400093, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:42:59-08:00   DEBUG 2022-11-21 08:42:59,928 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:42:59-08:00   DEBUG 2022-11-21 08:42:59,903 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:42:59-08:00   DEBUG 2022-11-21 08:42:59,903 [shard 0] cluster - health_monitor_backend.cc:620 - collecting health report with filter: {include_partitions: true, ntp_filters: {}}

2022-11-21T00:42:59-08:00   DEBUG 2022-11-21 08:42:59,903 [shard 0] cluster - health_monitor_backend.cc:567 - collecting cluster health statistics

2022-11-21T00:42:59-08:00   DEBUG 2022-11-21 08:42:59,903 [shard 0] cluster - health_monitor_backend.cc:312 - refreshing health cache, leader id: {0} self: 0

2022-11-21T00:42:59-08:00   DEBUG 2022-11-21 08:42:59,902 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:42:58-08:00   DEBUG 2022-11-21 08:42:58,671 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:42:56-08:00   INFO  2022-11-21 08:42:56,736 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}

2022-11-21T00:42:56-08:00   INFO  2022-11-21 08:42:56,736 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false

2022-11-21T00:42:55-08:00   DEBUG 2022-11-21 08:42:55,669 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:42:52-08:00   DEBUG 2022-11-21 08:42:52,668 [shard 0] cluster - health_monitor_backend.cc:421 - requesting cluster state report with filter: {per_node_filter: {include_partitions: true, ntp_filters: {}}, nodes: {}}, force refresh: false

2022-11-21T00:42:51-08:00   INFO  2022-11-21 08:42:51,735 [shard 0] cluster - members_backend.cc:559 - [update: {node_id: 2, type: recommissioned}] calculated reallocations: {}

2022-11-21T00:42:51-08:00   INFO  2022-11-21 08:42:51,735 [shard 0] cluster - members_backend.cc:549 - [update: {node_id: 2, type: recommissioned}] reconciliation loop - pending reallocation count: 0, finished: false

2022-11-21T00:42:49-08:00   TRACE 2022-11-21 08:42:49,902 [shard 0] cluster - health_monitor_backend.cc:304 - skipping metadata refresh request current metadata age: 0 ms

2022-11-21T00:42:49-08:00   TRACE 2022-11-21 08:42:49,901 [shard 0] cluster - health_monitor_backend.cc:304 - skipping metadata refresh request current metadata age: 0 ms

2022-11-21T00:42:49-08:00   DEBUG 2022-11-21 08:42:49,901 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 3: {id: 3, disks: {{path: /var/lib/redpanda/data, free: 283.466GiB, total: 319.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19194187}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 161312932, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:42:49-08:00   DEBUG 2022-11-21 08:42:49,901 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 3 health report: {id: 3, disks: {{path: /var/lib/redpanda/data, free: 283.466GiB, total: 319.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19195077}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000022-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12929, size_bytes: 4260}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331245}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000015-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12928, size_bytes: 4643}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 161322960, logical_version: 5, drain_status: {nullopt}}

2022-11-21T00:42:49-08:00   DEBUG 2022-11-21 08:42:49,901 [shard 0] cluster - health_monitor_backend.cc:599 - (previous node report from 2: {id: 2, disks: {{path: /var/lib/redpanda/data, free: 145.924GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331247}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19194187}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 319169503, logical_version: 5, drain_status: {nullopt}})

2022-11-21T00:42:49-08:00   DEBUG 2022-11-21 08:42:49,901 [shard 0] cluster - health_monitor_backend.cc:590 - collected node 2 health report: {id: 2, disks: {{path: /var/lib/redpanda/data, free: 145.924GiB, total: 199.489GiB}}, topics: {{topic: {ns: {kafka}, topic: {catablog-xtdb-txlog}}, leaders: {{id: 0, term: 10047, leader_id: {3}, revision_id: 8465, size_bytes: 6670708}}}, {topic: {ns: {kafka}, topic: {user-harvested-view}}, leaders: {{id: 0, term: 3152, leader_id: {3}, revision_id: 8468, size_bytes: 135184}}}, {topic: {ns: {redpanda}, topic: {controller}}, leaders: {{id: 0, term: 11269, leader_id: {0}, revision_id: 0, size_bytes: 7213477}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000017-changelog}}, leaders: {{id: 0, term: 7468, leader_id: {0}, revision_id: 8466, size_bytes: 3331247}}}, {topic: {ns: {kafka}, topic: {tag}}, leaders: {{id: 0, term: 3427, leader_id: {0}, revision_id: 8469, size_bytes: 143751}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000007-changelog}}, leaders: {{id: 0, term: 4039, leader_id: {2}, revision_id: 8475, size_bytes: 1830653}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000012-changelog}}, leaders: {{id: 0, term: 5544, leader_id: {0}, revision_id: 8476, size_bytes: 2449819}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-REDUCE-STATE-STORE-0000000019-changelog}}, leaders: {{id: 0, term: 6653, leader_id: {0}, revision_id: 8472, size_bytes: 3003968}}}, {topic: {ns: {kafka}, topic: {user-session}}, leaders: {{id: 0, term: 4126, leader_id: {3}, revision_id: 8474, size_bytes: 183609}}}, {topic: {ns: {kafka}, topic: {user-visited-view}}, leaders: {{id: 0, term: 8519, leader_id: {3}, revision_id: 8477, size_bytes: 343541}}}, {topic: {ns: {kafka_internal}, topic: {id_allocator}}, leaders: {{id: 0, term: 5044, leader_id: {2}, revision_id: 4762, size_bytes: 2168451}}}, {topic: {ns: {kafka}, topic: {seed-updated}}, leaders: {{id: 0, term: 3798, leader_id: {0}, revision_id: 8473, size_bytes: 3985314}}}, {topic: {ns: {kafka}, topic: {user-clicked-link}}, leaders: {{id: 0, term: 3296, leader_id: {3}, revision_id: 8470, size_bytes: 264404}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-changelog}}, leaders: {{id: 0, term: 3131, leader_id: {0}, revision_id: 8478, size_bytes: 33548}}}, {topic: {ns: {kafka}, topic: {events-KSTREAM-AGGREGATE-STATE-STORE-0000000004-repartition}}, leaders: {{id: 0, term: 6, leader_id: {1}, revision_id: 12926, size_bytes: 123670}}}, {topic: {ns: {kafka}, topic: {trivia-submit}}, leaders: {{id: 0, term: 88, leader_id: {1}, revision_id: 11715, size_bytes: 219219}}}, {topic: {ns: {kafka}, topic: {events-KTABLE-SUPPRESS-STATE-STORE-0000000010-changelog}}, leaders: {{id: 0, term: 13, leader_id: {1}, revision_id: 12930, size_bytes: 18978}}}, {topic: {ns: {kafka}, topic: {__consumer_offsets}}, leaders: {{id: 2, term: 16302, leader_id: {2}, revision_id: 8467, size_bytes: 7360178}, {id: 1, term: 7999, leader_id: {2}, revision_id: 8479, size_bytes: 3901995}, {id: 0, term: 13143, leader_id: {2}, revision_id: 8464, size_bytes: 19195077}}}, {topic: {ns: {kafka}, topic: {user-consumed-item}}, leaders: {{id: 0, term: 3791, leader_id: {3}, revision_id: 8471, size_bytes: 182364}}}}, redpanda_version: {v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001}, uptime: 319179547, logical_version: 5, drain_status: {nullopt}}
nivekuil commented 1 year ago

broker api:

[{"node_id": 0, "num_cores": 1, "membership_status": "active", "is_alive": true, "disk_space": [{"path": "/var/lib/redpanda/data", "free": 185875738624, "total": 214199939072}], "version": "v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001", "maintenance_status": {"draining": false, "finished": false, "errors": false, "partitions": 0, "eligible": 0, "transferring": 0, "failed": 0}},{"node_id": 1, "num_cores": 1, "membership_status": "active", "is_alive": true, "disk_space": [{"path": "/var/lib/redpanda/data", "free": 161685327872, "total": 214199939072}], "version": "v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001", "maintenance_status": {"draining": false, "finished": false, "errors": false, "partitions": 0, "eligible": 0, "transferring": 0, "failed": 0}},{"node_id": 2, "num_cores": 1, "membership_status": "active", "is_alive": true, "disk_space": [{"path": "/var/lib/redpanda/data", "free": 156810334208, "total": 214199939072}], "version": "v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001", "maintenance_status": {"draining": false, "finished": false, "errors": false, "partitions": 0, "eligible": 0, "transferring": 0, "failed": 0}},{"node_id": 3, "num_cores": 1, "membership_status": "draining", "is_alive": true, "disk_space": [{"path": "/var/lib/redpanda/data", "free": 304502644736, "total": 343048957952}], "version": "v22.2.7 - 96ed1cce3b4d0c85fa863038b3501113cc1e5001", "maintenance_status": {"draining": false, "finished": false, "errors": false, "partitions": 0, "eligible": 0, "transferring": 0, "failed": 0}}]
freef4ll commented 1 year ago

Also ran into this on v23.1.3 (rev 67f1cfc), while adding a new node which wasn't successful. Similar to https://github.com/redpanda-data/redpanda/issues/3668 except firs registration:

Apr 13 11:03:30 ip-172-31-27-245 rpk[1148]: INFO  2023-04-13 11:03:30,695 [shard 0] cluster - health_monitor_backend.cc:288 - unable to refresh health metadata, no leader controller
Apr 13 11:03:30 ip-172-31-27-245 rpk[1148]: INFO  2023-04-13 11:03:30,695 [shard 0] cluster - error refreshing cluster health state - Currently there is no leader controller elected in the cluster
Apr 13 11:03:30 ip-172-31-27-245 rpk[1148]: INFO  2023-04-13 11:03:30,695 [shard 0] cluster - metadata_dissemination_service.cc:357 - unable to retrieve cluster health report - Currently there is no leader controller elected in the cluster
Apr 13 11:03:32 ip-172-31-27-245 rpk[1148]: INFO  2023-04-13 11:03:32,770 [shard 0] cluster - members_manager.cc:470 - Sending join request to {host: 192.168.0.5, port: 33145}
Apr 13 11:03:32 ip-172-31-27-245 rpk[1148]: WARN  2023-04-13 11:03:32,874 [shard 0] cluster - members_manager.cc:604 - Error joining cluster using {host: 192.168.0.5, port: 33145} seed server - not allowed to join
Apr 13 11:03:32 ip-172-31-27-245 rpk[1148]: INFO  2023-04-13 11:03:32,874 [shard 0] cluster - members_manager.cc:470 - Sending join request to {host: 192.168.0.6, port: 33145}
Apr 13 11:03:32 ip-172-31-27-245 rpk[1148]: WARN  2023-04-13 11:03:32,978 [shard 0] cluster - members_manager.cc:604 - Error joining cluster using {host: 192.168.0.6, port: 33145} seed server - not allowed to join
Apr 13 11:03:32 ip-172-31-27-245 rpk[1148]: INFO  2023-04-13 11:03:32,978 [shard 0] cluster - members_manager.cc:470 - Sending join request to {host: 192.168.0.7, port: 33145}
Apr 13 11:03:33 ip-172-31-27-245 rpk[1148]: WARN  2023-04-13 11:03:33,082 [shard 0] cluster - members_manager.cc:604 - Error joining cluster using {host: 192.168.0.7, port: 33145} seed server - not allowed to join
Apr 13 11:03:33 ip-172-31-27-245 rpk[1148]: INFO  2023-04-13 11:03:33,082 [shard 0] cluster - members_manager.cc:470 - Sending join request to {host: 10.142.65.197, port: 33145}
Apr 13 11:03:33 ip-172-31-27-245 rpk[1148]: WARN  2023-04-13 11:03:33,180 [shard 0] cluster - members_manager.cc:604 - Error joining cluster using {host: 10.142.65.197, port: 33145} seed server - not allowed to join

Afterwards have a cluster ID 6 which is allocated partially but not throughout and can't force remove it:

Apr 13 11:46:39 mini2 rpk[2952531]: INFO  2023-04-13 11:46:39,355 [shard 0] cluster - members_backend.cc:837 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] decommissioning finished, removing node from cluster
Apr 13 11:46:39 mini2 rpk[2952531]: WARN  2023-04-13 11:46:39,712 [shard 0] cluster - metadata_dissemination_service.cc:486 - Error sending metadata update rpc::errc::exponential_backoff to 6
Apr 13 11:46:42 mini2 rpk[2952531]: WARN  2023-04-13 11:46:42,713 [shard 0] cluster - metadata_dissemination_service.cc:486 - Error sending metadata update rpc::errc::exponential_backoff to 6
Apr 13 11:46:44 mini2 rpk[2952531]: INFO  2023-04-13 11:46:44,363 [shard 0] cluster - members_backend.cc:753 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] reconciliation loop - pending reallocation count: 0, finished: false
Apr 13 11:46:44 mini2 rpk[2952531]: INFO  2023-04-13 11:46:44,363 [shard 0] cluster - members_backend.cc:768 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] calculated reallocations: {}
Apr 13 11:46:44 mini2 rpk[2952531]: INFO  2023-04-13 11:46:44,363 [shard 0] cluster - members_backend.cc:837 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] decommissioning finished, removing node from cluster
Apr 13 11:46:45 mini2 rpk[2952531]: WARN  2023-04-13 11:46:45,714 [shard 0] cluster - metadata_dissemination_service.cc:486 - Error sending metadata update rpc::errc::exponential_backoff to 6
Apr 13 11:46:48 mini2 rpk[2952531]: WARN  2023-04-13 11:46:48,715 [shard 0] cluster - metadata_dissemination_service.cc:486 - Error sending metadata update rpc::errc::exponential_backoff to 6
Apr 13 11:46:49 mini2 rpk[2952531]: INFO  2023-04-13 11:46:49,373 [shard 0] cluster - members_backend.cc:753 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] reconciliation loop - pending reallocation count: 0, finished: false
Apr 13 11:46:49 mini2 rpk[2952531]: INFO  2023-04-13 11:46:49,373 [shard 0] cluster - members_backend.cc:768 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] calculated reallocations: {}
Apr 13 11:46:49 mini2 rpk[2952531]: INFO  2023-04-13 11:46:49,373 [shard 0] cluster - members_backend.cc:837 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] decommissioning finished, removing node from cluster
Apr 13 11:46:51 mini2 rpk[2952531]: WARN  2023-04-13 11:46:51,716 [shard 0] cluster - metadata_dissemination_service.cc:486 - Error sending metadata update rpc::errc::exponential_backoff to 6
Apr 13 11:46:54 mini2 rpk[2952531]: INFO  2023-04-13 11:46:54,379 [shard 0] cluster - members_backend.cc:753 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] reconciliation loop - pending reallocation count: 0, finished: false
Apr 13 11:46:54 mini2 rpk[2952531]: INFO  2023-04-13 11:46:54,379 [shard 0] cluster - members_backend.cc:768 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] calculated reallocations: {}
Apr 13 11:46:54 mini2 rpk[2952531]: INFO  2023-04-13 11:46:54,379 [shard 0] cluster - members_backend.cc:837 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] decommissioning finished, removing node from cluster
Apr 13 11:46:57 mini2 rpk[2952531]: WARN  2023-04-13 11:46:57,717 [shard 0] cluster - metadata_dissemination_service.cc:486 - Error sending metadata update rpc::errc::disconnected_endpoint(node down) to 6
Apr 13 11:46:59 mini2 rpk[2952531]: INFO  2023-04-13 11:46:59,385 [shard 0] cluster - members_backend.cc:753 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] reconciliation loop - pending reallocation count: 0, finished: false
Apr 13 11:46:59 mini2 rpk[2952531]: INFO  2023-04-13 11:46:59,385 [shard 0] cluster - members_backend.cc:768 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] calculated reallocations: {}
Apr 13 11:46:59 mini2 rpk[2952531]: INFO  2023-04-13 11:46:59,385 [shard 0] cluster - members_backend.cc:837 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] decommissioning finished, removing node from cluster
Apr 13 11:47:00 mini2 rpk[2952531]: WARN  2023-04-13 11:47:00,718 [shard 0] cluster - metadata_dissemination_service.cc:486 - Error sending metadata update rpc::errc::exponential_backoff to 6
Apr 13 11:47:03 mini2 rpk[2952531]: WARN  2023-04-13 11:47:03,719 [shard 0] cluster - metadata_dissemination_service.cc:486 - Error sending metadata update rpc::errc::exponential_backoff to 6
Apr 13 11:47:04 mini2 rpk[2952531]: INFO  2023-04-13 11:47:04,392 [shard 0] cluster - members_backend.cc:753 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] reconciliation loop - pending reallocation count: 0, finished: false
Apr 13 11:47:04 mini2 rpk[2952531]: INFO  2023-04-13 11:47:04,392 [shard 0] cluster - members_backend.cc:768 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] calculated reallocations: {}
Apr 13 11:47:04 mini2 rpk[2952531]: INFO  2023-04-13 11:47:04,392 [shard 0] cluster - members_backend.cc:837 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] decommissioning finished, removing node from cluster
Apr 13 11:47:06 mini2 rpk[2952531]: WARN  2023-04-13 11:47:06,719 [shard 0] cluster - metadata_dissemination_service.cc:486 - Error sending metadata update rpc::errc::exponential_backoff to 6
Apr 13 11:47:09 mini2 rpk[2952531]: INFO  2023-04-13 11:47:09,398 [shard 0] cluster - members_backend.cc:753 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] reconciliation loop - pending reallocation count: 0, finished: false
Apr 13 11:47:09 mini2 rpk[2952531]: INFO  2023-04-13 11:47:09,398 [shard 0] cluster - members_backend.cc:768 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] calculated reallocations: {}
Apr 13 11:47:09 mini2 rpk[2952531]: INFO  2023-04-13 11:47:09,398 [shard 0] cluster - members_backend.cc:837 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] decommissioning finished, removing node from cluster
Apr 13 11:47:09 mini2 rpk[2952531]: WARN  2023-04-13 11:47:09,720 [shard 0] cluster - metadata_dissemination_service.cc:486 - Error sending metadata update rpc::errc::exponential_backoff to 6
Apr 13 11:47:12 mini2 rpk[2952531]: WARN  2023-04-13 11:47:12,721 [shard 0] cluster - metadata_dissemination_service.cc:486 - Error sending metadata update rpc::errc::exponential_backoff to 6
Apr 13 11:47:14 mini2 rpk[2952531]: INFO  2023-04-13 11:47:14,404 [shard 0] cluster - members_backend.cc:753 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] reconciliation loop - pending reallocation count: 0, finished: false
Apr 13 11:47:14 mini2 rpk[2952531]: INFO  2023-04-13 11:47:14,404 [shard 0] cluster - members_backend.cc:768 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] calculated reallocations: {}
Apr 13 11:47:14 mini2 rpk[2952531]: INFO  2023-04-13 11:47:14,404 [shard 0] cluster - members_backend.cc:837 - [update: {{node_id: 6, type: decommissioned, offset: 1461}}] decommissioning finished, removing node from cluster
Apr 13 11:47:15 mini2 rpk[2952531]: WARN  2023-04-13 11:47:15,722 [shard 0] cluster - metadata_dissemination_service.cc:486 - Error sending metadata update rpc::errc::exponential_backoff to 6
rpk cluster health
CLUSTER HEALTH OVERVIEW
=======================
Healthy:               false
Controller ID:               1
All nodes:                   [5 4 6 1 0 3 2]
Nodes down:                  [6]
Leaderless partitions:       []
Under-replicated partitions: [redpanda/controller/0]
ngalanis930 commented 1 month ago

Experiencing weird behavior too in version 24.1.16. The broker is not alive and --force is not working whilst using the decommission command and the node status is stuck in draining.

unable to decommission broker: request PUT https://redpanda-0.redpanda.redpanda.svc.cluster.local.:9644/v1/brokers/2/decommission failed: Bad Request, body: "{\"message\": \"can not update broker 2 state, invalid state transition requested\", \"code\": 400}"

NODE-ID  NUM-CORES  MEMBERSHIP-STATUS  IS-ALIVE  BROKER-VERSION
0        2          active             true      v24.1.16 - 5aa52e4c069c5f66cea24c425aeb31fa70b71bda
1        2          active             true      v24.1.16 - 5aa52e4c069c5f66cea24c425aeb31fa70b71bda
2        2          draining           false
3        2          active             true      v24.1.16 - 5aa52e4c069c5f66cea24c425aeb31fa70b71bda
piyushredpanda commented 1 month ago

@ngalanis930 few things to check would be if the cluster has sufficient space for hosting the replicas that were erstwhile on the node in you are trying to decommission; were there any RF=1 topics/partitions on that node? Also, is that node alive/reachable with disk or is it gone?

Please also upgrade to v24.2.x, there's a lot of goodness in our latest major, including a handful of related changes that should increase robustness and stability of the system.

ngalanis930 commented 1 month ago

Hmm, it seems the topic with replica 1 was causing my issue. Conducted further testing and had no further issues. Thanks for your reply.