This PR is to add new action: partitions-status for topicctl get
Expectations:
topicctl get partitions-status (optional --topics flag)
topicctl get partitions-status --status <offline|under-replicated|preferred-leader|not-preferred-leader> (optional --topics flag)
topicctl get partitions-status --status <offline|under-replicated|preferred-leader|not-preferred-leader> --full (optional --topics flag)
partitions-status - Under replicated partitions:
Kafka cluster is in under replicated state if the number of ISR are less than the Replicas available for the partition
partitions-status - Offline partitions:
Kafka cluster is in offline state if partition leader broker ID is not found. (i.e) kafka-go metadata call with partitions having ListenerNotFound Error observed for leader partition
Gotcha: what if offline partition has valid ISRs? Not sure how to replicate this! No matter what, Partition is offline is there is no Leader broker ID
partitions-status - Preferred Leader partitions:
if the leader partition broker id is equal to first available Replicas broker id
partitions-status - Not Preferred Leader partitions:
if the leader partition broker id is not equal to first available Replicas broker id
NOTE:
We leverage kafka-go metadata for all the above partitions statuses
Unlike sarama which gives the offline leader broker id as -1, kafka-go gives the offline broker id as 0 (also Port is 0 with Partition Error information etc)
To make readability better for users, we will modify broker ID from:
any ISR broker ID that does not have valid Host or Port from 0 to -1
any Replica broker ID that does not have valid Host or Port from 0 to -1
(Offline) Leader Broker ID that does not have a valid Host or Port from 0 to -1
In general, partition error precedence is - Offline > UnderReplicated > NotPreferredLeader. BUT offline partition triumphs everything
Local Set UP Details:
Set up type: Docker
Number of Kafka Nodes: 3 (one in each rack). Broker IDs - 0,1,2 running on ports 19092, 29092, 39092. Kafka containers are kafka-1 (broker ID-0), kafka-2(broker ID-1), kafka-3(broker ID-2)
Number of zookeeper Nodes: 1, running on port 12181
A Mock Producer produces to topic: test-1
A Mock Consumer reads from topic: test-1
topic: __consumer_offsets - partitions: 50, replication: 3, strategy - any
topic: test-1 - partitions 1, replication 3, strategy - any
topic: threepartition - partitions 3, replication 2, strategy - Fixed placement. We will have partitions 0,1 spread in brokers 1,2, partition 2 in brokers 0,1
Build
Build to run locally - go build -o ./build/topicctl -a ./cmd/topicctl
Build to run inside container GOOS=linux GOARCH=amd64 go build -o ./build/topicctl-linux-amd64 -a ./cmd/topicctl
(Used later) Copy the amd64 build to zookeeper docker
# ./build/topicctl get partitions-status --broker-addr localhost:29092
Get all partitions-status via kafka broker url --full
NOTE: --full flag is not applicable to partitions-status
Run the same command with --zk-addr
NOTE: since this is local set up, we will build an run this inside the zookeeper. Because zookeeper stores the kafka Broker name information and network can get tricky with Docker from local machine
# docker exec -u root -it zookeeper /tmp/topicctl-linux-amd64 get partitions-status --zk-addr localhost:2181
Partitions Information (created a topic threepartition)
topic: threepartition information
If you observe, partitions 0,1 replicas are in brokers 1, 2
Description
This PR is to add new action:
partitions-status
for topicctl getExpectations:
partitions-status - Under replicated partitions:
Kafka cluster is in under replicated state if the number of ISR are less than the Replicas available for the partition
partitions-status - Offline partitions:
Kafka cluster is in offline state if partition leader broker ID is not found. (i.e) kafka-go metadata call with partitions having ListenerNotFound Error observed for leader partition
Gotcha: what if offline partition has valid ISRs? Not sure how to replicate this! No matter what, Partition is offline is there is no Leader broker ID
partitions-status - Preferred Leader partitions:
if the leader partition broker id is equal to first available Replicas broker id
partitions-status - Not Preferred Leader partitions:
if the leader partition broker id is not equal to first available Replicas broker id
NOTE:
Local Set UP Details:
topic: __consumer_offsets
- partitions: 50, replication: 3, strategy - anytopic: test-1
- partitions 1, replication 3, strategy - anytopic: threepartition
- partitions 3, replication 2, strategy - Fixed placement. We will have partitions 0,1 spread in brokers 1,2, partition 2 in brokers 0,1Build
go build -o ./build/topicctl -a ./cmd/topicctl
GOOS=linux GOARCH=amd64 go build -o ./build/topicctl-linux-amd64 -a ./cmd/topicctl
(Used later) Copy the amd64 build to zookeeper docker
docker cp ./build/topicctl-linux-amd64 zookeeper:/tmp
Run help
#./build/topicctl get partitions-status --help
Get all partitions-status via kafka broker url
# ./build/topicctl get partitions-status --broker-addr localhost:29092
Get all partitions-status via kafka broker url --full
NOTE: --full flag is not applicable to partitions-status
Run the same command with
--zk-addr
NOTE: since this is local set up, we will build an run this inside the zookeeper. Because zookeeper stores the kafka Broker name information and network can get tricky with Docker from local machine
# docker exec -u root -it zookeeper /tmp/topicctl-linux-amd64 get partitions-status --zk-addr localhost:2181
Partitions Information (created a topic threepartition)
topic: threepartition
informationIf you observe, partitions 0,1 replicas are in brokers 1, 2
Get all partitions-status via --zk-addr
# docker exec -u root -it zookeeper /tmp/topicctl-linux-amd64 get partitions-status --zk-addr localhost:2181
Get preferred-leader partitions only
# docker exec -u root -it zookeeper /tmp/topicctl-linux-amd64 get partitions-status --zk-addr localhost:2181 --status preferred-leader
Get preferred-leader partitions --full mode
# ./build/topicctl get partitions-status --broker-addr localhost:39092 --status preferred-leader --full
Get preferred-leader partitions --topics test-1,threepartition
# ./build/topicctl get partitions-status --broker-addr localhost:39092 --status preferred-leader --full --topics test-1,threepartition
# ./build/topicctl get partitions-status --broker-addr localhost:39092 --status preferred-leader --topics test-1,threepartition
Get Under replicated partitions
# docker exec -u root -it zookeeper /tmp/topicctl-linux-amd64 get partitions-status --zk-addr localhost:2181 --status under-replicated
# docker exec -u root -it zookeeper /tmp/topicctl-linux-amd64 get partitions-status --zk-addr localhost:2181 --status under-replicated --full
Kill Broker IDs 1, 2
# docker stop kafka-2; docker stop kafka-3
Getting all partition status (via --zk-addr)
# docker exec -u root -it zookeeper /tmp/topicctl-linux-amd64 get partitions-status --zk-addr localhost:2181
Get all under-replicated partitions (via broker-addr)
# ./build/topicctl get partitions-status --broker-addr localhost:19092 --status preferred-leader
# ./build/topicctl get partitions-status --broker-addr localhost:19092 --status preferred-leader --full
Get all offline partitions (via --zk-addr)
# docker exec -u root -it zookeeper /tmp/topicctl-linux-amd64 get partitions-status --zk-addr localhost:2181 --status offline
# docker exec -u root -it zookeeper /tmp/topicctl-linux-amd64 get partitions-status --zk-addr localhost:2181 --status offline --full
Get all preferred leader (via kafka broker-addr) filter for topics __consumer_offsets
# ./build/topicctl get partitions-status --broker-addr localhost:19092 --status preferred-leader --topics __consumer_offsets
# ./build/topicctl get partitions-status --broker-addr localhost:19092 --status preferred-leader --topics __consumer_offsets --full
Start Broker IDs 1, 2
# docker start kafka-2; docker start kafka-3
Get all partition Status
# ./build/topicctl get partitions-status --broker-addr localhost:29092
Get Not Preferred Leader partitions
# ./build/topicctl get partitions-status --broker-addr localhost:29092 --status not-preferred-leader
# ./build/topicctl get partitions-status --broker-addr localhost:29092 --status not-preferred-leader --full