kafka学习笔记

[huoarter]

注意： kafka一个分区只能分配给一个group中的一个consumer，所以有60个分区，最多可以起60个consumer

• 创建topics :(60个分区，1个副本（主副一共2个）):

  ./kafka-topics.sh --zookeeper localhost:2181  --create --topic eleme-nginx-log --replication-factor 2 --partitions 60

• 增加topic的分区数：

  ./bin/kafka-topics.sh –zookeeper   localhost:2181 –alter –partitions 60 –topic eleme-nginx-log 

• 平衡分区leader，重新选举：适用于节点挂掉恢复之后没有leader在原来的节点 等价于配置项：auto.leader.rebalance.enable

  bin/kafka-preferred-replica-election.sh --zookeeper localhost:2181 

• 平衡分区节点分布：

  
  bin/kafka-reassign-partitions.sh -zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3,4,5" --generate 

[root@xg-ops-elk-kafka-1 kafka]# cat topics-to-move.json {"topics": [{"topic": "java-standard-application-log"}], "version":1 }

- 生成推荐分布，也可以手动编辑，保存在java_balance_topics.json 例如：

{"version":1,"partitions":[{"topic":"java-standard-application-log","partition":30,"replicas":[2]},{"topic":"java-standard-application-log","partition":2,"replicas":[4]},{"topic":"java-standard-application-log","partition":40,"replicas":[2]},....

- 进行重新分配：

bin/kafka-reassign-partitions.sh --zookeeper xg-ops-elk-kafka-1:2181,xg-ops-elk-kafka-2:2181,xg-ops-elk-kafka-3:2181 --reassignment-json-file java_balance_topics.json --execute

- 验证分配进度：

bin/kafka-reassign-partitions.sh --zookeeper xg-ops-elk-kafka-1:2181,xg-ops-elk-kafka-2:2181,xg-ops-elk-kafka-3:2181 --reassignment-json-file java_balance_topics.json --verify

**压测数据：**

sh kafka-producer-perf-test.sh --topic ertao_test --num-records 5000000 --record-size 512 --throughput 1000000 --producer-props bootstrap.servers=xg-ops-elk-ngxkf-1:9092,xg-ops-elk-ngxkf-1:9092 acks=all retries=2 linger.ms=1 sh kafka-producer-perf-test.sh --topic ertao_test --num-records 5000000 --record-size 512 --throughput 100 0000 --producer-props bootstrap.servers=xg-ops-elk-ngxkf-1:9092,xg-ops-elk-ngxkf-1:9092 acks=all retries=2 linger.ms=1

sh kafka-producer-perf-test.sh --topic ertao_test --num-records 5000000 --record-size 512 --throughput 1000000 --producer-props bootstrap.servers=xg-ops-elk-ngxkf-1:9092,xg-ops-elk-ngxkf-1:9092 acks=1 retries=2 linger.ms=1

sh kafka-producer-perf-test.sh --topic ertao_test --num-records 5000000 --record-size 512 --throughput 1000000 --producer-props bootstrap.servers=xg-ops-elk-ngxkf-1:9092,xg-ops-elk-ngxkf-1:9092 acks=0 retries=2 linger.ms=1

./kafka-consumer-perf-test.sh --zookeeper localhost:2181 --messages 50000000 --topic ertao_test --threads 1