search

xiaogui2011 / xiaogui2011.github.io

The startup template for Chirpy.
https://chirpy.cotes.page
MIT License
0 stars 0 forks source link

kafka部署集群 #21

Closed xiaogui2011 closed 1 month ago

xiaogui2011 commented 1 month ago

https://blog.renzicu.com/2023/06/226.html

xiaogui2011 commented 1 month ago

https://blog.csdn.net/qq_41118173/article/details/140183152

xiaogui2011 commented 1 month ago 
er)
[2024-09-06 09:56:00,661] ERROR [RaftManager id=2] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=1875, data=VoteResponseData(errorCode=104, topics=[]), sourceId=3) (org.apache.kafka.raft.KafkaRaftClient)
[2024-09-06 09:56:00,683] ERROR [RaftManager id=2] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=1877, data=VoteResponseData(errorCode=104, topics=[]), sourceId=3) (org.apache.kafka.raft.KafkaRaftClient)
[2024-09-06 09:56:00,705] ERROR [RaftManager id=2] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=1879, data=VoteResponseData(errorCode=104, topics=[]), sourceId=3) (org.apache.kafka.raft.KafkaRaftClient)
[2024-09-06 09:56:00,727] ERROR [RaftManager id=2] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=1881, data=VoteResponseData(errorCode=104, topics=[]), sourceId=3) (org.apache.kafka.raft.KafkaRaftClient)
[2024-09-06 09:56:00,747] INFO [MetadataLoader id=2] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader)
[2024-09-06 09:56:00,748] ERROR [RaftManager id=2] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=1883, data=VoteResponseData(errorCode=104, topics=[]), sourceId=3) (org.apache.kafka.raft.KafkaRaftClient)
[2024-09-06 09:56:00,769] INFO [RaftManager id=2] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
[2024-09-06 09:56:00,770] WARN [RaftManager id=2] Connection to node 1 (/10.130.9.36:9093) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-09-06 09:56:00,770] ERROR [RaftManager id=2] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=1885, data=VoteResponseData(errorCode=104, topics=[]), sourceId=3) (org.apache.kafka.raft.KafkaRaftClient)
[2024-09-06 09:56:00,791] ERROR [RaftManager id=2] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=1887, data=VoteResponseData(errorCode=104, topics=[]), sourceId=3) (org.apache.kafka.raft.KafkaRaftClient)
[2024-09-06 09:56:00,813] ERROR [RaftManager id=2] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=1889, data=VoteResponseData(errorCode=104, topics=[]), sourceId=3) (org.apache.kafka.raft.KafkaRaftClient)
[2024-09-06 09:56:00,834] ERROR [RaftManager id=2] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=1891, data=VoteResponseData(errorCode=104, topics=[]), sourceId=3) (org.apache.kafka.raft.KafkaRaftClient)
[2024-09-06 09:56:00,848] INFO [MetadataLoader id=2] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader)
[2024-09-06 09:56:00,855] ERROR [RaftManager id=2] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=1893, data=VoteResponseData(errorCode=104, topics=[]), sourceId=3) (org.apache.kafka.raft.KafkaRaftClient)
[2024-09-06 09:56:00,876] ERROR [RaftManager id=2] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=1895, data=VoteResponseData(errorCode=104, topics=[]), sourceId=3) (org.apache.kafka.raft.KafkaRaftClient)
[2024-09-06 09:56:00,896] ERROR [RaftManager id=2] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=1897, data=VoteResponseData(errorCode=104, topics=[]), sourceId=3) (org.apache.kafka.raft.KafkaRaftClient)
[2024-09-06 09:56:00,917] ERROR [RaftManager id=2] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=1899, data=VoteResponseData(errorCode=104, topics=[]), sourceId=3) (org.apache.kafka.raft.KafkaRaftClient)
[2024-09-06 09:56:00,938] ERROR [RaftManager id=2] Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response: InboundResponse(correlationId=1901, data=VoteResponseData(errorCode=104, topics=[]), sourceId=3) (org.apache.kafka.raft.KafkaRaftClient)
[2024-09-06 09:56:00,948] INFO [MetadataLoader id=2] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoad
xiaogui2011 commented 1 month ago

https://blog.csdn.net/qq_38263083/article/details/133313443 成功部署kafka

xiaogui2011 commented 1 month ago

kafka-ui

 kafka-ui:
    image: provectuslabs/kafka-ui:master
    container_name: kafka-ui
    ports:
      - "8910:8080"
    restart: always
    environment:
      - KAFKA_CLUSTERS_0_NAME=local
      - DYNAMIC_CONFIG_ENABLED=true
      - AUTH_TYPE=LOGIN_FORM
      - SPRING_SECURITY_USER_NAME=admin
      - SPRING_SECURITY_USER_PASSWORD=admin
    depends_on:
      - kafka
    networks:
      - kafka_net

networks:
  kafka_net:
    driver: bridge
xiaogui2011 commented 1 month ago

为什么相同的消费者租的数量超过了1, 就出现了负载均衡的问题, 无法正常消费

xiaogui2011 commented 1 month ago
  1. 一个消费者组里有多个topic, 有一个消费者组
  2. 一个消费者组里有一个topic, 有多个消费者组

哪种性能高

在你的场景中，有两三个高频率的Topic（每分钟几万条数据），同时还有低频率的Topic（每分钟仅一条数据），这种情况下可以考虑以下因素来优化性能：

  1. 一个消费者组处理多个Topic 优势：对于高频Topic来说，消费者组的消费者可以分配多个分区，并发处理，能有效分摊负载。如果消费者数量足够，并且Kafka的分区策略设置合理，消息消费可以达到高吞吐量。 劣势：低频Topic的数据可能会被高频Topic“淹没”，因为消费者可能主要处理高频Topic的数据，导致低频Topic的消息消费延迟增加。这种情况下，整体的资源利用率不均衡。
  2. 多个消费者组分别处理单个Topic 优势： 高频Topic的消费者组可以独立配置更多的消费者，以确保高并发处理，提升吞吐量。 低频Topic的消费者组则可以单独配置更少的资源，避免资源浪费，并且可以确保低频Topic的数据及时被消费，不会因为高频Topic而受到影响。 劣势：增加了管理多个消费者组的开销，但这对Kafka性能的影响相对较小，尤其是在资源和隔离的需求明确时。 性能对比与建议 基于你的场景，考虑以下优化：

高频Topic与低频Topic分离：

使用多个消费者组处理高频和低频的Topic： 高频Topic：独立一个消费者组，分配足够的消费者，确保并发处理和高吞吐量。Kafka可以根据分区数将分区分配给不同的消费者，最大化处理性能。 低频Topic：独立另一个消费者组，配置少量消费者或单个消费者，这样不会浪费资源，同时能够确保低频消息的消费及时性。 考虑分区策略：对于高频Topic，确保分区数足够多，Kafka会将这些分区分配给消费者组中的消费者并行处理，能够进一步提升吞吐量。

总结 对于你这样的场景，多个消费者组分别处理高频和低频的Topic会更好： 高频Topic使用一个独立的消费者组，确保最大限度并发处理和高吞吐量。 低频Topic单独使用一个消费者组，节约资源，同时保证消费的实时性。

xiaogui2011 commented 1 month ago

无认证的

参考 kafka kraft集群

[root@worker4 kafka]# cat kafka.yaml
version: '3'
services:
  kafka-node2:
    image: bitnami/kafka:3.7.0
    user: root
    container_name: kafka-node1
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker, controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://10.130.9.36:9092
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_NODE_ID=1
      - KAFKA_KRAFT_CLUSTER_ID=manongdashu66666666666
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@10.130.9.36:9093,2@10.130.9.50:9093,3@10.130.9.87:9093
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - /data/kafka/kraft:/bitnami/kafka:rw
    network_mode: host
[root@worker5 kafka]# cat kafka.yaml
version: '3'
services:
  kafka-node2:
    image: bitnami/kafka:3.7.0
    user: root
    container_name: kafka-node2
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker, controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://10.130.9.50:9092
      - KAFKA_BROKER_ID=2
      - KAFKA_CFG_NODE_ID=2
      - KAFKA_KRAFT_CLUSTER_ID=manongdashu66666666666
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@10.130.9.36:9093,2@10.130.9.50:9093,3@10.130.9.87:9093
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - /data/kafka/kraft:/bitnami/kafka:rw
    network_mode: host
[root@worker6 kafka]# cat kafka.yaml
version: '3'
services:
  kafka-node2:
    image: bitnami/kafka:3.7.0
    user: root
    container_name: kafka-node3
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker, controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://10.130.9.87:9092
      - KAFKA_BROKER_ID=3
      - KAFKA_CFG_NODE_ID=3
      - KAFKA_KRAFT_CLUSTER_ID=manongdashu66666666666
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@10.130.9.36:9093,2@10.130.9.50:9093,3@10.130.9.87:9093
      - ALLOW_PLAINTEXT_LISTENER=yes
    volumes:
      - /data/kafka/kraft:/bitnami/kafka:rw
    network_mode: host
xiaogui2011 commented 1 month ago

SASL认证成功

version: '3'
services:
  kafka-node2:
    image: bitnami/kafka:3.7.0
    user: root
    container_name: kafka-node1
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker, controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=CLIENT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,CLIENT:SASL_PLAINTEXT
      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://10.130.9.36:9092
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_NODE_ID=1
      - KAFKA_KRAFT_CLUSTER_ID=manongdashu66666666666
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@10.130.9.36:9093,2@10.130.9.50:9093,3@10.130.9.87:9093
      - ALLOW_PLAINTEXT_LISTENER=no
      - KAFKA_CFG_SASL_ENABLED_MECHANISMS=SCRAM-SHA-512
      - KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=SCRAM-SHA-512
      - KAFKA_CFG_SUPER_USERS=User:test
      - KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND=false
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
      - KAFKA_CLIENT_USERS=test
      - KAFKA_CLIENT_PASSWORDS=123456

    volumes:
      - /data/kafka/kraft:/bitnami/kafka:rw
    network_mode: host
version: '3'
services:
  kafka-node2:
    image: bitnami/kafka:3.7.0
    user: root
    container_name: kafka-node2
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker, controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=CLIENT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,CLIENT:SASL_PLAINTEXT
      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://10.130.9.50:9092
      - KAFKA_BROKER_ID=2
      - KAFKA_CFG_NODE_ID=2
      - KAFKA_KRAFT_CLUSTER_ID=manongdashu66666666666
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@10.130.9.36:9093,2@10.130.9.50:9093,3@10.130.9.87:9093
      - ALLOW_PLAINTEXT_LISTENER=no
      - KAFKA_CFG_SASL_ENABLED_MECHANISMS=SCRAM-SHA-512
      - KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=SCRAM-SHA-512
      - KAFKA_CFG_SUPER_USERS=User:test
      - KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND=false
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
      - KAFKA_CLIENT_USERS=test
      - KAFKA_CLIENT_PASSWORDS=123456
    volumes:
      - /data/kafka/kraft:/bitnami/kafka:rw
    network_mode: host
version: '3'
services:
  kafka-node2:
    image: bitnami/kafka:3.7.0
    user: root
    container_name: kafka-node3
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker, controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=CLIENT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,CLIENT:SASL_PLAINTEXT
      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://10.130.9.87:9092
      - KAFKA_BROKER_ID=3
      - KAFKA_CFG_NODE_ID=3
      - KAFKA_KRAFT_CLUSTER_ID=manongdashu66666666666
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@10.130.9.36:9093,2@10.130.9.50:9093,3@10.130.9.87:9093
      - ALLOW_PLAINTEXT_LISTENER=no
      - KAFKA_CFG_SASL_ENABLED_MECHANISMS=SCRAM-SHA-512
      - KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=SCRAM-SHA-512
      - KAFKA_CFG_SUPER_USERS=User:test
      - KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND=false
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
      - KAFKA_CLIENT_USERS=test
      - KAFKA_CLIENT_PASSWORDS=123456
    volumes:
      - /data/kafka/kraft:/bitnami/kafka:rw
    network_mode: host