search

kmg28801 / kafka-study

1 stars 0 forks source link

Chapter 6. 카프카 운영 가이드 #6

Closed kmg28801 closed 1 year ago

kmg28801 commented 1 year ago

들어가기 전에.. 저는 모든 실습을 컨테이너 환경에서 진행하였습니다.

필수 카프카 명령어

# 카프카 토픽 생성 명령어 
root@18ed8f2bfb8d:/opt/kafka/bin# kafka-topics.sh --zookeeper zookeeper1:2181,zookeeper2:2182,zookeeper3:2183 --topic peter-topic --create --replication-factor 1 --partitions 1
Created topic peter-topic.
# 카프카 토픽 조회 명령어
root@18ed8f2bfb8d:/opt/kafka/bin# kafka-topics.sh --zookeeper zookeeper1:2181,zookeeper2:2182,zookeeper3:2183 --list
peter-topic
# 카프카 토픽 상세 정보 조회 명령어
root@18ed8f2bfb8d:/opt/kafka/bin# kafka-topics.sh --zookeeper zookeeper1:2181,zookeeper2:2182,zookeeper3:2183 --topic peter-topic --describe
Topic: peter-topic  TopicId: twtcaEsNRDCzaIkBvvRsXw PartitionCount: 1   ReplicationFactor: 1    Configs:
    Topic: peter-topic  Partition: 0    Leader: 1   Replicas: 1 Isr: 1
Topic: peter-topic  TopicId: twtcaEsNRDCzaIkBvvRsXw PartitionCount: 1   ReplicationFactor: 1    Configs:
    Topic: peter-topic  Partition: 0    Leader: 1   Replicas: 1 Isr: 1

토픽 설정 변경 (중요)

잘 적용되었는지 확인하는 명령어

root@18ed8f2bfb8d:/opt/kafka/bin# kafka-topics.sh --zookeeper zookeeper1:2181,zookeeper2:2182,zookeeper3:2183 --topic peter-topic --describe Topic: peter-topic TopicId: twtcaEsNRDCzaIkBvvRsXw PartitionCount: 1 ReplicationFactor: 1 Configs: retention.ms=3600000 Topic: peter-topic Partition: 0 Leader: 1 Replicas: 1 Isr: 1


- 만약 추가 옵션으로 설정한 보관주기 1시간 설정을 삭제하고 싶은 경우 `delete-config`를 추가하고 삭제할 옵션인 `retention.ms`를 입력하면 된다.

```cmd
# config 삭제
root@18ed8f2bfb8d:/opt/kafka/bin# kafka-configs.sh --zookeeper zookeeper1:2181,zookeeper2:2182,zookeeper3:2183 --alter --entity-type topics --entity-name peter-topic --delete-config retention.ms
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Completed updating config for entity: topic 'peter-topic'.
# 조회
root@18ed8f2bfb8d:/opt/kafka/bin#  kafka-topics.sh --zookeeper zookeeper1:2181,zookeeper2:2182,zookeeper3:2183 --topic peter-topic --describe
Topic: peter-topic  TopicId: twtcaEsNRDCzaIkBvvRsXw PartitionCount: 1   ReplicationFactor: 1    Configs:
    Topic: peter-topic  Partition: 0    Leader: 1   Replicas: 1 Isr: 1

토픽의 파티션 수 변경

root@18ed8f2bfb8d:/opt/kafka/bin# kafka-topics.sh --zookeeper zookeeper1:2181,zookeeper2:2182,zookeeper3:2183 --alter --topic peter-topic --partitions 2
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!

위 프로세스에 대한 순차적인 실습은 아래를 따라하시면 됩니다.

# 토픽 생성
root@18ed8f2bfb8d:/opt/kafka/bin# kafka-topics.sh --zookeeper zookeeper1:2181,zookeeper2:2182,zookeeper3:2183 --topic peter-test --create --replication-factor 1 --partitions 1
Created topic peter-test.

# 메시지 프로듀싱
root@18ed8f2bfb8d:/opt/kafka/bin# kafka-console-producer.sh --topic peter-test --broker-list kafka1:9092,kafka2:9092,kafka3:9092
>a
>b
>c
>

# 메시지 컨슈밍
 user  ~/Desktop/kafka/kafka  docker-compose exec kafka1 /bin/bash
root@18ed8f2bfb8d:/# cd opt
root@18ed8f2bfb8d:/opt# cd kafka
root@18ed8f2bfb8d:/opt/kafka# cd bin
root@18ed8f2bfb8d:/opt/kafka/bin# kafka-console-consumer.sh --topic peter-test --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092
a
b
c

여기서 삽질한 내용 공유드립니다. 저는 기존에는 집에서만 실습을 진행했는데, 카페에서 실습하다보니 Local PC IP가 바뀌는 현상이 발생했습니다. 이때는 꼭 hosts 파일에서 IP를 수정해줘야, 스프링 애플리케이션에서 카프카 컨테이너에 정상 접근이 됩니다.https://github.com/kmg28801/kafka-study/issues/4#issuecomment-1529604700

Produce message : taejoong
Topic: peter-test, Partition: 0, Offset: 59, Key: 2, Value: taejoong1
Topic: peter-test, Partition: 0, Offset: 60, Key: 2, Value: taejoong3
Topic: peter-test, Partition: 0, Offset: 61, Key: 2, Value: taejoong5
Topic: peter-test, Partition: 0, Offset: 62, Key: 2, Value: taejoong7
Topic: peter-test, Partition: 0, Offset: 63, Key: 2, Value: taejoong9
Topic: peter-test, Partition: 3, Offset: 15, Key: 1, Value: taejoong0
Topic: peter-test, Partition: 3, Offset: 16, Key: 1, Value: taejoong2
Topic: peter-test, Partition: 3, Offset: 17, Key: 1, Value: taejoong4
Topic: peter-test, Partition: 3, Offset: 18, Key: 1, Value: taejoong6
Topic: peter-test, Partition: 3, Offset: 19, Key: 1, Value: taejoong8

토픽의 리플리케이션 팩터 변경


- 위 json 파일 설명
  - 파티션 0번은 replicas [2,3]으로 설정하였는데 맨 처음 숫자가 리더가 존재하는 브로커 ID입니다. 저의 경우 리더는 브로커2, 리플리카는 브로커3으로 설정하였습니다.
  - 여기서 주의할 점으로는, replicas의 첫 번째 숫자를 현재 상태의 토픽 파티션 정보를 확인한 후 각 파티션의 현재 리더 정보와 일치하도록 설정해 파티션의 리더가 변경되지 않게 해야 한다는 점입니다.
  - 리더가 변경되지 않기 때문에 토픽의 리플리케이션 팩터를 변경해도 프로듀 서와 컨슈머에 영향을 주지 않을 수 있습니다.
  - 토픽의 리플리케이션 팩터를 2가 아닌 3으로 설정하기를 원한다면, replicas 설정에 숫자를 하나 더 추가하면 됩니다.

- 토픽의 리플리케이션 팩터를 변경하는 카프카 명령어는 `kafka-reassign-partitions.sh`
- 추가 옵션으로는 `reassignment-json-file` 옵션에 미리 만들어둔 json 형식의 파일 경로와 파일명을 입력 후 `execute`로 실행
```cmd
root@18ed8f2bfb8d:/opt/kafka/bin# kafka-reassign-partitions.sh --zookeeper zookeeper1:2181,zookeeper2:2182,zookeeper3:2183 --reassignment-json-file rf.json --execute
Warning: --zookeeper is deprecated, and will be removed in a future version of Kafka.
Current partition replica assignment

{"version":1,"partitions":[{"topic":"peter-topic","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"peter-topic","partition":1,"replicas":[3],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignments for peter-topic-0,peter-topic-1

# 결과 확인
root@18ed8f2bfb8d:/opt/kafka/bin# kafka-topics.sh --zookeeper zookeeper1:2181,zookeeper2:2182,zookeeper3:2183 --topic peter-topic --describe
Topic: peter-topic  TopicId: 8Bm3oe9VR1Sd9xE-r0pi2Q PartitionCount: 2   ReplicationFactor: 2    Configs:
    Topic: peter-topic  Partition: 0    Leader: 2   Replicas: 2,3   Isr: 2,3
    Topic: peter-topic  Partition: 1    Leader: 3   Replicas: 3,1   Isr: 3,1

컨슈머 그룹 리스트 확인

GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID peter-topic peter-topic 0 5 5 0 consumer-peter-topic-1-570089fb-7039-4363-8005-2b22a11f3ce6 /172.19.0.1 consumer-peter-topic-1 peter-topic peter-topic 1 5 5 0 consumer-peter-topic-1-570089fb-7039-4363-8005-2b22a11f3ce6 /172.19.0.1 consumer-peter-topic-1


- LAG은 현재 토픽의 저장된 메시지와 컨슈머가 가져간 메시지의 차이를 의미
- 아래 그림을 보면 프로듀서가 10개의 메시지를 카프카로 보냈고, 컨슈머가 모든 메시지를 가져갔다면, 컨슈머는 카프카에서 더 이상 가져갈 메시지가 없고 이런 상태를 LAG = 0이라고 한다.
- LAG 숫자가 높다는 것은 해당 토픽 또는 파티션에 컨슈머가 읽어가지 못한 메시지가 많이 있다는 의미
- LAG이 계속 증가하는 상황이라면 컨슈머 처리가 늦어지고 있는 것이기 때문에 컨슈머나 파티션 수를 늘려서 대응을 해야 하며, 특정 파티션에서만 LAG이 증가한다면 해당 파티션에 연결된 컨슈머를 체크해야 한다.
<img width="1335" alt="image" src="https://github.com/kmg28801/kafka-study/assets/53685313/16615ebe-546a-4da7-aa0d-41fcd066e96e">

# 주키퍼 스케일 아웃

```yaml
version: '3.7'
services: 

  zk1:
    container_name: zookeeper1
    image: wurstmeister/zookeeper:latest
    restart: always
    hostname: zk1
    ports:
      - "2181:2181"
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=zk3:2888:3888;2181 server.4=zk4:2888:3888;2181 server.5=zk5:2888:3888;2181
    volumes:
      - "~/zk-cluster/zk1/data:/data"

  zk2:
    container_name: zookeeper2
    image: wurstmeister/zookeeper:latest
    restart: always
    hostname: zk2
    ports:
      - "2182:2181"
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=zk3:2888:3888;2181 server.4=zk4:2888:3888;2181 server.5=zk5:2888:3888;2181
    volumes:
      - "~/zk-cluster/zk2/data:/data"

  zk3:
    container_name: zookeeper3
    image: wurstmeister/zookeeper:latest
    restart: always
    hostname: zk3
    ports:
      - "2183:2181"
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=zk3:2888:3888;2181 server.4=zk4:2888:3888;2181 server.5=zk5:2888:3888;2181
    volumes:
      - "~/zk-cluster/zk3/data:/data"

  zk4:
    container_name: zookeeper4
    image: wurstmeister/zookeeper:latest
    restart: always
    hostname: zk4
    ports:
      - "2184:2181"
    environment:
      ZOO_MY_ID: 4
      ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=zk3:2888:3888;2181 server.4=zk4:2888:3888;2181 server.5=zk5:2888:3888;2181
    volumes:
      - "~/zk-cluster/zk4/data:/data"

  zk5:
    container_name: zookeeper5
    image: wurstmeister/zookeeper:latest
    restart: always
    hostname: zk5
    ports:
      - "2185:2181"
    environment:
      ZOO_MY_ID: 5
      ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=zk3:2888:3888;2181 server.4=zk4:2888:3888;2181 server.5=zk5:2888:3888;2181
    volumes:
      - "~/zk-cluster/zk5/data:/data"   

  kafka1:
    container_name: kafka1
    image: wurstmeister/kafka:latest
    restart: on-failure
    depends_on:
      - zk1
      - zk2
      - zk3
      - zk4
      - zk5
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_HOST_NAME: host.docker.internal
      BOOTSTRAP_SERVERS: host.docker.internal:9092, host.docker.internal:9093, host.docker.internal:9094
      KAFKA_ZOOKEEPER_CONNECT: "zk1:2181,zk2:2182,zk3:2183,zk4:2184,zk5:2185"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2

  kafka2:
    container_name: kafka2
    image: wurstmeister/kafka:latest
    restart: on-failure
    depends_on:
      - zk1
      - zk2
      - zk3
      - zk4
      - zk5
    ports:
      - "9093:9092"
    volumes:
       - /var/run/docker.sock:/var/run/docker.sock
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ADVERTISED_HOST_NAME: host.docker.internal
      BOOTSTRAP_SERVERS: host.docker.internal:9092, host.docker.internal:9093, host.docker.internal:9094
      KAFKA_ZOOKEEPER_CONNECT: "zk1:2181,zk2:2182,zk3:2183,zk4:2184,zk5:2185"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2

  kafka3:
    container_name: kafka3
    image: wurstmeister/kafka:latest
    restart: on-failure
    depends_on:
      - zk1
      - zk2
      - zk3
      - zk4
      - zk5
    ports:
      - "9094:9092"
    volumes:
       - /var/run/docker.sock:/var/run/docker.sock
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ADVERTISED_HOST_NAME: host.docker.internal
      BOOTSTRAP_SERVERS: host.docker.internal:9092, host.docker.internal:9093, host.docker.internal:9094
      KAFKA_ZOOKEEPER_CONNECT: "zk1:2181,zk2:2182,zk3:2183,zk4:2184,zk5:2185"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2

  kafka-ui:
    image: provectuslabs/kafka-ui
    container_name: kafka-ui
    ports:
      - "10000:8080"
    restart: always
    environment:
      - KAFKA_CLUSTERS_0_NAME=local
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=host.docker.internal:9092,host.docker.internal:9093,host.docker.internal:9094
      - KAFKA_CLUSTERS_0_ZOOKEEPER=zk1:2181,zk2:2182,zk3:2183,zk4:2184,zk5:2185

카프카 매니저

kmg28801 commented 1 year ago

카프카 매니저 docker 구축

https://haithai91.medium.com/zookeeper-kafka-with-docker-b1d4b33c0d5e

yaml 수정 필요

version: '3'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181
    volumes:
      - ./pub-technology/zookeeper/data:/data
      - ./pub-technology/zookeeper/datalog:/datalog

  kafka-server-1:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-server-1:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./pub-technology/kafka1/data:/var/lib/kafka/data
  kafka-server-2:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 39092:39092
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-server-2:9092,PLAINTEXT_HOST://localhost:39092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./pub-technology/kafka2/data:/var/lib/kafka/data
  manager:
    image: sheepkiller/kafka-manager
    ports:
      - 9000:9000
    environment:
      ZK_HOSTS: zookeeper:2181
    depends_on:
      - zookeeper