wvanbergen / kafka

Load-balancing, resuming Kafka consumer for go, backed by Zookeeper.
MIT License
373 stars 143 forks source link

consumergroup not work in docker #117

Open weisd opened 7 years ago

weisd commented 7 years ago

I run Kafka in Docker use images: wurstmeister/kafka:0.9.0.1 docker-compose.yml looks like:

### zookeeper    #########################################
    zookeeper:
      image: wurstmeister/zookeeper
      ports:
        - "2181:2181"
      networks:
        - backend

### kafka    #########################################
    kafka:
      image: wurstmeister/kafka:0.9.0.1
      ports:
        - "9092:9092"
      environment:
        KAFKA_ADVERTISED_PORT: 9092
        KAFKA_ADVERTISED_HOST_NAME: "172.23.0.8"
        KAFKA_CREATE_TOPICS: "KtRoomMessage:1:1,KtRoomMergeMessage:1:1,KtRoomDelMessage:1:1,KtRoomEditMessage:1:1,KtMessageFeed:1:1,KafkaPushsTopic:1:1"
        KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
        KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      # volumes:
        # - ../database/kafka:/tmp/kafka-logs
      #   - ../database/zookeeper:/tmp/zookeeper
      # volumes:
      #   - /var/run/docker.sock:/var/run/docker.sock
      depends_on:
        - zookeeper
      networks:
        backend:
          ipv4_address: 172.23.0.8
### Networks Setup ############################################

networks:
  frontend:
    driver: "bridge"
  backend:
    driver: "bridge"
    ipam:
      config:
      - subnet: 172.23.0.0/24

it works fine when use kafka-console-consumer.sh

➜  docker git:(master) ✗ docker-compose exec kafka bash
bash-4.3# cd /opt/kafka_2.11-0.9.0.1/
bash-4.3# ./bin/kafka-console-consumer.sh --zookeeper zookeeper:2181 --topic KtRoomMessage
{"id":1395866053756864,"room_id":1371649991298042,"uid":1800329207,"username":"","room_type":"vip","message_type":"normal","content":"sdf","image_urls":"","date":"2017-04-11","status":1,"ip":"172.23.0.14","created_at":"2017-04-11 16:30:30","updated_at":"2017-04-11 16:30:30","subRoomId":1371649991298042}

and works fine use github.com/wvanbergen/kafka/tools/consoleconsumer

root@101fe7f7fb4d:/go/src/github.com/wvanbergen/kafka/tools/consoleconsumer# ./consoleconsumer -brokers kafka:9092 -topic KtRoomMessage
Offset: 0
Key:
Value:  {"id":1395864139704472,"room_id":1371649991298042,"uid":1800329207,"username":"","room_type":"vip","message_type":"normal","content":"sdf","image_urls":"","date":"2017-04-11","status":1,"ip":"172.23.0.14","created_at":"2017-04-11 15:58:36","updated_at":"2017-04-11 15:58:36","subRoomId":1371649991298042}

but not work use consumergroup

root@101fe7f7fb4d:/go/src/github.com/wvanbergen/kafka/tools/consoleconsumer# consumergroup -zookeeper zookeeper:2181 -topics KtRoomMessage -gro dada
2017/04/11 16:33:09 Connected to 172.23.0.4:2181
2017/04/11 16:33:09 Authenticated: id=97772968327577636, timeout=4000
2017/04/11 16:33:09 Re-submitting `0` credentials after reconnect
[Sarama] 2017/04/11 16:33:09 Initializing new client
[Sarama] 2017/04/11 16:33:09 client/metadata fetching metadata for all topics from broker 172.23.0.8:9092
[Sarama] 2017/04/11 16:33:09 Connected to broker at 172.23.0.8:9092 (unregistered)
[Sarama] 2017/04/11 16:33:09 client/brokers registered new broker #1001 at 172.23.0.8:9092
[Sarama] 2017/04/11 16:33:09 Successfully initialized new client
[Sarama] 2017/04/11 16:33:09 [dada/fcad8b192b4e] Consumer instance registered (101fe7f7fb4d:1ef0f883-7b41-4f63-a970-fcad8b192b4e).
[Sarama] 2017/04/11 16:33:09 [dada/fcad8b192b4e] Currently registered consumers: 1
[Sarama] 2017/04/11 16:33:09 [dada/fcad8b192b4e] KtRoomMessage :: Started topic consumer
[Sarama] 2017/04/11 16:33:09 [dada/fcad8b192b4e] KtRoomMessage :: Claiming 1 of 1 partitions
[Sarama] 2017/04/11 16:33:09 [dada/fcad8b192b4e] KtRoomMessage/0 :: Partition consumer starting at offset 8.
[Sarama] 2017/04/11 16:33:09 Connected to broker at 172.23.0.8:9092 (registered as #1001)
[Sarama] 2017/04/11 16:33:09 consumer/broker/1001 added subscription to KtRoomMessage/0

consumergroup can not receive any message when i push message to Topic KtRoomMessage

wvanbergen commented 7 years ago

[Sarama] 2017/04/11 16:33:09 [dada/fcad8b192b4e] KtRoomMessage/0 :: Partition consumer starting at offset 8.

It's resuming at offset 8. Are there any new messages after that?