aio-libs / aiokafka

asyncio client for kafka
http://aiokafka.readthedocs.io/
Apache License 2.0
1.17k stars 234 forks source link

[QUESTION] Consumer group doesn't get any partitions by the RoundRobinPartitionAssignor #906

Open MaximMukhametov opened 1 year ago

MaximMukhametov commented 1 year ago

Hi πŸ‘‹πŸ» I have a pytest test with parametrization (7 tests), that produce and consume simple msg with simple logic. It uses kafka cluster with 3 nodes, each test creates topic with new name test_toipic<β„–_test>with 3 partition and 2 replica-factor, and delete this topic after each test. Each test I run 3 consumers with new group test_group<β„–_test> Every time I run this test, 1/7 or 2/7 fails, because there are no partitions for the consumer group: [WARNING] No partition metadata for topic test_topic_4 (roundrobin.py:61)

I use KafkaAdminClient to inspect some details, so I get: topic:

{'error_code': 0,
  'topic': 'test_topic_4',
  'is_internal': False,
  'partitions': [{'error_code': 0,
    'partition': 0,
    'leader': 4,
    'replicas': [4, 2],
    'isr': [4, 2],
    'offline_replicas': []},
   {'error_code': 0,
    'partition': 2,
    'leader': 3,
    'replicas': [3, 4],
    'isr': [3, 4],
    'offline_replicas': []},
   {'error_code': 0,
    'partition': 1,
    'leader': 2,
    'replicas': [2, 3],
    'isr': [2, 3],
    'offline_replicas': []}]},

group:

 GroupInformation(error_code=0, group='test_group4', state='Stable', protocol_type='consumer', protocol='roundrobin', members=[MemberInformation(member_id='aiokafka-0.8.0-f8ed0518-2198-4f40-84e4-35263e3796cd', client_id='aiokafka-0.8.0', client_host='/172.31.0.1', member_metadata=ConsumerProtocolMemberMetadata(version=0, subscription=['test_topic_4'], user_data=b''), member_assignment=ConsumerProtocolMemberAssignment(version=0, assignment=[], user_data=b'')), MemberInformation(member_id='aiokafka-0.8.0-6c0c504d-8a40-4ab8-9d24-8304da934bb0', client_id='aiokafka-0.8.0', client_host='/172.31.0.1', member_metadata=ConsumerProtocolMemberMetadata(version=0, subscription=['test_topic_4'], user_data=b''), member_assignment=ConsumerProtocolMemberAssignment(version=0, assignment=[], user_data=b'')), MemberInformation(member_id='aiokafka-0.8.0-c4f69127-7feb-4a19-b9f3-0270e57745b2', client_id='aiokafka-0.8.0', client_host='/172.31.0.1', member_metadata=ConsumerProtocolMemberMetadata(version=0, subscription=['test_topic_4'], user_data=b''), member_assignment=ConsumerProtocolMemberAssignment(version=0, assignment=[], user_data=b''))], authorized_operations=None),

As you can see, the topic has 3 partitions, by all members of the group have assignment=[]

I also use kafka-ui and can share some details from there:

Screenshot 2023-07-13 at 12 25 05 Screenshot 2023-07-13 at 12 18 04 Screenshot 2023-07-13 at 12 18 48 Screenshot 2023-07-13 at 12 19 31

Do you have any idea why is this happeningπŸ™πŸ»? Let me know If you need any additional information