"import kafka" fails with "ModuleNotFoundError: No module named 'kafka.vendor.six.moves'" under Python 3.12

[gtzampanakis]

I have verified this on Linux and Windows, even though the output below is from Linux:

user1@CY1-WL-337:~/dev$ python3.12 -m venv venv
user1@CY1-WL-337:~/dev$ source venv/bin/activate
(venv) user1@CY1-WL-337:~/dev$ pip install kafka-python
Collecting kafka-python
  Using cached kafka_python-2.0.2-py2.py3-none-any.whl (246 kB)
Installing collected packages: kafka-python
Successfully installed kafka-python-2.0.2

(venv) user1@CY1-WL-337:~/dev$ python -c 'import kafka'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/__init__.py", line 23, in <module>
    from kafka.consumer import KafkaConsumer
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/__init__.py", line 3, in <module>
    from kafka.consumer.group import KafkaConsumer
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/group.py", line 13, in <module>
    from kafka.consumer.fetcher import Fetcher
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/fetcher.py", line 19, in <module>
    from kafka.record import MemoryRecords
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/__init__.py", line 1, in <module>
    from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/memory_records.py", line 27, in <module>
    from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/legacy_records.py", line 50, in <module>
    from kafka.codec import (
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/codec.py", line 9, in <module>
    from kafka.vendor.six.moves import range
ModuleNotFoundError: No module named 'kafka.vendor.six.moves'

[wbarnha]

This was patched in https://github.com/dpkp/kafka-python/pull/2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime.

Please understand that I am doing everything within my means to get a release out to the public.

[simcax]

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime.

Please understand that I am doing everything within my means to get a release out to the public.

Is there an idea of when the next release will be out?

[wbarnha]

No update, sorry. 😔

[Anitheanalyst]

Hi i am still having this issue. Is it resolved in the latest release.

here is my contect: Python 3.12.2 kafka-python 2.0.2

import procedure:

from kafka import KafkaProducer Error: ModuleNotFoundError Traceback (most recent call last) Cell In[7], line 1 ----> 1 from kafka import KafkaProducer

File ~\AppData\Lib\site-packages\kafka__init__.py:23 21 from kafka.admin import KafkaAdminClient 22 from kafka.client_async import KafkaClient ---> 23 from kafka.consumer import KafkaConsumer 24 from kafka.consumer.subscription_state import ConsumerRebalanceListener 25 from kafka.producer import KafkaProducer

File ~\AppData\Lib\site-packages\kafka\consumer__init.py:3 1 from future__ import absolute_import ----> 3 from kafka.consumer.group import KafkaConsumer 5 all = [ 6 'KafkaConsumer' 7 ]

File ~\AppData\Lib\site-packages\kafka\consumer\group.py:13 10 from kafka.vendor import six 12 from kafka.client_async import KafkaClient, selectors ---> 13 from kafka.consumer.fetcher import Fetcher 14 from kafka.consumer.subscription_state import SubscriptionState 15 from kafka.coordinator.consumer import ConsumerCoordinator

File ~\AppData\Lib\site-packages\kafka\consumer\fetcher.py:19 15 from kafka.protocol.fetch import FetchRequest 16 from kafka.protocol.offset import ( 17 OffsetRequest, OffsetResetStrategy, UNKNOWN_OFFSET 18 ) ---> 19 from kafka.record import MemoryRecords 20 from kafka.serializer import Deserializer 21 from kafka.structs import TopicPartition, OffsetAndTimestamp

File ~\AppData\Lib\site-packages\kafka\record__init__.py:1 ----> 1 from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder 3 all = ["MemoryRecords", "MemoryRecordsBuilder"]

File ~\AppData\Lib\site-packages\kafka\record\memory_records.py:27 25 from kafka.errors import CorruptRecordException 26 from kafka.record.abc import ABCRecords ---> 27 from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder 28 from kafka.record.default_records import DefaultRecordBatch, DefaultRecordBatchBuilder 31 class MemoryRecords(ABCRecords):

File ~\AppData\Lib\site-packages\kafka\record\legacy_records.py:50 47 from kafka.record.abc import ABCRecord, ABCRecordBatch, ABCRecordBatchBuilder 48 from kafka.record.util import calc_crc32 ---> 50 from kafka.codec import ( 51 gzip_encode, snappy_encode, lz4_encode, lz4_encode_old_kafka, 52 gzip_decode, snappy_decode, lz4_decode, lz4_decode_old_kafka, 53 ) 54 import kafka.codec as codecs 55 from kafka.errors import CorruptRecordException, UnsupportedCodecError

File ~\AppData\Lib\site-packages\kafka\codec.py:9 6 import struct 8 from kafka.vendor import six ----> 9 from kafka.vendor.six.moves import range 11 _XERIAL_V1_HEADER = (-126, b'S', b'N', b'A', b'P', b'P', b'Y', 0, 1, 1) 12 _XERIAL_V1_FORMAT = 'bccccccBii'

ModuleNotFoundError: No module named 'kafka.vendor.six.moves'

[VishnuBhaarath]

Hi @wbarnha , The issue still persists and I am facing the same issue, can you pls provide an update when the fix will be rolloed out. Thanks

[wbarnha]

I am aware of the issue. Please refer to https://github.com/dpkp/kafka-python/issues/2412#issuecomment-1806341342.

[VishnuBhaarath]

Thanks for it @wbarnha , I downgraded my python version and got it rectified.

[ankibunkers]

Did we ever merge this fix in for 3.12?

[ankibunkers]

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime.

Please understand that I am doing everything within my means to get a release out to the public.

@wbarnha Thank you William for your support here. Installing python -m pip install --break-system-packages git+https://github.com/dpkp/kafka-python.git still is not resolving the issue.

i can not downgrade python 3.11 due to a HIGH vulnerability with ecdsa pacakge. Any work around solution for this problem at the moment?

[wbarnha]

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime. Please understand that I am doing everything within my means to get a release out to the public.

@wbarnha Thank you William for your support here. Installing python -m pip install --break-system-packages git+https://github.com/dpkp/kafka-python.git still is not resolving the issue.

i can not downgrade python 3.11 due to a HIGH vulnerability with ecdsa pacakge. Any work around solution for this problem at the moment?

The current workaround solution would be to replace your kafka-python installation with my fork at kafka-python-ng for now.

[ssbarnea]

Can we please reopen this bug because a critical bug like this should not be closed until the fix is released. I would even pin it after reopening it, just to remind everyone about it.

[winniegr]

I also have the same issue

[pauliverson]

i meet this problem in python3.12 kafka-python 2.0.2，so how to solve this problem

[wbarnha]

Can we please reopen this bug because a critical bug like this should not be closed until the fix is released. I would even pin it after reopening it, just to remind everyone about it.

It's a fair point, and since there's a demand to reopen this, I'll oblige!

[gwenshap]

This is still not fixed, and have been open for a year now. Does everyone really install from the github repo? Is it so hard to do a release?

Anyway, commenting to show additional demand. Although demand for "be able to use the library" seems self-explanatory...

[csharplus]

Will this fix be released in the near future? I'm using the following work-around in requirements.txt at present:

kafka-python==2.0.2; python_version < '3.12'
kafka-python @ git+https://github.com/dpkp/kafka-python.git ; python_version >= '3.12'

[HiMyNameIsBoxxy]

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime. Please understand that I am doing everything within my means to get a release out to the public.

@wbarnha Thank you William for your support here. Installing python -m pip install --break-system-packages git+https://github.com/dpkp/kafka-python.git still is not resolving the issue. i can not downgrade python 3.11 due to a HIGH vulnerability with ecdsa pacakge. Any work around solution for this problem at the moment?

The current workaround solution would be to replace your kafka-python installation with my fork at kafka-python-ng for now.

Thank god. The fork version saved the day.

[BSpendlove]

rip

[sfjhu]

+1 having this issue

[TarcisioOliveira2021]

Hey guys, this solved it for me: 1- Uninstall kafka-python using: pip uninstall kafka-python 2- Then run: pip install --break-system-packages git+https://github.com/dpkp/kafka-python.git

[csharplus]

The following code fix works for me with the current public build of kafka-python:

import sys

if sys.version_info >= (3, 12, 0):
    import six
    sys.modules['kafka.vendor.six.moves'] = six.moves

......

from kafka import KafkaConsumer, KafkaProducer

......

[ernestaskardzys]

@csharplus , can confirm, that your workaround helped me to resolve the issue with kafka-python:2.0.2

[aHemanth123]

@csharplus, It is working.