search

aio-libs / aiokafka

asyncio client for kafka
http://aiokafka.readthedocs.io/
Apache License 2.0
1.09k stars 224 forks source link

PY_SSIZE_T_CLEAN macro must be defined for '#' formats #926

Open paulefoa opened 8 months ago

paulefoa commented 8 months ago

Describe the bug I got this error when trying to produce the message using aiokafka 0.8.1 and Python 3.11.6

This urls might useful for someone who is smarter than me https://bugs.python.org/issue40943 https://peps.python.org/pep-0353/

Here's the stack trace:

[Tue 2023-10-10 11:13:34+0000] ERROR [aiokafka.producer.sender._sender_routine:167] Unexpected error in sender routine
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 155, in _sender_routine
    task.result()
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 260, in _send_produce_req
    await handler.do(node_id)
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 714, in do
    request = self.create_request()
              ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 678, in create_request
    (tp.partition, batch.get_data_buffer())
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/message_accumulator.py", line 234, in get_data_buffer
    return self._builder._build()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/message_accumulator.py", line 87, in _build
    self._buffer = self._builder.build()
                   ^^^^^^^^^^^^^^^^^^^^^
  File "aiokafka/record/_crecords/default_records.pyx", line 716, in aiokafka.record._crecords.default_records.DefaultRecordBatchBuilder.build
  File "aiokafka/record/_crecords/default_records.pyx", line 701, in aiokafka.record._crecords.default_records.DefaultRecordBatchBuilder._maybe_compress
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
[Tue 2023-10-10 11:13:34+0000] ERROR [asyncio.default_exception_handler:1771] Exception in callback Sender._fail_all(<Task finishe...ch delivery')>)
handle: <Handle Sender._fail_all(<Task finishe...ch delivery')>)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 70, in _fail_all
    self._txn_manager.fatal_error(task_exception)
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/transaction_manager.py", line 163, in fatal_error
    if self._transaction_waiter.done():
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'done'

I've tried to add this lines of code:

#ifndef PY_SSIZE_T_CLEAN
#define PY_SSIZE_T_CLEAN

to aiokafka/record/_crecords/crc32c.h and aiokafka/record/_crecords/crc32c.c And it worked, but when I used lz4 compression I got this error:

[Tue 2023-10-10 11:13:34+0000] ERROR [aiokafka.producer.sender._sender_routine:167] Unexpected error in sender routine
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 155, in _sender_routine
    task.result()
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 260, in _send_produce_req
    await handler.do(node_id)
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 714, in do
    request = self.create_request()
              ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 678, in create_request
    (tp.partition, batch.get_data_buffer())
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/message_accumulator.py", line 234, in get_data_buffer
    return self._builder._build()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/message_accumulator.py", line 87, in _build
    self._buffer = self._builder.build()
                   ^^^^^^^^^^^^^^^^^^^^^
  File "aiokafka/record/_crecords/default_records.pyx", line 716, in aiokafka.record._crecords.default_records.DefaultRecordBatchBuilder.build
  File "aiokafka/record/_crecords/default_records.pyx", line 701, in aiokafka.record._crecords.default_records.DefaultRecordBatchBuilder._maybe_compress
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
[Tue 2023-10-10 11:13:34+0000] ERROR [asyncio.default_exception_handler:1771] Exception in callback Sender._fail_all(<Task finishe...ch delivery')>)
handle: <Handle Sender._fail_all(<Task finishe...ch delivery')>)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 70, in _fail_all
    self._txn_manager.fatal_error(task_exception)
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/transaction_manager.py", line 163, in fatal_error
    if self._transaction_waiter.done():
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'done'

So I guess that helps somewhat but kafka.codecs also has this problem?

Environment (please complete the following information):

ods commented 8 months ago

Why do you suspect crc32c.c/crc32s.h? They don't include Python.h and don't deal with Python objects conversion.

Do you install binary package or compile it from source? If later, what Cython version do you use? What OS? CI pipeline should have catch the error, if it was in pre-build packages.