I'm not sure how much inter-communication there is between this project and kafka-python, so I'm opening a separate ticket here to discuss an idea I discussed over there that applies to this project as well. This is more of a discussion really, but I couldn't see the "Discussions" section enabled on this project, so I'm filing it as a "question" issue instead.
In https://github.com/dpkp/kafka-python/issues/2380 I mention how during some profiling we observed some bottlenecks around crc32c calculation because, for some reason, the python fallback routines were being used. After doing a deep dive, I found that both this project and kafka-python have python fallbacks, while at the same time catering for faster versions of the checksums: this project ships with its own C/Cython-based extension, while kafka-python optionally uses the crc32c package.
My question there, and now here, is: would you consider adding a hard dependency on crc32c to avoid having python fallbacks in the first place, while also not having to maintain a C/Cython-based module for this purpose?
Please see the above thread for a longer discussion, and thanks in advance for any consideration given to this idea.
Hi,
I'm not sure how much inter-communication there is between this project and
kafka-python, so I'm opening a separate ticket here to discuss an idea I discussed over there that applies to this project as well. This is more of a discussion really, but I couldn't see the "Discussions" section enabled on this project, so I'm filing it as a "question" issue instead.In https://github.com/dpkp/kafka-python/issues/2380 I mention how during some profiling we observed some bottlenecks around crc32c calculation because, for some reason, the python fallback routines were being used. After doing a deep dive, I found that both this project and
kafka-pythonhave python fallbacks, while at the same time catering for faster versions of the checksums: this project ships with its own C/Cython-based extension, whilekafka-pythonoptionally uses thecrc32cpackage.My question there, and now here, is: would you consider adding a hard dependency on
crc32cto avoid having python fallbacks in the first place, while also not having to maintain a C/Cython-based module for this purpose?Please see the above thread for a longer discussion, and thanks in advance for any consideration given to this idea.