Open alm0ra opened 9 months ago
@ods
Yes, we're interested in adding type hints. I'd like to highlight some issues here.
Thank you for bringing up those concerns. I understand the importance of ensuring that type hints are accurate and consistent. Regarding the Gradual approach doesn't work good here
point, I believe we can still make progress by prioritizing certain areas for annotation while acknowledging that some parts may remain untyped initially.
By leveraging tools like MyPy, we can focus on annotating sections of the codebase where types are clearer, thereby incrementally improving the overall typing coverage. We can then gradually address the more complex or ambiguous sections, such as the custom serialization functions, as we gain more clarity on their typings.
https://github.com/aio-libs/aiokafka/pull/981 should I close this PR?
I hope to find time on weekend to look through it.
Beartype could be of interest here. I've successfully used it in unit tests to increase confidence that type hints are accurate. It can be injected by using an early stage pytest hook.
@antonagestam, thanks for the suggestion! It looks interesting, definitely worth to try. Do you use it via pytest-beartype or in some other way?
@ods Currently I use the decorator, mostly because the code-base is too large for it to be feasible to introduce it everywhere at once, and we don't want to enable it for all test suites. After some initial acclimatization I hope we'll be able to use one of the import hooks.
Here is the signature of AIOKafkaProducer:
def __init__(self, *, loop=None, bootstrap_servers='localhost', ...)
The bootstrap_servers
parameter has a default value of localhost
, so the type inference system infers it as a string (and throws an error when a list of strings is passed in).
This can be confusing for many users who use the type system. Can we prioritize adding type annotations for these classes?
As a user of aiokafka, I like typing hints. I want to contribute to the typing feature. Where should I start from?
Hi @odysa, thank you for your willingness to contribute! Probably we need more communication with interested people to discuss problems. Let me highlight some difficult places:
mypy
doesn't work well with partially typed code, as it assume Any
everywhere instead of type inference. Probably some way of runtime verification is needed.ptotocol
package. Right now it has with methods in subclasses not following initial signature, which doesn't work when type checked.@ods please reopen this issue :)
Following pep-561 should be mentioned here too.
...
- Some features like custom serialization functions don't allow us to provide good annotations. So, we may want to consider deprecating such features first.
I believe this isn't too difficult, as long as you make AIOKafkaConsumer
and AIOKafkaProducer
generic, IOW:
KT = TypeVar("KT", covariant=True)
VT = TypeVar("VT", covariant=True)
class AIOKafkaConsumer(Generic[KT, VT]):
...
def __init__(
self,
...
key_deserializer: Callable[[bytes], KT]=lambda x: x,
value_deserializer: Callable[[bytes], VT]=lambda x: x,
...
def _identity(data: bytes) -> bytes:
return data
KT = TypeVar("KT", contravariant=True)
VT = TypeVar("VT", contravariant=True)
class AIOKafkaProducer(Generic[KT, VT]):
...
def __init__(
self,
...
key_serializer: Callable[[KT], bytes]=_identity,
value_serializer: Callable[[VT], bytes]=_identity,
...
Describe the solution you'd like I'd like to propose adding type hints to the
Aiokafka
project. I've usedmypy
to analyze the codebase, and I believe that adding type hints could significantly improve the code's readability, maintainability, and robustness.result for mypy
By adding type hints, developers can better understand function signatures and catch potential bugs early in the development process. Additionally, it can make it easier for newcomers to contribute to the project and for existing contributors to navigate the codebase.
I'm willing to contribute to this effort if the maintainers are interested in pursuing it. Adding type hints could be a valuable enhancement to the Aiokafka project.
I can start with aiokafka/consumer/consumer.py and fix all issue in this file first and continue with other files in another PR's or you can assign others to contribute