aio-libs / aiokafka

asyncio client for kafka
http://aiokafka.readthedocs.io/
Apache License 2.0
1.16k stars 233 forks source link

Proposal to Add Type Hints #980

Open alm0ra opened 8 months ago

alm0ra commented 8 months ago

Describe the solution you'd like I'd like to propose adding type hints to the Aiokafka project. I've used mypy to analyze the codebase, and I believe that adding type hints could significantly improve the code's readability, maintainability, and robustness.

result for mypy

Found 2475 errors in 109 files (checked 133 source files)

By adding type hints, developers can better understand function signatures and catch potential bugs early in the development process. Additionally, it can make it easier for newcomers to contribute to the project and for existing contributors to navigate the codebase.

I'm willing to contribute to this effort if the maintainers are interested in pursuing it. Adding type hints could be a valuable enhancement to the Aiokafka project.

I can start with aiokafka/consumer/consumer.py and fix all issue in this file first and continue with other files in another PR's or you can assign others to contribute

alm0ra commented 8 months ago

@ods

ods commented 8 months ago

Yes, we're interested in adding type hints. I'd like to highlight some issues here.

alm0ra commented 8 months ago

Thank you for bringing up those concerns. I understand the importance of ensuring that type hints are accurate and consistent. Regarding the Gradual approach doesn't work good here point, I believe we can still make progress by prioritizing certain areas for annotation while acknowledging that some parts may remain untyped initially.

By leveraging tools like MyPy, we can focus on annotating sections of the codebase where types are clearer, thereby incrementally improving the overall typing coverage. We can then gradually address the more complex or ambiguous sections, such as the custom serialization functions, as we gain more clarity on their typings.

alm0ra commented 8 months ago

981 should I close this PR?

ods commented 8 months ago

https://github.com/aio-libs/aiokafka/pull/981 should I close this PR?

I hope to find time on weekend to look through it.

antonagestam commented 8 months ago

Beartype could be of interest here. I've successfully used it in unit tests to increase confidence that type hints are accurate. It can be injected by using an early stage pytest hook.

https://github.com/beartype/beartype

ods commented 8 months ago

@antonagestam, thanks for the suggestion! It looks interesting, definitely worth to try. Do you use it via pytest-beartype or in some other way?

antonagestam commented 8 months ago

@ods Currently I use the decorator, mostly because the code-base is too large for it to be feasible to introduce it everywhere at once, and we don't want to enable it for all test suites. After some initial acclimatization I hope we'll be able to use one of the import hooks.

laipz8200 commented 7 months ago

Here is the signature of AIOKafkaProducer:

def __init__(self, *, loop=None, bootstrap_servers='localhost', ...)

The bootstrap_servers parameter has a default value of localhost, so the type inference system infers it as a string (and throws an error when a list of strings is passed in).

This can be confusing for many users who use the type system. Can we prioritize adding type annotations for these classes?

odysa commented 7 months ago

As a user of aiokafka, I like typing hints. I want to contribute to the typing feature. Where should I start from?

ods commented 7 months ago

Hi @odysa, thank you for your willingness to contribute! Probably we need more communication with interested people to discuss problems. Let me highlight some difficult places:

dimastbk commented 6 months ago

@ods please reopen this issue :)

ssbarnea commented 2 months ago

Following pep-561 should be mentioned here too.

jackgene commented 3 weeks ago

...

  • Some features like custom serialization functions don't allow us to provide good annotations. So, we may want to consider deprecating such features first.

I believe this isn't too difficult, as long as you make AIOKafkaConsumer and AIOKafkaProducer generic, IOW:

KT = TypeVar("KT", covariant=True)
VT = TypeVar("VT", covariant=True)

class AIOKafkaConsumer(Generic[KT, VT]):
    ...
    def __init__(
        self,
        ...
        key_deserializer: Callable[[bytes], KT]=lambda x: x,
        value_deserializer: Callable[[bytes], VT]=lambda x: x,
        ...
def _identity(data: bytes) -> bytes:
    return data

KT = TypeVar("KT", contravariant=True)
VT = TypeVar("VT", contravariant=True)

class AIOKafkaProducer(Generic[KT, VT]):
    ...
    def __init__(
        self,
        ...
        key_serializer: Callable[[KT], bytes]=_identity,
        value_serializer: Callable[[VT], bytes]=_identity,
        ...