Point72 / csp

csp is a high performance reactive stream processing library, written in C++ and Python
https://github.com/Point72/csp/wiki
Apache License 2.0
191 stars 33 forks source link

Ensure that adapters can be constructed in a simpler manner than they are currently #272

Open timkpaine opened 3 months ago

timkpaine commented 3 months ago

1. Adapters + Impls where the Impl is tiny:

https://github.com/Point72/csp/blob/234f94cfd9924ca479485b97d56f7cc91141f3d2/examples/04_writing_adapters/e2_pullinput.py#L19

MyPullAdapter = py_pull_adapter_def("MyPullAdapter", MyPullAdapterImpl, ts[int], interval=timedelta, num_ticks=int)

One would expect we could do this via type annotations in some form or another, or something like pydantic since the py_pull_adapter_def mostly serves to wrap the impl inputs.

2. Minimal Managers / other places where excess classes are needed

https://github.com/Point72/csp/blob/234f94cfd9924ca479485b97d56f7cc91141f3d2/examples/04_writing_adapters/e3_adaptermanager_pullinput.py https://github.com/Point72/csp/blob/main/examples/04_writing_adapters/e5_adaptermanager_pushinput.py

These examples requires 4+ classes (!!):

This is pretty bad UX/design.

3. Automatic py_push_adapter_def / py_output_adapter_def, etc

Similar to the above, we often create these "anonymous" objects for binding the adapter-specific code to the engine wrapping necessary, I think we can consolidate this in some form https://github.com/Point72/csp/blob/234f94cfd9924ca479485b97d56f7cc91141f3d2/csp/adapters/kafka.py#L187