search

numaproj / numaflow-python

Numaflow Python SDK
Apache License 2.0
50 stars 17 forks source link

feat: Implementation of BatchMap SDK client #178

Closed magelisk closed 2 months ago

magelisk commented 2 months ago

This creates BatchMap SDK implementation. Based on current MR: https://github.com/numaproj/numaflow/pull/1778

I tried to follow all existing patterns so this should fit in smoothly. However, I have made one change which is provide the servicer/server interaction the ability to have more custom data presentations. Since bidirectional stream lends itself to a few unique challenges, these updates provide an "advanced" interface for users to provide their own data access patterns. I've provided two additional implementations to server as examples as well as common use cases that were discussed in design talks I was included in.

codecov[bot] commented 2 months ago

Codecov Report

Attention: Patch coverage is 70.52980% with 89 lines in your changes missing coverage. Please review.

Project coverage is 91.49%. Comparing base (02009c8) to head (7dc2b5d).

Files Patch % Lines
pynumaflow/batchmapper/servicer/async_servicer.py 61.11% 31 Missing and 11 partials :warning:
pynumaflow/batchmapper/servicer/utils.py 0.00% 18 Missing :warning:
pynumaflow/batchmapper/multiproc_server.py 0.00% 17 Missing :warning:
pynumaflow/batchmapper/_dtypes.py 91.73% 8 Missing and 2 partials :warning:
pynumaflow/batchmapper/async_server.py 93.93% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #178 +/- ## ========================================== - Coverage 94.65% 91.49% -3.17% ========================================== Files 52 58 +6 Lines 2003 2305 +302 Branches 119 148 +29 ========================================== + Hits 1896 2109 +213 - Misses 78 154 +76 - Partials 29 42 +13 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

magelisk commented 2 months ago

I was apparently blind and did not see https://github.com/numaproj/numaflow-python/pull/177 when I made this, so this is duplicative - and seems to go beyond current desired objectives. Talking with @kohlisid on Slack, I'll plan for an MR later to bring the flexibility updates in another update