numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs
https://numaflow.numaproj.io
Apache License 2.0
1.01k stars 98 forks source link

feat: use protobuf to store header and messages in ISB #1771

Closed yhl25 closed 5 days ago

yhl25 commented 1 week ago

fixes #1768

codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 73.40426% with 25 lines in your changes missing coverage. Please review.

Project coverage is 61.27%. Comparing base (4e88f32) to head (f05932e).

Files Patch % Lines
pkg/isb/serde.go 78.94% 14 Missing and 2 partials :warning:
pkg/isb/stores/simplebuffer/buffer.go 58.33% 4 Missing and 1 partial :warning:
pkg/isb/stores/redis/write.go 40.00% 2 Missing and 1 partial :warning:
pkg/isb/errors.go 0.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1771 +/- ## ========================================== + Coverage 60.83% 61.27% +0.43% ========================================== Files 232 231 -1 Lines 20014 19830 -184 ========================================== - Hits 12175 12150 -25 + Misses 6993 6886 -107 + Partials 846 794 -52 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

vigith commented 6 days ago

watermark propagation is still using encoding/binary; let's take that up too, next?

yhl25 commented 5 days ago

any performance regressions?

i see that the WAL size can come down, so that is good :)

I didn't notice much difference with a low TPS pipeline. I have yet to perform some performance testing with a high TPS pipeline.