search

SeldonIO / seldon-core

An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models
https://www.seldon.io/tech/products/core/
Other
4.37k stars 831 forks source link

Compression of Kafka messages to reduce costs #5041

Open mdworsky opened 1 year ago

mdworsky commented 1 year ago

We’d like to use Seldon Core, but our throughput is high enough to make uncompressed Kafka messages cost-prohibitive. But we think compression would reduce our costs enough to unblock us. ZSTD or GZIP would be nice, although if you're building support for even one codec, it shouldn't be too hard to support all the codecs that your Kafka client handles out of the box. Thanks!

ethanresnick commented 1 year ago

In particular, this would be for compressing messages on the producer side before they're sent to the broker.

ukclivecox commented 1 year ago

The PR above has docs that show how you can add producer settings for Kafka. Can you try?

ethanresnick commented 1 year ago

Thanks @cliveseldon! It's awesome to see that this should already be supported, and those docs help! We're still at least a few days away from actually spinning up a Seldon Core install but, when we do, we'll try it right away and let you know.