search

sudoblockio / icon-etl

MIT License
0 stars 0 forks source link

Validate compressionType needed when topics are created externally #2

Closed robcxyz closed 2 years ago

robcxyz commented 3 years ago

Per title, is this relevant anymore? For situations where you don't create topics out of band, I would imagine this is useful.

shinyfoil commented 3 years ago

https://newbedev.com/if-i-set-compression-type-at-topic-level-and-producer-level-which-takes-precedence

It looks like we'd want to set a sane default on the topic and ensure every client is capable of using it. CF did some benchmarking on types (https://blog.cloudflare.com/squeezing-the-firehose/) and seemed to like snappy for some record types, but maybe we can also do some benchmarking?