I have an application which writes large volume of data to Victoria Metrics using prometheus remote write API. The API specs requires compressing the data being written using snappy encoding. When I do memory profile of my application, the snappy encoding alone is using ~900MB of memory.
This is problematic because I process data in real time using multiple replicas of my application and then write the processed data to Victoria Metrics. So the total memory usage by all replicas becomes too high.
I have an application which writes large volume of data to Victoria Metrics using prometheus remote write API. The API specs requires compressing the data being written using snappy encoding. When I do memory profile of my application, the snappy encoding alone is using ~900MB of memory.
This is problematic because I process data in real time using multiple replicas of my application and then write the processed data to Victoria Metrics. So the total memory usage by all replicas becomes too high.
A similar issue, where snappy decode is consuming high memory in prometheus: https://github.com/prometheus/prometheus/issues/13174