Compress replication commands when serializing

line / centraldogma

Highly-available version-controlled service configuration repository based on Git, ZooKeeper and HTTP/2

https://line.github.io/centraldogma/

Apache License 2.0

601 stars 117 forks source link

Compress replication commands when serializing #480

Open trustin opened 4 years ago

trustin commented 4 years ago

When a user creates a large amount of big commits in a replicated environment, a replica might suffer from heap shortage, because we serialize commits (PushAsIsCommand to be precise) into JSON and store it into ZooKeeper.

To reduce the risk of running out of memory, we could compress the serialized replication command (zstd?)

trustin commented 4 years ago

Allow to disable compression so that gradual rollout is possible.
Consider supporting multiple compression algorithms - 'Zstd' if JNI library is available. 'deflate' otherwise.
- Think about the ways to expose compression options such as level.

0x1306e6d commented 2 years ago

I'm interested in this issue.

Think about the ways to expose compression options such as level.

I think that we can provide an option like CompressionType that has ZSTD, and DEFLATE (and BROTLI?).