When using the gzip codec, topic messages occupy a minimum size of more than 32 KiB, due to the gzip.Reader stored inside each message. The reader is not utilized until the message is actually read. This results in excessive memory usage and in our case led to OOM.
Describe Preferred Solution
The topic reader uncompresses messages in the background.
Describe Alternatives
The reader is created upon the first call to the Read method of the message.
Feature Request
Describe the Feature Request
When using the gzip codec, topic messages occupy a minimum size of more than 32 KiB, due to the gzip.Reader stored inside each message. The reader is not utilized until the message is actually read. This results in excessive memory usage and in our case led to OOM.
Describe Preferred Solution
The topic reader uncompresses messages in the background.
Describe Alternatives
The reader is created upon the first call to the Read method of the message.
Related Code
The reader for each message is created here: https://github.com/ydb-platform/ydb-go-sdk/blob/1b2f8d406425b28c18fae7301639c2a4204f5e70/internal/topic/topicreaderinternal/batch.go#L82
Message Read method: https://github.com/ydb-platform/ydb-go-sdk/blob/43b57d6c1463c492b99cb25d9655a66c7b36bd8f/internal/topic/topicreaderinternal/message.go#L72
Additional Context
If the feature request is approved, would you be willing to submit a PR? Yes