search

twmb / franz-go

franz-go contains a feature complete, pure Go library for interacting with Kafka from 0.8.0 through 3.6+. Producing, consuming, transacting, administrating, etc.
BSD 3-Clause "New" or "Revised" License
1.61k stars 158 forks source link

Expose compression method to caller for PollRecords/PollFetches #714

Closed genzgd closed 2 months ago

genzgd commented 2 months ago

Our consumer application aggressively manages Kubernetes memory requests, so we try to dynamically calculate how much memory is used by "in flight" fetches. Since Franz transparently decompresses the payload, it's difficult to tell the actual size and memory that the original fetch used. I know that this could change at any time, but having an idea of what compression method the producer used could improve our "memory needed" calculations. It would also be useful for metrics about what compression our customers used.

twmb commented 2 months ago

I think the information you're looking for is available here:

https://pkg.go.dev/github.com/twmb/franz-go/pkg/kgo#HookProduceBatchWritten https://pkg.go.dev/github.com/twmb/franz-go/pkg/kgo#HookFetchBatchRead

genzgd commented 2 months ago

Awesome, I'll take a look!