Is your feature request related to a problem? Please describe.
The segments written by the segment writer process always contain a fixed number of log entries which is configurable on a per ra system basis (segment_max_entries). The number of entries per segment needs to be chosen when a segment is created as the segment file both contains the index and the data.
This is fine if commands written within a system are roughly of the same size and the system is appropriately configured according to the average size.
In RabbitMQ however the size of each enqueue command could vary from a few bytes to several MB which means we could be create both oversized and undersized segments. E.g. a quorum queue with 50MB messages would eventually create a segment of over 200GB in size with a segment_max_entries of 4096. This is clearly not optimal.
Describe the solution you'd like
Each Ra system should have a segment target size that the segment writer will attempt to adjust the number of entries for each segment accordingly to get close to the target size.
To safely do so the starting number of entries for the first segment should be a low value that is doubled or halved based on the final size of the previous segment.
To more accurately size the segment write could even base the size of the results of several past segments should this info be available (as may be the case when the segment writer is writing multiple segments in one "flush")
Is your feature request related to a problem? Please describe.
The segments written by the segment writer process always contain a fixed number of log entries which is configurable on a per ra system basis (
segment_max_entries). The number of entries per segment needs to be chosen when a segment is created as the segment file both contains the index and the data. This is fine if commands written within a system are roughly of the same size and the system is appropriately configured according to the average size.In RabbitMQ however the size of each enqueue command could vary from a few bytes to several MB which means we could be create both oversized and undersized segments. E.g. a quorum queue with 50MB messages would eventually create a segment of over 200GB in size with a segment_max_entries of 4096. This is clearly not optimal.
Describe the solution you'd like
Each Ra system should have a segment target size that the segment writer will attempt to adjust the number of entries for each segment accordingly to get close to the target size.
To safely do so the starting number of entries for the first segment should be a low value that is doubled or halved based on the final size of the previous segment.
To more accurately size the segment write could even base the size of the results of several past segments should this info be available (as may be the case when the segment writer is writing multiple segments in one "flush")
Describe alternatives you've considered
No response
Additional context
No response