How low can memory per shard go?

[mfleming]

The current recommendation is to run RP with somewhere betweeen 1-2GiB per shard because there is a certain amount of "static" memory overhead. We don't currently have a good number for what the static overhead is but understanding it will be very helpful in knowhing how far we can push the partition per shard density.

• Run MPT after configuring a memory per partition replica (topic_memory_per_partition) to 2MB and total memory limit for a shard that is low, e.g. 1.5GiB. Keep decreasing the total memory to find the limit.

Expected outcome

• At some point MPT should fail
• Using the memory profiler we should be able to attribute the static memory overhead to subsystems?

JIRA Link: CORE-1598

[mfleming]

See https://github.com/redpanda-data/core-internal/issues/922#issuecomment-1830894151 for some more info.

[StephanDollberg]

Did a run of the MPT today with 4MB@250 per shard and it passed fine (compacted and not).

We could probably matrixize the MPT and always run this test.

[StephanDollberg]

2MB@500 also worked (non-TS).