martinsumner
commented
2 years ago Changing the single block carrier threshold to a lower value on the eheap_alloc has a very negative impact. This change does not help
erlang.eheap_memory.sbct = 128
recon_alloc:memory(usage).
0.34210991984944206
(dev11@192.168.6.11)5> erlang:memory().
[{total,7731430560},
{processes,6054481520},
{processes_used,6054116632},
{system,1676949040},
{atom,1017017},
{atom_used,1009277},
{binary,1307030592},
{code,15398935},
{ets,300506192}]
(dev11@192.168.6.11)6> recon_alloc:memory(allocated).
22463975424
(dev11@192.168.6.11)7> rp(recon_alloc:fragmentation(current)).
[{{eheap_alloc,7},
[{sbcs_usage,0.21770690720393554},
{mbcs_usage,0.6661469330658784},
{sbcs_block_size,384285504},
{sbcs_carriers_size,1765150720},
{mbcs_block_size,16152944},
{mbcs_carriers_size,24248320}]},
{{eheap_alloc,23},
[{sbcs_usage,0.23321187683249425},
{mbcs_usage,0.5346889303188131},
{sbcs_block_size,249886832},
{sbcs_carriers_size,1071501312},
{mbcs_block_size,6938192},
{mbcs_carriers_size,12976128}]},
{{eheap_alloc,1},
...
hmmr
There have been reports from users of Riak suffering memory fragmentation with large clusters with leveled backends.
This appears to be an issue with both OTP 22.3 as well as OTP R16basho.
Primary indicator is:
recon_alloc:memory(usage).Which can be between 50% and 80%.
This is not universal, some leveled/Riak clusters do not see this. However, on clusters that do see it there is limited mitigation other than restarting nodes. The excess memory takes from the Page Cache, and so can have a direct impact on performance. In extreme cases unexpected and sudden memory exhaustion has been observed.
Horizontal scaling currently appears to offer better mitigation than vertical scaling.
The fragmentation is primarily with the
eheap_allocandbinary_allocand associated with inefficient use of multiblock carriers, primarily these carriers not seeming to ever transition to empty and be reclaimed.