Use Jemalloc for significant memory reduction

[asdacap]

• When looking at memory allocation it turns out that a large proportion of the memory allocation is due to glibc malloc fragmentation.
• I call this "magic overhead" which can be measured via .net gc info, Load-(Committed + DB memory).
• This is surprisingly common for application utilizing rocksdb:
  • https://blog.cloudflare.com/the-effect-of-switching-to-tcmalloc-on-rocksdb-memory-use/
  • https://engineering.linkedin.com/blog/2021/taming-memory-fragmentation-in-venice-with-jemalloc
• This can be validated by limiting the number of heap via env variable MALLOC_ARENA_MAX to something like 4. The default is core count * 8 on 64 bit system.
• Here are the overhead size on various configuration. Tested on a Ryzen 9 7950X, memory limited to 32GB via systemd-run:

Configuration	Overhead (GB)
Default	13.5
Call malloc_trim every 5 minute	7.5
MALLOC_ARENA_MAX=32	4.5
MALLOC_MMAP_THRESHOLD=64000	5.5
Jemalloc via LD_PRELOAD	1.5

• Which basically mean, jemalloc is the end goal. However, I've no idea how to deploy this in a cross platform manner.

• Adjusting malloc parameters comes with tradeoff of either lock contention (reduced arena count) or higher system CPU usage (MMAP_THRESHOLD and friends).

• It is possible to change the malloc config via mallopts call at runtime, however it does not work for limiting num of arena as the arena would have already been created by .net runtime by the time managed code start running.

• Using mallopts for limiting MALLOC_MMAP_THRESHOLD seems promising, but it does not work as well as just limiting arena count to 32.

• Graph is (jemalloc, default, max arena 32, jemalloc, default, call trim every 5 minute)

[asdacap]

There seems to be an attempt at rocksdb binding library to bundle jemalloc https://github.com/curiosity-ai/rocksdb-sharp/commit/dee0d243c7aa3b219cfc0282a979756a7de5cf73

[benaadams]

The allocator contention is very bad during Old Receipts on Windows (enough to push block processing over 24secs)