jemalloc failure on raspberry pi 5

quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.

https://quickwit.io

Other

7.42k stars 300 forks source link

jemalloc failure on raspberry pi 5 #4785

Open nickkaltner opened 4 months ago

nickkaltner commented 4 months ago

Describe the bug Running on a raspberry pi 5, there are 16KB page sizes which breaks the jemalloc configuration on quickwit

It won't run, here is the output;

./quickwit-v0.8.0/quickwit --help
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed
Aborted

confirmation of the page size;

$ getconf PAGE_SIZE
16384

Steps to reproduce (if applicable) Steps to reproduce the behavior:

follow the installation instructions on a raspberry pi 5
watch it crash

Expected behavior

It should run as per the instructions

Configuration: Please provide:

Output of quickwit --version

n/a because this crashes but the version i was running is 0.8.0 on aarch64-unknown-linux-gnu (7afa596)

The index_config.yaml

n/a

nickkaltner commented 4 months ago

note there is a temporary workaround which is adding the line kernel=kernel8.img to /boot/firmware/config.txt and rebooting.

you can confirm the fix has applied by

$ getconf PAGE_SIZE
4096

and then

./quickwit --version
Quickwit 0.8.0 (aarch64-unknown-linux-gnu 2024-03-18T15:20:23Z 7af4596)

trinity-1686a commented 4 months ago

dumping some observations:

cross-compiling with JEMALLOC_SYS_WITH_LG_PAGE=16 should make things work on a 16k page system, and not break on 4k according to https://github.com/home-assistant/docker-base/pull/248. (we probably want 16 and not 14 because BCM2712 reportedly default to 64k pages). I don't know the exact tradeoff, but i guess this would cause jemalloc to allocate slightly more than required
cross-compiling without --features jemalloc (which is implied by --features release-feature-set) would use system allocator and not cause issue. In my experience, GNU allocator tend to bloat a lot with rust, so targetting musl instead of gnu might be a decent idea too.
compiling directly on a system with PAGE_SIZE=16384 should also work as jemalloc auto-detect host's configuration.

fulmicoton commented 4 months ago

A simpler alternative would be to compile without jemalloc. It is an optional feature of quickwit.