ArroyoSystems / arroyo

Distributed stream processing engine in Rust
https://arroyo.dev
Apache License 2.0
3.8k stars 220 forks source link

Disable jemalloc on macos due to potential UDF FFI allocation inconsistencies #750

Closed mwylde closed 1 month ago

mwylde commented 1 month ago

In the UDF FFI pathway, we rely on allocating memory in the host which is freed in the plugin and vice-versa. This is reliable on Linux with both the standard allocator and jemalloc (which we introduced in #719 for its improved performance and profiling capabilities). However, on MacOS this approach has the potential to segfault with jemalloc:

arroyo(46639,0x16bec3000) malloc: *** error for object 0x110768140: pointer being realloc'd was not allocated
arroyo(46639,0x16bec3000) malloc: *** set a breakpoint in malloc_error_break to debug

This occurs when accessing the plugin-allocated memory on the host. On Linux the same allocator will be used in FFI code and in the host code, but that does not appear to be the case for darwin; I believe that on macos the UDF is using the system allocator while the host is using jemalloc.

There may be a path forward for jemalloc on macos, but since most of the benefits will be in production on linux the fastest path here seems to be to disable it on mac.