Open ukolovda opened 10 months ago
Hey @ukolovda 👋
Can you provide us with some more information? Most importantly: Can you reproduce it? And if so, can you provide a minimal example?
Here’s a small Dockerfile
which reproduces this issue on both amd64 and arm64:
FROM ruby:3.2.2-slim-bookworm
RUN \
apt-get update --yes && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
build-essential libjemalloc2 && \
ln -s /usr/lib/*-linux-gnu/libjemalloc.so.2 /usr/lib/libjemalloc.so.2
ENV LD_PRELOAD=/usr/lib/libjemalloc.so.2
RUN gem install mini_racer
CMD ruby -r mini_racer -e 'p MiniRacer::Context.new.eval("1+2")'
I did some testing with other base images and it seems like ruby:3.1-slim-bullseye
with libjemalloc2
5.2.1-3 works fine but ruby:3.1-slim-bookworm
with 5.3.0-1 does not. Installing the older jemalloc into Bookworm did not work so presumably it must be something else other than the jemalloc version?
ah, jemalloc 😞
We had several Ruby and mini_racer issues with jemalloc in the past, e.g. https://github.com/rubyjs/mini_racer/issues/242.
I think that might be something @lloeki or @SamSaffron can take a look?
I use jemalloc too...
@jasoncodes , thank you!
I'm wondering whether it would get confused at times as to which malloc/free symbol is which depending on how the ruby jemalloc linking happens (I think glibc is always dynamic, dunno about musl).
Oh wait, I didn't see that:
ENV LD_PRELOAD=/usr/lib/libjemalloc.so.2
So you're injecting jemalloc, not linking Ruby against jemalloc at Ruby build time? Do you get the same result with a Ruby build against jemalloc?
Do you get the same result with a Ruby build against jemalloc?
Good question. Looks to work fine if the Ruby is built using jemalloc instead of injecting.
I tested this using the first Ruby 3.2.2 w/ jemalloc base image I found which I could confirm has a working jemalloc with MALLOC_CONF=stats_print:true ruby -e exit
:
FROM jiting/ruby:3.2.2-slim-bookworm
RUN apt-get update && apt-get -y install build-essential
RUN gem install mini_racer
CMD ruby -r mini_racer -e 'p MiniRacer::Context.new.eval("1+2")'
Okay so I think what might happen is:
malloc
+free
symbols to glibcLD_PRELOAD
is used to hook jemalloc in it applies to Ruby but it also applies to any other thing that dynamically links to libc's malloc
+free
(which would be any Ruby extension having linked to glibc)mini_racer_loader
that binds symbols with RTLD_LOCAL
and RTLD_DEEPBIND
when dlopen
ing mini_racer_extension
, this is supposed to make mini_racer_extension
symbols hidden from outside and prefer its own internal symbols, but there is no malloc
+free
inside, so they resolve to glibc's malloc
+free
.malloc
+free
internal to mini_racer_extension
, depending of how linking is done inside the mini_racer_extension
shared library for these symbols (e.g malloc+free are internal but dynamic symbols) it might be so that LD_PRELOAD
takes precedence over RTLD_DEEPBIND
. Not the case here but we can entertain the thought.LD_PRELOAD
, libv8 will use jemalloc instead of glibc's malloc
+free
When ruby is built statically against jemalloc, the malloc
+free
symbols are entirely static (and might even be elided entirely, instead replaced by pure addresses), in which case they are unavailable to extensions. Thus Ruby can do its things with jemalloc and libv8 its things with glibc, and since ownership is split (nothing allocated from one is ever freed by the other) they both live happily ever after.
The remaining question is why libv8 would not like having its allocator swapped with jemalloc via LD_PRELOAD
but I guess that's an upstream libv8 question.
This morning I came across https://github.com/docker-library/ruby/issues/182#issuecomment-1368432448 which suggests using patchelf
instead of LD_PRELOAD
:
FROM ruby:3.2.2-slim-bookworm
RUN \
apt-get update --yes && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
build-essential libjemalloc2 patchelf && \
patchelf --add-needed libjemalloc.so.2 /usr/local/bin/ruby
RUN gem install mini_racer
CMD ruby -r mini_racer -e 'p MiniRacer::Context.new.eval("1+2")'
This seems like a nicer way to patch an existing Ruby as it avoids affecting all processes. Unfortunately it doesn’t solve the problem here as dynamic linking still results in libv8 using jemalloc and crashing on exit. I’m guessing within the single Ruby process, patchelf
and LD_PRELOAD
are effectively the same thing.
Only option right now to use mini_racer with jemalloc seems to be building Ruby with jemalloc.
Only option right now to use mini_racer with jemalloc seems to be building Ruby with jemalloc.
At least it seems to be more reliable. I'm running with LD_PRELOAD
in production for a while and haven't had an issue so far; I'm still using ruby:3.2.2-bullseye
(docker) though.
Since this is popping up again and again over time, I'm wondering if we should add something to the troubleshooting section of the README to recommend not using jmalloc for now, or use Ruby statically build against it. What do you think, @lloeki?
FYI Rails has merged a PR to LD_PRELOAD
jemalloc in their default Dockerfile template: https://github.com/rails/rails/pull/50943.
In lieu of something in the README, may I suggest renaming this Issue to mention jemalloc?
Segfault occured when I stop the docker container.
I'm not sure that is Mini_racer trouble, but call stack is in it.
Dump is in the attachment.
MiniRacer SegFault dump.txt