libjemalloc.so.2: cannot allocate memory in static TLS block.

[oolonc]

Hello!

I having some trouble getting cl-rocksdb working with Jemalloc 5.3.0. When using the stock version from the Arch Linux repo it fails with the following error:

To load "cl-rocksdb/test":
  Load 1 ASDF system:
    cl-rocksdb/test
; Loading "cl-rocksdb/test"
..............[1/3] System 'cl-rocksdb/test' found. Loading the system..Aborted during step [1/3].
Unhandled LOAD-FOREIGN-LIBRARY-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                                  {10010881B3}>:
  Unable to load foreign library (ROCKSDB).
  Error opening shared object "librocksdb.so":
  /usr/lib/libjemalloc.so.2: cannot allocate memory in static TLS block.
--snip--

This issue describes same problem and was solved via the --disable-initial-exec-tls config option. Compiling Jemalloc with --disable-initial-exec-tls enabled gives a new and much shorter error:

To load "cl-rocksdb/test":
  Load 1 ASDF system:
    cl-rocksdb/test
; Loading "cl-rocksdb/test"
..............
Running test suite LOW-LEVEL-SUITE
free(): invalid pointer
fatal error encountered in SBCL pid 223476 tid 223476:
SIGABRT received.

   0: fp=0x7fd0e3a95c40 pc=0x7fd0e3b1e8ec Foreign function (null)

If anyone had any pointers how to proceed I'd be much obliged.

[veer66]

I had the same problem when I used Garuda Linux. I couldn't fix.

[veer66]

I built this Dockerfile for testing/debugging.

• Dockerfile

FROM archlinux

RUN mkdir -p ~/.config/common-lisp
RUN mkdir /work
WORKDIR /work
ENV HOME=/work
COPY source-registry.conf /work/.config/common-lisp/
RUN pacman -Syu --noconfirm sbcl rocksdb wget rlwrap
RUN wget https://beta.quicklisp.org/quicklisp.lisp
RUN sbcl --load quicklisp.lisp \
    --eval '(quicklisp-quickstart:install)' \
    --eval '(ql-util:without-prompting (ql:add-to-init-file))' \
    --quit
RUN sbcl --eval '(ql-dist:install-dist "http://dist.ultralisp.org/" :prompt nil)' --quit

• source-registry.conf

(:source-registry
     (:tree (:home "lisp")) 
     :inherit-configuration)

@oolonc How do you rebuild jemalloc on Arch Linux? Is there any easy way?

[richardwesthaver]

my naive understanding of this issue is that calling dlopen with jemalloc is generally ill-advised - I'm on arch, pretty sure I just compiled rocksdb from source and didn't have any issues. If there's something wrong with jemalloc it can be disabled with the ROCKSDB_DISABLE_JEMALLOC option https://github.com/facebook/rocksdb/issues/5787#issuecomment-608283999 but this requires a custom PKGBUILD or to compile from source - I would check in the AUR to see if there's a dlopen-friendly distribution. or if you really need rocksdb from pacman and jemalloc sounds like building jemalloc from source with that config option would be the best way.

[j0ni]

+1 to @richardwesthaver - I had to build a jemalloc-free rocksdb on arch, using the following make invocation:

make shared_lib LUA_PATH= PORTABLE=1 USE_RTTI=1 DISABLE_JEMALLOC=1

The middle two are in the stock PKGBUILD, the last is I guess a recent slimming down of the option @richardwesthaver mentioned, and LUA_PATH is I guess a recent addition (since I build HEAD).

I disabled lua because it wants a static lib to link to (even for the dynamic target), and arch doesn't do static lib builds. I could do a local lua build as part of a fresh PKGBUILD, but I am not quite that deeply committed to either rocksdb or lua at this point. 😄

[veer66]

@oolonc Does a custom PKGBUILD work for you?