search

jemalloc / jemalloc

http://jemalloc.net/
Other
9.59k stars 1.46k forks source link

OSX: linker omits zone.o when statically linking jemalloc #708

Open tamird opened 7 years ago

tamird commented 7 years ago

In my testing, the macOS linker always omits zone.o when linking statically. See https://github.com/cockroachdb/cockroach/pull/15275 where we ran into this after switching to jemalloc's native build system.

I believe this is at least a documentation issue, since anyone statically linking jemalloc on macOS is probably not getting zone registration in their binary.

EDIT: the below was found to be a red herring, but remains for posterity.

We observed an issue where jemalloc was not being called from our C++ code in CockroachDB.

For context, we are (ab)using the Go build system to link jemalloc, rocksdb, and other non-Go libraries into CockroachDB. A consequence of this is that those libraries are linked statically.

For reasons I don't fully understand, the dynamic linker on OSX behaves in such a way that when libc++ is dynamically linked, the C++ operators new and delete do not call jemalloc as I would expect, instead apparently using the system allocator. The problem is not observed when linking jemalloc dynamically.

cc @petermattis @bdarnell @benesch @a-robinson

tamird commented 7 years ago

cc @glandium

glandium commented 7 years ago

It's totally fine to go through the system allocator on Mac, because it should circle back in jemalloc.

Is enable_zone_allocator properly set in your jemalloc Makefile?

tamird commented 7 years ago 
$ grep -F enable_zone_allocator Makefile
enable_zone_allocator := 1
ifeq ($(enable_zone_allocator), 1)

It's totally fine to go through the system allocator on Mac, because it should circle back in jemalloc.

You say that, but that's not what's happening; notice that jemalloc stats are not picking up any of the allocations.

FWIW, configure was invoked as ./configure --with-jemalloc-prefix=''.

glandium commented 7 years ago

What may be happening is that by statically linking, zone.o ends up being excluded from the final binary. Try to link your program with -Wl,-all_load

tamird commented 7 years ago

Yup, that fixed it. Yikes, this means that this probably doesn't work correctly for most users on OS X.

tamird commented 7 years ago

Turns out this was a bug in our abuse of the Go build system; we weren't building zone.c.

I'm going to close this as https://github.com/cockroachdb/c-jemalloc/pull/21 is going to fix our issue, but there still exists a documentation issue here; naive users of jemalloc+cpp are likely going to hit that issue where the linker is excluding zone.o.

tamird commented 7 years ago

We just ran into this again after having switched to jemalloc's native build system. This time, it's the linker issue where zone.o is being omitted because its symbols aren't referenced. See https://github.com/cockroachdb/cockroach/pull/15275.

I've updated the issue description. I'm reopening because it seems that this deserves to be documented, at least, or perhaps deserves being solved in such a way that doesn't require special handling by users.