Something wrong using gperftools to do heap profiling under JVM and native DSO

[zhanglistar]

Environment: ubuntu 20.04 Linux 5.4.0-99-generic clang-16 openjdk version "1.8.0_382" OpenJDK Runtime Environment (build 1.8.0_382-8u382-ga-1~20.04.1-b05) OpenJDK 64-Bit Server VM (build 25.382-b05, mixed mode)

Running environment: libch.so, C++ native library compiled by clang++-16 used by Spark thrift server on JVM using the following command:

export LD_PRELOAD="/work/gperftools-2.13/libs/libtcmalloc_and_profiler.so $LIBCH"
export HEAPPROFILE=/root/
./sbin/start-thriftserver.sh \
  --hiveconf hive.server2.thrift.port=9999 \
  --master local[1] \
  --driver-memory 16g --driver-cores 4 \
  --total-executor-cores 90 --executor-memory 60g --executor-cores 15 \
  --conf spark.driver.memoryOverhead=8G \

And we got several heap profiles, using the command to get callgraph.

pprof --svg /usr/lib/jvm/java-8-openjdk-amd64/bin/java _3157452.0130.heap  > 130.svg

And we got it. But the flame graph shows that JVM_FindSignal uses the most memory. But I check this function in jdk-8 source code, it's really simple, https://github.com/keerath/openjdk-8-source/blob/5f6e9d42a9f6b6736100c9c6f43f5f5ea1570cfb/hotspot/src/os/linux/vm/jvm_linux.cpp#L178

Above is the heap profile.

Did I miss something? Thanks in advance.

[alk]

This could be for example due to lack of debug symbols in your JVM.

BTW do keep in mind that we don't support perl pprof anymore. Please switch to go version (go get -u github.com/google/pprof). It has native flame graphs support. I recommend pprof --http=: . It'll open browser, then you'll be able to select flame graph. It is also somewhat interactive, so more helpful too.

I've just now tried sweethome3d (random java package I found on my distro) and heap profile looks sane-ish (it allocates a lot of X/GLX bits).

[alk]

Ah, so to clarify. At least on my debian sid box openjdk17 packages do seem to have plausibly looking debug symbols. From that sweethome3d run I mentioned above. Do try go pprof and maybe it'll work better.

[zhanglistar]

@alk Thanks for your reply. I will try to get jdk 8 with debug info.