search

memtt / numaprof

NUMAPROF is a NUMA memory profliler based on Pintool to track your remote memory accesses.
https://memtt.github.io/numaprof
Other
45 stars 7 forks source link

pintool integration on SLES/12.SP3 #25

Closed jgphpc closed 1 year ago

jgphpc commented 5 years ago

Hello,

Trying to build+use numaprof on a Cray system, i get:

numaprof -V ./src/lib/tests/TestMallocTracker

E: Unable to load numaprof-1.0.0/JG/lib/libnumaprof-pintool.so: 
dlopen failed: library "libstdc++.so.6" not found

ldd /numaprof-1.0.0/JG/lib/libnumaprof-pintool.so

    linux-vdso.so.1 (0x00007ffe1ab97000)
    libstdc++.so.6 => /opt/gcc/6.2.0/snos/lib64/libstdc++.so.6 (0x00002ab003af5000)
    libxed.so => /p/pin/src/pin-3.7-97619-g0d0c92f4f-gcc-linux/extras/xed-intel64/lib/libxed.so (0x00002ab003e7e000)
    libpin3dwarf.so => /p/pin/src/pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/lib-ext/libpin3dwarf.so (0x00002ab004358000)
    libdl-dynamic.so => /p/pin/src/pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libdl-dynamic.so (0x00002ab0034ad000)
    libstlport-dynamic.so => /p/pin/src/pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libstlport-dynamic.so (0x00002ab0034b0000)
    libm-dynamic.so => /p/pin/src/pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libm-dynamic.so (0x00002ab003575000)
    libc-dynamic.so => /p/pin/src/pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libc-dynamic.so (0x00002ab0035a6000)
    libunwind-dynamic.so => /p/pin/src/pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libunwind-dynamic.so (0x00002ab00363c000)
    libm.so.6 => /lib64/libm.so.6 (0x00002ab0048b3000)
    libc.so.6 => /lib64/libc.so.6 (0x00002ab004bb0000)
    /lib64/ld-linux-x86-64.so.2 (0x00002ab003489000)
    libgcc_s.so.1 => /opt/gcc/6.2.0/snos/lib64/libgcc_s.so.1 (0x00002ab004f55000)

Some relevant info about the system:

Currently Loaded Modulefiles:
gcc/6.2.0
craype-haswell
PrgEnv-gnu/6.0.4
CrayGNU/.18.08
numactl/2.0.12-CrayGNU-18.08

Interestingly, i get the same when i try to run pin directly. Have you tried numaprof on SLES ?

jg.

svalat commented 5 years ago

Hi, thanks very much for reporting and all details that will be usefull.

No I never yet tried on SLES, this might be something intersting to try and fix. Will try to give a look.

In case you have hints. Is pin working by itself on you distrib if you try some of their internal examples or do you think I made a mistake in my build rules ?

Cheers, Sébastien.

jgphpc commented 5 years ago

I would blame pin itself as it fails with the example from the README (using gcc/4.9.3):

../../../pin -t obj-intel64/opcodemix.so -- /bin/ls

E: Unable to load
pin-3.7-97619-g0d0c92f4f-gcc-linux/source/tools/SimpleExamples/obj-intel64/opcodemix.so:
dlopen failed: cannot locate symbol "_ZTVN10__cxxabiv120__si_class_type_infoE"
referenced by
"pin-3.7-97619-g0d0c92f4f-gcc-linux/source/tools/SimpleExamples/obj-intel64/opcodemix.so"...

c++filt _ZTVN10cxxabiv120si_class_type_infoE

vtable for __cxxabiv1::__si_class_type_info

Example usage

This example applies to a 64-bit application. For a 32-bit application please use "obj-ia32" instead of "obj-intel64" and add "TARGET=ia32" to the make command. To build and run a sample tool on Linux*: cd source/tools/SimpleExamples make obj-intel64/opcodemix.so ../../../pin -t obj-intel64/opcodemix.so -- /bin/ls This will instrument and run /bin/ls, the output for this tool is in opcodemix.out.

ldd SimpleExamples/obj-intel64/opcodemix.so

linux-vdso.so.1 (0x00007ffdc378e000)
libxed.so => pin-3.7-97619-g0d0c92f4f-gcc-linux/extras/xed-intel64/lib/libxed.so (0x00002b850d0e4000)
libpin3dwarf.so => pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/lib-ext/libpin3dwarf.so (0x00002b850d5be000)
libdl-dynamic.so => pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libdl-dynamic.so (0x00002b850ca5c000)
libstlport-dynamic.so => pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libstlport-dynamic.so (0x00002b850ca60000)
libm-dynamic.so => pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libm-dynamic.so (0x00002b850cb25000)
libc-dynamic.so => pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libc-dynamic.so (0x00002b850cb56000)
libunwind-dynamic.so => pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libunwind-dynamic.so (0x00002b850cbec000)

make obj-intel64/opcodemix.so -j2

icc: warning #10315: specifying -lm before files may supersede the
Intel(R) math library and affect performance

g++ -Wall -Werror -Wno-unknown-pragmas -D__PIN__=1 -DPIN_CRT=1 -DTARGET_IA32E
-DHOST_IA32E -fPIC -DTARGET_LINUX -fabi-version=2
-I../../../source/include/pin -I../../../source/include/pin/gen -isystem
pin-3.7-97619-g0d0c92f4f-gcc-linux/extras/stlport/include -isystem
pin-3.7-97619-g0d0c92f4f-gcc-linux/extras/libstdc++/include -isystem
pin-3.7-97619-g0d0c92f4f-gcc-linux/extras/crt/include -isystem
pin-3.7-97619-g0d0c92f4f-gcc-linux/extras/crt/include/arch-x86_64 -isystem
pin-3.7-97619-g0d0c92f4f-gcc-linux/extras/crt/include/kernel/uapi -isystem
pin-3.7-97619-g0d0c92f4f-gcc-linux/extras/crt/include/kernel/uapi/asm-x86
-I../../../extras/components/include -I../../../extras/xed-intel64/include/xed
-I../../../source/tools/InstLib -O3 -fomit-frame-pointer -fno-strict-aliasing
-c -o obj-intel64/opcodemix.o opcodemix.cpp

g++ -shared ../../../intel64/runtime/pincrt/crtbeginS.o -Wl,-Bsymbolic
-Wl,--version-script=../../../source/include/pin/pintool.ver -fabi-version=2
-o obj-intel64/opcodemix.so obj-intel64/opcodemix.o
../../../source/tools/InstLib/obj-intel64/controller.a
-L../../../intel64/runtime/pincrt -L../../../intel64/lib
-L../../../intel64/lib-ext -L../../../extras/xed-intel64/lib -lpin -lxed
../../../intel64/runtime/pincrt/crtendS.o -lpin3dwarf  -ldl-dynamic -nostdlib
-lstlport-dynamic -lm-dynamic -lc-dynamic -lunwind-dynamic

Maybe an additional compiler flag could help resolving __cxxabiv1::__si_class_type_info ?

svalat commented 5 years ago

Yes I also have similar (not exact details) issues with pin on last ubuntu release :(.

Still looking on a work arround. I will give a look on SLES when I can, this might give me other hints.

This is why I want to find a way to make a dual port on DynamoRIO in addition to Pin which might avoid those binary issues. But don't have enouth time to make the port myself now. Hope one day....

jgphpc commented 5 years ago

It looks better with the Intel compiler instead of gcc:

make obj-intel64/opcodemix.so ICC=1 CCPATH=`which icc` CXXPATH=`which icpc`

../../../pin -t obj-intel64/opcodemix.so -- /bin/ls

cat opcodemix.out

# $static-counts
#
#     opcode       count-unpredicated    count-predicated
#
   5 ADC                           61                0
   8 ADD                        13981                0
  11 ADDSD                          3                0
  12 ADDSS                         13                0
  15 ADD_LOCK                       9                0
...
svalat commented 3 years ago

This should be fixed by support of last version of pintool.