zbeekman
commented
7 years ago also, I tried removing the craype-hugepages2M module because I saw the runtime warning/error but then compilation fails.
Open zbeekman opened 7 years ago
zbeekman
commented
7 years ago also, I tried removing the craype-hugepages2M module because I saw the runtime warning/error but then compilation fails.
nchaimov
commented
7 years ago In the debug output from 2.26.3, you're not using sampling, you're using source instrumentation:
[DEBUG] Configuring experiment Onyx-serialPi-profile
Can you confirm that this worked with 2.26.3 with sampling rather than profiling?
zbeekman
commented
7 years ago I could have sworn that that I had selected sampling. I'll confirm again whether or not each work with tau-nightly and 2.26.3, stay tuned...
nchaimov
commented
7 years ago Assuming that it doesn't work in either 2.26.3 or nightly, what I think is wrong is that TAU Commander is linking the executable against TAU by using the TAU compiler wrappers with -optLinkOnly but nothing ever calls TAU. -optLinkOnly would work fine if the application used any libraries that TAU has wrappers for (MPI, pthreads, etc.), but since serialPi.cpp is serial, nothing in the application ever calls anything in TAU, so the symbols are unused and not included (and even if they were, without a use of a wrapper, nothing would ever call TAU_INIT, so sampling would not start.)
At least, that is my understanding of what happens with -optLinkOnly. I could be wrong.
zbeekman
commented
7 years ago Ah, interesting, so if I run mm under MPI then it will likely work, if that hypothesis is true. I could swear I've used sampling on a serial code, but it was a Fortran only code and I think I enable IO profiling too.
On Tue, Nov 7, 2017 at 3:41 PM Nicholas Chaimov notifications@github.com wrote:
Assuming that it doesn't work in either 2.26.3 or nightly, what I think is wrong is that TAU Commander is linking the executable against TAU by using the TAU compiler wrappers with -optLinkOnly but nothing ever calls TAU. -optLinkOnly would work fine if the application used any libraries that TAU has wrappers for (MPI, pthreads, etc.), but since serialPi.cpp is serial, nothing in the application ever calls anything in TAU, so the symbols are unused and not included (and even if they were, without a use of a wrapper, nothing would ever call TAU_INIT, so sampling would not start.)
At least, that is my understanding of what happens with -optLinkOnly. I could be wrong.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ParaToolsInc/taucmdr/issues/236#issuecomment-342615077, or mute the thread https://github.com/notifications/unsubscribe-auth/AAREPBeZtlSv4mqu3Om24AoSsBMPFm-Pks5s0MAygaJpZM4QU0sY .
jlinford
commented
7 years ago Oooooh this reminds me, I found this issue over a year ago. Sameer told me it's a feature, not a bug:
On 4/21/16:
Hi Sameer,
I discovered that I can force the link by replacing -ltau-intel with -Wl,--whole-archive,-ltau-intel,--no-whole-archive. No nm -A shows TAU symbols in the executable, but setting TAU environment variables still has no effect:
nm -A ./a.out | grep TAU
./a.out:00000000008e2d54 D TAU_ALARM_TYPE
./a.out:0000000000419b00 T TAU_ALLOC
./a.out:0000000000418ef0 T TAU_CONTEXT_EVENT
./a.out:0000000000419e70 T TAU_DB_DUMP
./a.out:0000000000419e60 T TAU_DB_DUMP_PREFIX
./a.out:0000000000418be0 T TAU_DB_PURGE
./a.out:0000000000419c60 T TAU_DEALLOC
./a.out:0000000000418c60 T TAU_DISABLE_ALL_GROUPS
.......
export TAU_SAMPLING=1
export TAU_VERBOSE=1
./a.out
# No profiles, no TAU messages.
What should I try next? Thanks,
~John C.
Hi John,
Yes, we can say that they need to build dynamic if they want to sample uninstrumented serial code. For MPI codes, it is possible to link in the MPI wrapper and so we can instrument and sample. Looks like a bug with -optTrackIO with fwrite wrapper.
Thanks,
- Sameer@nchaimov definitely happens with tau 2.26.3, and judging from John's comment the best we can do is add a compat entry with Measurement.discourage (or exclude maybe?) when linkage is static and no MPI, IO, etc. is turned on. I may leave this open as a reminder to add a Measurement.discourage in the compat. (Actually it will have to be a custom callback checking heaps of things. I'm very tempted to just close this issue and walk quietly away...)
nchaimov
commented
7 years ago @zbeekman, yes it should work with MPI (so long as the application actually calls MPI_Init). Otherwise with a serial code we'd have to dynamically link it and then run it through tau_exec.
Enabling IO wrapping will work too, as that will cause the TAU symbols to be included and the IO wrapper initializes TAU when it intercepts the first call.
As @jlinford points out above, we can force the linker to include the symbols, but something still has to initialize TAU for TAU to do anything. With tau_exec, the LD_PRELOADed libTAU-preload.so initializes TAU when it is loaded by the dynamic linker as it has a constructor function, but I'm not sure if there's a way to do something equivalent with a static executable.
EDIT (Tue Nov 7 22:02:06 UTC 2017): See John's comment below. This is intentional tau behavior. The best case scenario, at least for now, is to warn users about it. Leaving this issue open as a reminder to implement the warning.
I'm trying with the examples/pi directory. With stable tau (2.26.3) at compile time I get:
incompatible redefinition of macro "TAU_ENABLED"(here is some debug output) BUT, at leastnmreports tau symbols (nm serialPi | grep -i tau)When using tau-nightly, as of this morning, the error goes away, but no profile data is produced, AND
nm serialPi | grep -i tauproduces NO output, so there are no tau symbols.Here is the output when running the example using tau nightly, but neither stable tau nor nightly . produce profiles:
Here are my loaded modules: