search

mercury-hpc / mercury

Mercury is a C library for implementing RPC, optimized for HPC.
http://www.mcs.anl.gov/projects/mercury/
BSD 3-Clause "New" or "Revised" License
163 stars 60 forks source link

gperftools and mercury interaction #409

Closed roblatham00 closed 3 years ago

roblatham00 commented 3 years ago

Describe the bug

mercury's progress engine gets terribly confused when google perftools loaded

To Reproduce Steps to reproduce the behavior:

export CPUPROFILE=/gpfs/alpine/scratch/robl/csc332/testing/ior.prof
mpiexec -f clients -launcher ssh -prepend-rank -ppn $((22)) -n $((CLIENTS*10)) --env LD_PRELOAD=$(spack location -i gperftools)/lib/libprofiler.so ${WORKDIR}/./src/ior

this mpi program fails with the following error stack:

[0] # HG -- Error -- /tmp/robl/spack-stage/spack-stage-mercury-2.0.0rc2-uqqexcg5hwzwcdi5vaiinswfegc7mura/spack-src/src/mercury_core.c:3814
[0]  # hg_core_progress(): Invalid type of poll event (0)
[0] # HG -- Error -- /tmp/robl/spack-stage/spack-stage-mercury-2.0.0rc2-uqqexcg5hwzwcdi5vaiinswfegc7mura/spack-src/src/mercury_core.c:4994
[0]  # HG_Core_progress(): Could not make progress
[0] # HG -- Error -- /tmp/robl/spack-stage/spack-stage-mercury-2.0.0rc2-uqqexcg5hwzwcdi5vaiinswfegc7mura/spack-src/src/mercury.c:1996
[0]  # HG_Progress(): Could not make progress on context (HG_INVALID_ARG)
[0] WARNING: unexpected return code (11) from HG_Progress()

I know i've used the google perftools before but it has been a while. I don't know yet if this is summit specific. Expected behavior

Was hoping to collect performance profiles for a mochi service

Platform (please complete the following information):

Additional context Add any other context about the problem here.

soumagne commented 3 years ago

@roblatham00 This was fixed with latest release with #392. You're not supposed to use rc2 anymore :)