Closed gentryx closed 8 years ago
Does this happen with other allocators as well?
Also, how can I reproduce that?
Steps to reproduce:
git clone https://github.com/STEllAR-GROUP/hpx.git
mkdir hpx/build
git clone https://github.com/gentryx/libgeodecomp
mkdir libgeodecomp/build
cmake -DCMAKE_INSTALL_PREFIX=$HOME/test_build_libgeodecomp_intel_mic/install -DHPX_WITH_PARCELPORT_MPI=true -DHPX_WITH_PARCELPORT_TCP=true -DCMAKE_CXX_COMPILER=g++-5.3.0 ../
make -j20
make install
cd ../../libgeodecomp/build/
cmake -DCMAKE_CXX_COMPILER=g++-5.3.0 -DWITH_CUDA=false -DWITH_HPX=true -DWITH_FORTRAN=true -DHPX_IGNORE_COMPILER_COMPATIBILITY=true -DCMAKE_Fortran_COMPILER=gfortran-5.3.0 -DWITH_CUDA=false -DWITH_CPP14=true -DCMAKE_PREFIX_PATH=$HOME/test_build_libgeodecomp_intel_mic/install ../
make -j16
./src/testbed/hpxperformancetests/hpxperformancetests --hpx:threads=16 0 0
I did bisect this and git tells me:
2bf3b80ce493de23bd9fa60c056ca99dbe19b587 is the first bad commit
commit 2bf3b80ce493de23bd9fa60c056ca99dbe19b587
Author: Agustin K-ballo Berge <k@fusionfenix.com>
Date: Tue Mar 29 20:26:50 2016 -0300
Avoid unnecessary relocking when waiting on a detail::cv, fix missing #include ripples
:040000 040000 8d79ddc61b54f0c7dcd22025d5fff806e7ff9817 a0cba1633367367adc4f3da2e4c6f59cbc909444 M hpx
:040000 040000 178fd74655d63e2ae8a5bbd71cd92791b6f1b155 84c3b8f3038d8f4be7ced144906f20e78d865d56 M src
I'll check another allocator in a bit.
@K-ballo have you seen this?
@hkaiser I've seen it now. I do not have an explanation for the race condition.
@gentryx A shot from the hip: could you try whether removing the lock.unlock()
helps here: https://github.com/STEllAR-GROUP/hpx/blob/master/src/lcos/local/detail/condition_variable.cpp#L83, please?
@hkaiser Changing that line did not fix the race, but it did reduce the probability a bit (tests would sometimes (30%) succeed):
I did also test different allocators: jemalloc fails with current trunk, system succeeds (maybe because it's so slow?).
@gentryx What happens if you use top of master with the commit reverted you identified? I still think the changes applied by that commit are correct. They do however change overall timings and thread execution sequencing, which may make the race to become visible.
@gentryx I tried to reproduce this on my system yesterday. Unfortunately everything works as expected :/ I'll keep trying, though.
I'm closing this since the race seems to have been resolved on master. Thanks!
@gentryx The funny thing is that we have not done anything to fix it...
@hkaiser I assume you also didn't do anything to cause it, so apparently this race condition just went back to lurking beneath the surface. I could only reproduce it on one of our test machines anyway.
Reopening as this race resurfaced, this time on more machines. Good thing: I can reproduce it on the Marvin nodes. I'll add a script for bug reproduction momentarily.
`[01:43:41]:aschafer@deneb01.hermione:/home/aschafer:0:$ cat test.sh
set -e
cd $HOME rm -f .cmake/packages/HPX/*
mkdir test_race_lgd_hpx cd test_race_lgd_hpx git clone https://github.com/STEllAR-GROUP/hpx.git git clone https://github.com/stellar-group/libgeodecomp
mkdir hpx/build cd hpx/build cmake -DHPX_WITH_EXAMPLES=false -DHPX_WITH_PARCELPORT_MPI=true -DHPX_WITH_PARCELPORT_TCP=true -DCMAKE_INSTALL_PREFIX=$HOME/test_race_lgd_hpx/local_install -DHPX_WITH_CXX14=false -DHPX_WITH_MALLOC=system -DBOOST_ROOT=/opt/boost/1.57.0-release/ .. make -j40 make install cd ../../
mkdir libgeodecomp/build cd libgeodecomp/build cmake -DWITH_CUDA=false -DWITH_CPP14=true -DWITH_HPX=true -DBOOST_ROOT=/opt/boost/1.57.0-release/ .. make -j40
for ((i=0;i<10;++i)); do ./src/testbed/hpxperformancetests/hpxperformancetests --hpx:threads=32 0 0 done
[01:43:45]:aschafer@deneb01.hermione:/home/aschafer:0:$ sbatch -p marvin --exclusive test.sh Submitted batch job 1714003 `
@gentryx could you try the fix_wait_all (see #2165) branch and see if that fixes your issue?
@sithhell I can confirm that my issue is not present on the fix_wait_all branch. :-)
I'm certain this fixes not only your issue but a couple of similarly dubious problems we've been seeing over the last months.
This can be closed now as #2165 has been merged.
I'm seeing an invalid free error when running the LibGeoDecomp performance tests with recent HPX commits. The backtrace below was generated with commit ID 6f79ea91120f4c4212b07dea92cf4d6de299facc. It didn't occur with the same LibGeoDecomp code and the HPX trunk approx. 4 weeks ago. The invalid free goes away if I sprinkle the performance test with printf, so I assume it's a race condition and not a normal invalid free.
I can provide more details if necessary.
Error:
Backtrace: