Running the H5Bench sync-write-read-contig-1d-small benchmark at some scales results in the Mercury timeout:
margo_forward_timed() failed - HG_TIMEOUT
Increasing UNIFYFS_MARGO_CLIENT_TIMEOUT fixed two different tests having this issue, but seems to have no effect on this one.
This error occurs using 1N/1n, 2N/2n, and 8N/32n.
However, this same test passes using 4N/8n and 16N/128n without needing to increase UNIFYFS_MARGO_CLIENT_TIMEOUT at all.
Error occurs on Frontier, haven't attempted to reproduce on other systems yet.
Describe how to reproduce the problem
Same h5bench config will work for both write and read:
ml gcc/12.2.0
ml PrgEnv-gnu
ml cray-hdf5-parallel/1.12.2.1
module unload darshan-runtime
module use /sw/frontier/unifyfs/modulefiles
module load unifyfs/1.1/gcc-12.2.0
export UNIFYFS_LOG_VERBOSITY=5
Start UnifyFS and then run h5bench_write followed by h5bench_read:
Describe the problem you're observing
Running the H5Bench
sync-write-read-contig-1d-small
benchmark at some scales results in the Mercury timeout:Increasing
UNIFYFS_MARGO_CLIENT_TIMEOUT
fixed two different tests having this issue, but seems to have no effect on this one.This error occurs using 1N/1n, 2N/2n, and 8N/32n. However, this same test passes using 4N/8n and 16N/128n without needing to increase
UNIFYFS_MARGO_CLIENT_TIMEOUT
at all.Error occurs on Frontier, haven't attempted to reproduce on other systems yet.
Describe how to reproduce the problem
Same h5bench config will work for both write and read:
Set up environment and UnifyFS:
Start UnifyFS and then run
h5bench_write
followed byh5bench_read
:Include any warning or errors or releveant debugging data
Increased
UNIFYFS_MARGO_CLIENT_TIMEOUT
multiple times (up to 10 minutes as shown here), but still get the same error.Server log around same time as the
HG_TIMEOUT
: