Assertion Failure when Running Metadata Stress with MPI

[zhanghb55]

Hello,

I encountered an issue while running Metadata Stress from h5bench-1.4. After installation, I executed ./h5bench_hdf5_iotest hdf5_iotest.ini and received the following output:

Config loaded from 'hdf5_iotest.ini':
    steps=20, arrays=500, rows=100, columns=200, scaling=weak
    proc-grid=1x1, slowest-dimension=step, rank=4
    layout=contiguous, mpi-io=independent

Wall clock [s]:     1.95
File size [B]:      1600002048
---------------------------------------------
Measurement:        _MIN (over MPI ranks)
            ^MAX (over MPI ranks)
---------------------------------------------
Write phase [s]:    _1.51
            ^1.51
Create time [s]:    _0.00
            ^0.00
Write time [s]:     _1.50
            ^1.50
Write rate [MiB/s]: _1019.80
            ^1019.80
Read phase [s]:     _0.31
            ^0.31
Read time [s]:      _0.30
            ^0.30
Read rate [MiB/s]:  _5133.93
            ^5133.93

I attempted to use MPI to execute this benchmark with the command mpirun -n 4 ./h5bench_hdf5_iotest hdf5_iotest.ini, but I encountered an error:

h5bench_hdf5_iotest: /home/zhb/h5bench-1.4/metadata_stress/configuration.c:156: validate: Assertion `pconfig->proc_rows * pconfig->proc_cols == (unsigned)size' failed.
[ubuntu:257356] *** Process received signal ***
[ubuntu:257356] Signal: Aborted (6)
[ubuntu:257356] Signal code:  (-6)
[ubuntu:257356] [ 0] /lib64/libpthread.so.0(+0xf5d0)[0x2ac3162125d0]
[ubuntu:257356] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2ac316455207]
[ubuntu:257356] [ 2] /lib64/libc.so.6(abort+0x148)[0x2ac3164568f8]
[ubuntu:257356] [ 3] /lib64/libc.so.6(+0x2f026)[0x2ac31644e026]
[ubuntu:257356] [ 4] /lib64/libc.so.6(+0x2f0d2)[0x2ac31644e0d2]
[ubuntu:257356] [ 5] ./h5bench_hdf5_iotest[0x404f6a]
[ubuntu:257356] [ 6] ./h5bench_hdf5_iotest[0x40244d]
[ubuntu:257356] [ 7] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2ac3164413d5]
[ubuntu:257356] [ 8] ./h5bench_hdf5_iotest[0x402259]
[ubuntu:257356] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node ubuntu exited on signal 6 (Aborted).
--------------------------------------------------------------------------

It appears that there may be a configuration error. However, my configuration follows the guidelines provided in the documentation:

[DEFAULT]
version = 0
steps = 20
arrays = 500
rows = 100
columns = 200
process-rows = 1
process-columns = 1
scaling = weak
dataset-rank = 4
slowest-dimension = step
layout = contiguous
mpi-io = independent
hdf5-file = hdf5_iotest.h5
csv-file = hdf5_iotest.csv

Could you please assist me in resolving this issue?

[jeanbez]

@zhanghb55, you should not invoke h5bench_hdf5_iotest directly, but rather through the main h5bench executable and configuration file. Regarding the error, the assertion is related to the invalid configuration (pconfig->proc_rows * pconfig->proc_cols == (unsigned)size). You are running with 4 processes (size) but with 100 rows and 200 columns.

[jeanbez]

Let me know if that does not work and we can re-open this issue.

[zhanghb55]

Unfortunately, despite following your suggested approach, I'm still encountering the same issue as before. My current configuration, which is strictly copied from the repository with the exception of retaining only the metadata test, is as follows:

{
    "mpi": {
        "command": "mpirun",
        "ranks": "4",
        "configuration": "-np 8 --oversubscribe"
    },
    "vol": {
        "library": "/vol-async/src:/hdf5-async-vol-register-install/lib:/argobots/install/lib:/hdf5-install/install:",
        "path": "/vol-async/src",
        "connector": "async under_vol=0;under_info={}"
    },
    "file-system": {
        "lustre": {
            "stripe-size": "1M",
            "stripe-count": "4"
        }
    },
    "directory": "full-teste",
    "benchmarks": [
        {
            "benchmark": "metadata",
            "file": "hdf5_iotest.h5",
            "configuration": {
                "version": "0",
                "steps": "20",
                "arrays": "500",
                "rows": "100",
                "columns": "200",
                "process-rows": "2",
                "process-columns": "2",
                "scaling": "weak",
                "dataset-rank": "4",
                "slowest-dimension": "step",
                "layout": "contiguous",
                "mpi-io": "independent",
                "csv-file": "hdf5_iotest.csv"
            }
        }
    ]
}

The corresponding error is as follows:

$ cat full-teste/a992698e/stderr 
h5bench_hdf5_iotest: /home/zhb/benchmark/h5bench-1.4/metadata_stress/configuration.c:156: validate: Assertion `pconfig->proc_rows * pconfig->proc_cols == (unsigned)size' failed.
[gpu13:225176] *** Process received signal ***
[gpu13:225176] Signal: Aborted (6)
[gpu13:225176] Signal code:  (-6)
[gpu13:225176] [ 0] /lib64/libpthread.so.0(+0xf5d0)[0x2ac25f2245d0]
[gpu13:225176] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2ac25f467207]
[gpu13:225176] [ 2] /lib64/libc.so.6(abort+0x148)[0x2ac25f4688f8]
[gpu13:225176] [ 3] /lib64/libc.so.6(+0x2f026)[0x2ac25f460026]
[gpu13:225176] [ 4] /lib64/libc.so.6(+0x2f0d2)[0x2ac25f4600d2]
[gpu13:225176] [ 5] /home/zhb/benchmark/h5bench-1.4/install/bin//h5bench_hdf5_iotest[0x404f6a]
[gpu13:225176] [ 6] /home/zhb/benchmark/h5bench-1.4/install/bin//h5bench_hdf5_iotest[0x40244d]
[gpu13:225176] [ 7] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2ac25f4533d5]
[gpu13:225176] [ 8] /home/zhb/benchmark/h5bench-1.4/install/bin//h5bench_hdf5_iotest[0x402259]
[gpu13:225176] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node gpu13 exited on signal 6 (Aborted).
--------------------------------------------------------------------------

I also attempted using mpi -np 2, but the outcome remained identical.

It appears I may have misunderstood the documentation and your instructions, and I'm at a loss for how to proceed. Could you please provide further assistance?

[zhanghb55]

Upon reading the code, I found that it's not enough to merely duplicate the configuration file. It's essential to validate the configuration file as well. To be precise, the multiplication of 'process-rows' and 'process-columns' should equal the overall process count.