Is sequential HDF5 supported?

[candiceT233]

Issue Description I'd like to know if h5bench supports building with the sequential HDF5.

This is what I have tried so far: my HDF5_PATH is built with only h5cc.

cd build

cmake .. \
    -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR \
    -DCMAKE_C_FLAGS="-I/$HDF5_PATH/include \
    -L/$HDF5_PATH/lib"
make 
make install

This is the error I am facing during installation:

-- h5bench baseline: ON
-- h5bench METADATA: OFF
-- h5bench EXERCISER: OFF
-- h5bench AMREX: OFF
-- h5bench OPENPMD: OFF
-- h5bench E3SM: OFF
-- h5bench MACSIO: OFF
-- Found HDF5: hdf5-shared (found version "1.14.0")  
-- Using HDF5 version: 1.14.0
-- Looking for H5_HAVE_SUBFILING_VFD
-- Looking for H5_HAVE_SUBFILING_VFD - not found
--    Detected HDF5 subfiling support: 
-- HDF5 VOL ASYNC: OFF
-- Found Python3: /share/apps/python/miniconda3.7/bin/python3.7 (found version "3.7.7") found components: Interpreter 
-- Configuring done
-- Generating done
-- Build files have been written to: /MY_PATH/scripts/vlen_workflow/h5bench/build
[  8%] Building C object CMakeFiles/h5bench_util.dir/commons/h5bench_util.c.o
[ 16%] Linking C static library libh5bench_util.a
[ 16%] Built target h5bench_util
[ 25%] Building C object CMakeFiles/h5bench_read.dir/h5bench_patterns/h5bench_read.c.o
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c: In function ‘set_dspace_plist’:
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:83:9: warning: implicit declaration of function ‘H5Pset_dxpl_mpio’; did you mean ‘H5Pset_fapl_stdio’? [-Wimplicit-function-declaration]
   83 |         H5Pset_dxpl_mpio(*plist_id_out, H5FD_MPIO_COLLECTIVE);
      |         ^~~~~~~~~~~~~~~~
      |         H5Pset_fapl_stdio
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c: In function ‘read_h5_data’:
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:97:5: warning: implicit declaration of function ‘H5Pset_all_coll_metadata_ops’ [-Wimplicit-function-declaration]
   97 |     H5Pset_all_coll_metadata_ops(dapl, true);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c: In function ‘set_pl’:
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:397:9: warning: implicit declaration of function ‘H5Pset_fapl_mpio’; did you mean ‘H5Pset_fapl_stdio’? [-Wimplicit-function-declaration]
  397 |         H5Pset_fapl_mpio(*fapl, MPI_COMM_WORLD, MPI_INFO_NULL);
      |         ^~~~~~~~~~~~~~~~
      |         H5Pset_fapl_stdio
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:397:33: error: ‘MPI_COMM_WORLD’ undeclared (first use in this function)
  397 |         H5Pset_fapl_mpio(*fapl, MPI_COMM_WORLD, MPI_INFO_NULL);
      |                                 ^~~~~~~~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:397:33: note: each undeclared identifier is reported only once for each function it appears in
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:397:49: error: ‘MPI_INFO_NULL’ undeclared (first use in this function)
  397 |         H5Pset_fapl_mpio(*fapl, MPI_COMM_WORLD, MPI_INFO_NULL);
      |                                                 ^~~~~~~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:401:5: warning: implicit declaration of function ‘H5Pset_coll_metadata_write’ [-Wimplicit-function-declaration]
  401 |     H5Pset_coll_metadata_write(*fapl, true);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c: In function ‘main’:
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:415:5: warning: implicit declaration of function ‘MPI_Init_thread’ [-Wimplicit-function-declaration]
  415 |     MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &mpi_thread_lvl_provided);
      |     ^~~~~~~~~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:415:35: error: ‘MPI_THREAD_MULTIPLE’ undeclared (first use in this function)
  415 |     MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &mpi_thread_lvl_provided);
      |                                   ^~~~~~~~~~~~~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:417:5: warning: implicit declaration of function ‘MPI_Comm_rank’ [-Wimplicit-function-declaration]
  417 |     MPI_Comm_rank(MPI_COMM_WORLD, &MY_RANK);
      |     ^~~~~~~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:417:19: error: ‘MPI_COMM_WORLD’ undeclared (first use in this function)
  417 |     MPI_Comm_rank(MPI_COMM_WORLD, &MY_RANK);
      |                   ^~~~~~~~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:418:5: warning: implicit declaration of function ‘MPI_Comm_size’ [-Wimplicit-function-declaration]
  418 |     MPI_Comm_size(MPI_COMM_WORLD, &NUM_RANKS);
      |     ^~~~~~~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:513:5: error: unknown type name ‘MPI_Info’
  513 |     MPI_Info info = MPI_INFO_NULL;
      |     ^~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:513:21: error: ‘MPI_INFO_NULL’ undeclared (first use in this function)
  513 |     MPI_Info info = MPI_INFO_NULL;
      |                     ^~~~~~~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:519:5: warning: implicit declaration of function ‘MPI_Barrier’ [-Wimplicit-function-declaration]
  519 |     MPI_Barrier(MPI_COMM_WORLD);
      |     ^~~~~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:521:5: warning: implicit declaration of function ‘MPI_Allreduce’ [-Wimplicit-function-declaration]
  521 |     MPI_Allreduce(&NUM_PARTICLES, &TOTAL_PARTICLES, 1, MPI_LONG_LONG, MPI_SUM, MPI_COMM_WORLD);
      |     ^~~~~~~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:521:56: error: ‘MPI_LONG_LONG’ undeclared (first use in this function)
  521 |     MPI_Allreduce(&NUM_PARTICLES, &TOTAL_PARTICLES, 1, MPI_LONG_LONG, MPI_SUM, MPI_COMM_WORLD);
      |                                                        ^~~~~~~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:521:71: error: ‘MPI_SUM’ undeclared (first use in this function)
  521 |     MPI_Allreduce(&NUM_PARTICLES, &TOTAL_PARTICLES, 1, MPI_LONG_LONG, MPI_SUM, MPI_COMM_WORLD);
      |                                                                       ^~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:522:5: warning: implicit declaration of function ‘MPI_Scan’ [-Wimplicit-function-declaration]
  522 |     MPI_Scan(&NUM_PARTICLES, &FILE_OFFSET, 1, MPI_LONG_LONG, MPI_SUM, MPI_COMM_WORLD);
      |     ^~~~~~~~
/MY_PATH/scripts/vlen_workflow/h5bench/h5bench_patterns/h5bench_read.c:634:5: warning: implicit declaration of function ‘MPI_Finalize’ [-Wimplicit-function-declaration]
  634 |     MPI_Finalize();
      |     ^~~~~~~~~~~~
make[2]: *** [CMakeFiles/h5bench_read.dir/h5bench_patterns/h5bench_read.c.o] Error 1
make[1]: *** [CMakeFiles/h5bench_read.dir/all] Error 2
make: *** [all] Error 2

Software Environment

• version of h5bench: 1.4
• installed h5bench using: from source
• operating system: CentOS7
• machine: supercomputer
• version of HDF5: 1.14.0

[jeanbez]

Hi! h5bench benchmark and kernels require parallel HDF5 support.

[candiceT233]

Hi! h5bench benchmark and kernels require parallel HDF5 support.

@jeanbez Thank you. I am doing a research project and I want to run the H5Bench with the Hermes VFD. Currently, Hermes only supports sequential HDF5. I'd like to know if it's doable, with not too many code modification on the h5bench_read.c and h5bench_write.c files, I can compile H5Bench and run it with sequential the HDF5?

[jeanbez]

It should be doable, but there will be quite some code changes I would say. You would have to drop all MPI-related functions and adapt the problem accordingly.

[sbyna]

Is there a specific pattern of h5bench you are trying to use?

Instead of trying to convert h5bench to a sequential version, it may be good to use some existing hdf5 sequential examples. Some of them are available at https://github.com/HDFGroup/hdf5/tree/develop/examples https://github.com/HDFGroup/hdf5/blob/develop/examples/h5_select.c may be something to define a hyperslab and read, which can be modified to mimic patterns for reading full dataset or a subset of a dataset.

[candiceT233]

Thank you for the help. I can successfully compile and run the h5bench_read.cc and h5bench_write.cc code with sequential HDF5. What I did is comment out all the MPI-related commands, and set the below variables:

NUM_RANKS = 1
TOTAL_PARTICLES = NUM_PARTICLES

and change H5Pset_fapl_mpio(fapl, comm, info) to H5Pset_fapl_sec2(fapl).