search

ComputationalRadiationPhysics / libSplash

libSplash - Simple Parallel file output Library for Accumulating Simulation data using Hdf5
GNU Lesser General Public License v3.0
15 stars 15 forks source link

Evaluation: Speedup of Close #165

Open ax3l opened 9 years ago

ax3l commented 9 years ago

@psychocoderHPC found in a past evaluation paper (ref?), that the time for an H5Fclose can be reduced by about a factor 3 if the file-consistency check (H5F.c) is skipped in the HDF5 implementation.

Consequences for MPI-I/O output could be evaluated. It might be worth to consider adding a hook/option in the HDF5 API and to propose it for upcoming versions of the library.

There are also various close degrees available such as H5F_CLOSE_WEAK while the default for MPI-I/O is H5F_CLOSE_SEMI.

psychocoderHPC commented 9 years ago

Tuning HDF5 for Lustre File Systems (2010)

 ... However, when operating
in a parallel application, this operation resulted in a call
to MPI_File_set_size, which currently has very poor
performance characteristics on Lustre file systems. 1 Because
an HDF5 file’s size is not required to be accurately set until
the file is closed, this operation was removed from H5Fflush
and added to the code for closing a file.
ax3l commented 9 years ago

And the good old Parallel HDF5 Hints