DAOS incompatibility: NetCDF assumes POSIX files, Native HDF5 files and Native HDF5 APIs

[brtnfld]

• the version of the software with which you are encountering an issue master
• environmental information (i.e., Operating System, compiler info, java version, python version, etc.)

New DAOS [1] testbed.

[1] https://www.intel.com/content/www/us/en/high-performance-computing/daos-high-performance-storage-brief.html

• a description of the issue with the steps needed to reproduce it

(1) Update H5Gget_objinfo and H5Dvlen_reclaim to newer HDF5 APIs

ISSUE: Incompatibility with native specific HDF5 APIs and the DAOS VOL connector and NetCDF.

Both the H5Gget_objinfo and H5Dvlen_reclaim are native HDF5 specific APIs and will not work with the DAOS VOL connector. They should be replaced with H5Oget_info and H5Treclaim.

(2) Netcdf in NC_infermodel assumes that the file is POSIX, which is not the case with DAOS. There needs to be a test added using H5Fis_accessible to determine if the HDF5 file is a DAOS object and if so, then it bypasses all the POSIX specific tests.

[DennisHeimbigner]

Would H5F_IS_HDF5 work with DAOS?

[edwardhartnett]

Perhaps a little background would be helpful.

@brtnfld is working for the HDF5 group, and wants to open up some of the HDF5 1.12.0 functionality to netcdf-4, which is great.

What he is working on here is access to DAOS, which, similarly to opendap, has a URL-like file name. But when such a file name is presented, netcdf assumes itt's an opendap file.

So what Scott is proposing here is an extra check at file open time, to see if the filename is actually a DAOS file, in which case HDF5, not opendap, would be used.

Scott, have I got it right? Can you give us an example of what the DAOS filename looks like?

[brtnfld]

For the most part, the DAOS filename follows the standard POSIX conventions. You can give it a path and a filename, but not a URL-like filename. Hence, it is not necessarily obvious from the filename alone that it is a DAOS object. Currently, you have to use HDF5 to read anything from the file (or get any information about the "file"), you can't use POSIX APIs. H5F_is_hdf5 does not work for a DAOS object, that is why we introduced H5Fis_accessible.

Thanks @edhartnett for providing the background summary.

[DennisHeimbigner]

Ok, so H5is_accessible and those other functions are 1.12.x and later, if I understand correctly. So assuming we have test for the version, I can fix our code to properly use H5is_accessible. That will fix part of the problem.

[brtnfld]

I also plan on submitting a pull request soon to address some of these issues.

[DennisHeimbigner]

Ok, but remember that our library has to continue to work with older versions of HDF5. So you should test your fixes with an older version.