Library Using HDF5 Has Failures When Using Vol Rest

[ron-kuhn]

I have a library that using HDF5 internally to save hierarchical big data. All works fine when using using local HDF5 files on local drive. When I try to use HSDS and the vol-rest, I am having problems. When I use a tool to copy the HDF5 file to HSDS, everything appears to work. Here are the current problems:

• Variable length strings are not supported for datasets or attributes. Is there plans to support in the future? Should I just avoid using variable length strings (i.e. performance reasons?)? I converted all to fixed length string as a workaround and that appears to work.
• I am getting an error trying to create a dataset. The hs.log file gives the following warning "Unexpected layout: H5D_CONTIGUOUS". The compound datatype contains: STD_I32LE, IEEE_F64LE and IEEE_F64LE.
• The library creates an HDF5 file and multiple child HDF5 files connected by external links. When the library attempts to create a child HDF5 file, it gets the directory of the parent file by using H5Fget_name on the group containing the external link. This is working for files. Is there something I can use when using REST VOL?

I'm sure there will be others once these are fixed.

[ron-kuhn]

Converted the dataset in second bullet above to an attribute which is better anyways. Now the HSDS log file fails to write out the attribute with the following message Expected: "40 bytes, but got 48". Datatype is STD_I32LE, IEEE_F64LE, IEEE_F64LE with dim 2. So (4+8+8)*2=40. Not sure why 48?

[ron-kuhn]

Fixed the 48 problem by adding pragma pack(push 1) around my structures making up the HDF5 types. This wasn't required by file-based storage as long as the offsets were correct. Why is this required?

[ron-kuhn]

As mentioned above, I have a parent HDF5 file that external links to other children HDF5 files. Here is the workflow:

• Create the child HDF5 file
• Create a group (/Images) from the root
• Create an external link in the parent to the child file (/Images) group
• When I try to create another group in the child file off of /images, I get an access violation exception on the following line:

rest_vol_group.c line 164 host_header_len = strlen(parent->domain->u.file.filepath_name) + strlen(host_string) + 1;

Is there a problem with multiple HDF5 files open in the same thread using HSDS? Again, this worked with files. All files are in the same MinIO bucket.

[jhendersonHDF]

Hi @ron-kuhn,

Variable length strings are not supported for datasets or attributes. Is there plans to support in the future? Should I just avoid using variable length strings (i.e. performance reasons?)? I converted all to fixed length string as a workaround and that appears to work.

Yes, we'd like to eventually support variable-length data, though we don't currently have a timeline for that feature. IIRC, at the time the REST VOL was originally written we were still deciding what a variable-length data stream should look like for HSDS. Now that HSDS supports this, we should be able to support it in the REST VOL as well.

Fixed the 48 problem by adding pragma pack(push 1) around my structures making up the HDF5 types. This wasn't required by file-based storage as long as the offsets were correct. Why is this required?

In this case, HSDS is expecting the compound datatype data exactly as the compound type is constructed, without any support for possible padding. Though we'd like to support this in the future, the REST VOL currently doesn't support datatype conversions, so it can't appropriately pack a compound datatype before sending it to the server; it will just send off the data buffer it's given for writing, which will include any possible padding in the C struct being written.

The library creates an HDF5 file and multiple child HDF5 files connected by external links. When the library attempts to create a child HDF5 file, it gets the directory of the parent file by using H5Fget_name on the group containing the external link. This is working for files. Is there something I can use when using REST VOL?

H5Fget_name should be supported by the REST VOL. If you encounter an issue with the same workflow when using the REST VOL, please open an issue so that we can look into that.

Is there a problem with multiple HDF5 files open in the same thread using HSDS? Again, this worked with files. All files are in the same MinIO bucket.

The REST VOL can handle multiple open files correctly, but is not extensively tested with external links so it's likely that you've run into a bug with the way that access to externally-linked files is handled by the connector.

[ron-kuhn]

Though we'd like to support this in the future, the REST VOL currently doesn't support datatype conversions,

One of the benefits of HDF5 is that it support multi-platform (big and little endian). So this isn't possible with the VOL? It doesn't even error out? I get big images from HDF5 and wanted to be able to get specific parts of the image (save time not sending whole image over REST). Is this possible? Where can I find a list of the supported features?

H5Fget_name should be supported by the REST VOL. If you encounter an issue with the same workflow when using the REST VOL, please open an issue so that we can look into that.

I will debug this further and open an issue with specifics.

It's likely that you've run into a bug with the way that access to externally-linked files is handled by the connector.

I create the link in the parent, but work on the child file separately (not through the link from the parent) during this time. I will debug this also.

[ron-kuhn]

Is anyone using the VOL in a production capacity?

[jhendersonHDF]

One of the benefits of HDF5 is that it support multi-platform (big and little endian). So this isn't possible with the VOL? It doesn't even error out?

This isn't currently possible, but support for it can be implemented. See also the comment in https://github.com/HDFGroup/vol-rest/issues/6 by @jreadey. The REST VOL was developed when the VOL interface in HDF5 was still very young so there are missing features that have yet to be implemented, but the REST VOL can essentially support exactly what HSDS can.

I get big images from HDF5 and wanted to be able to get specific parts of the image (save time not sending whole image over REST). Is this possible?

The REST VOL should be able to support this since HSDS supports selections on datasets, but there's currently an issue if you try to request too large of a dataspace selection which we hope to address in the future.

Where can I find a list of the supported features?

The connector has a User Guide under doc/users_guide.pdf which mentions the supported HDF5 features toward the end of the document. Though note that datatype conversions aren't explicitly called out as unsupported.

Is anyone using the VOL in a production capacity?

Not as far as I'm currently aware. The REST VOL should not currently be considered to be production ready, but it is back under development again and so some of the missing features mentioned in this thread will hopefully get implemented.

[ron-kuhn]

Thanks, good to know