search

mannau / h5

Interface to the HDF5 Library
Other
70 stars 22 forks source link

h5attr fails with large (~100 KB) values #62

Open jan-glx opened 7 years ago

jan-glx commented 7 years ago 
library(h5)
file <- h5file("test.h5")
h5attr(file, "bla") <- rep(8184)
# Error in FUN(.Object@pointer, attributename, dspace$typechar, dspace$dim,  : 
#   Creation of Attribute failed. Maybe attribute with same name is already existing at location.
h5attr(file, "bla") <- rep(8183)
# (no error)
ashiklom commented 6 years ago

Bumping because I get the same error. The following line appears in the HDF5 manual section on Attributes:

We generally consider the maximum size of an attribute to be 64K bytes. The library has two ways of storing attributes larger than 64K bytes: in dense attribute storage or in a separate dataset. Using dense attribute storage is described in this section, and storing in a separate dataset is described in the next section.

From the same document, looks like there's the following workaround:

To use dense attribute storage to store large attributes, set the number of attributes that will be stored in compact storage to 0 with the H5Pset_attr_phase_change function. This will force all attributes to be put into dense attribute storage and will avoid the 64KB size limitation for a single attribute in compact attribute storage.

But I guess in the meantime, since HDF5 is so flexible, it might make sense to just store large attributes in their own dataset.