search

Apollo3zehn / PureHDF

A pure .NET library that makes reading and writing of HDF5 files (groups, datasets, attributes, ...) very easy.
MIT License
55 stars 18 forks source link

Opaque datasets do not properly save when combined with attributes #79

Closed Blackclaws closed 7 months ago

Blackclaws commented 7 months ago

Two more problems I've encountered where I'm not sure whether it is a PureHDF or h5web problem:

var data = File.ReadAllBytes("/home/felix/screenshot.png");

var file = new H5File();
file["opaque"] = new H5Dataset(data, opaqueInfo: new H5OpaqueInfo((uint)data.Length, "test")); 

var opaqueGroup = new H5Group()
{
    Attributes =
    {
        {"Anything", "Data"}
    },
    ["stuff"] = new H5Group()
    {
     ["opaque"] = new H5Dataset(data, opaqueInfo: new H5OpaqueInfo((uint) data.Length, "test"))   
    }
};

file["group"] = opaqueGroup;

file.Write("testing.h5");

Writing this file (with a screenshot.png that is small enough that it doesn't directly break) one gets an error on opening the top group in h5web

testing.h5

group

stuff

opaque

opaque

group

Display

Inspect
HDF5-DIAG: Error detected in HDF5 (1.14.2) thread 0: #000: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5A.c line 1043 in H5Aread(): can't synchronously read data major: Attribute minor: Read failed #001: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5A.c line 1011 in H5A__read_api_common(): unable to read attribute major: Attribute minor: Read failed #002: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5VLcallback.c line 1236 in H5VL_attr_read(): attribute read failed major: Virtual Object Layer minor: Read failed #003: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5VLcallback.c line 1205 in H5VL__attr_read(): attribute read failed major: Virtual Object Layer minor: Read failed #004: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5VLnative_attr.c line 211 in H5VL__native_attr_read(): unable to read attribute major: Attribute minor: Read failed #005: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5Aint.c line 754 in H5A__read(): datatype conversion failed major: Attribute minor: Unable to encode value #006: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5T.c line 5308 in H5T_convert(): datatype conversion failed major: Datatype minor: Can't convert datatypes #007: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5Tconv.c line 3326 in H5T__conv_vlen(): can't read VL data major: Datatype minor: Read failed #008: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5Tvlen.c line 840 in H5T__vlen_disk_read(): unable to get blob major: Datatype minor: Can't get value #009: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5VLcallback.c line 7396 in H5VL_blob_get(): blob get failed major: Virtual Object Layer minor: Can't get value #010: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5VLcallback.c line 7367 in H5VL__blob_get(): blob get callback failed major: Virtual Object Layer minor: Can't get value #011: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5VLnative_blob.c line 123 in H5VL__native_blob_get(): Expected global heap object size does not match major: Virtual Object Layer minor: Unable to decode value

When leaving out the top opaque dataset:


var data = File.ReadAllBytes("/home/felix/screenshot.png");

var file = new H5File();

var opaqueGroup = new H5Group()
{
    Attributes =
    {
        {"Anything", "Data"}
    },
    ["stuff"] = new H5Group()
    {
     ["opaque"] = new H5Dataset(data, opaqueInfo: new H5OpaqueInfo((uint) data.Length, "test"))   
    }
};

file["group"] = opaqueGroup;

file.Write("testing.h5");

The dataset does not get written as opaque instead it gets written as Integer (unsigned), 8-bit, little-endian

Originally posted by @Blackclaws in https://github.com/Apollo3zehn/PureHDF/issues/76#issuecomment-2077317439

Blackclaws commented 7 months ago

Even more simple:

var data = File.ReadAllBytes("/home/felix/screenshot.png");

var file = new H5File()
{
    Attributes =
    {
        {"Anything", "Data"}
    },
     ["opaque"] = new H5Dataset(data, opaqueInfo: new H5OpaqueInfo((uint) data.Length, "test"))   
};

file.Write("testing.h5");

Already shows the issue.

Apollo3zehn commented 7 months ago

Thanks, will be investigated later today :-)