search

Apollo3zehn / PureHDF

A pure .NET library that makes reading and writing of HDF5 files (groups, datasets, attributes, ...) very easy.
MIT License
47 stars 16 forks source link

Byte order conversion is not (yet) support by PureHDF. #101

Closed migajek closed 3 weeks ago

migajek commented 3 weeks ago

While trying to read WRF "/west_east" dataset, I am getting the abovementioned exception. Is there any workaround, like accessing the raw memory before the conversion / check occurs, so I could handle the conversion along with endianess myself?

Apollo3zehn commented 3 weeks ago

Are you on a big-endian system or is your data big-endian? I will try to fix this today evening and release a new version :-)

migajek commented 3 weeks ago

@Apollo3zehn I'm on a little-endian system, apparently my data is big-endian. That'd be really great, I looked at the code but from my understanding there's no way to hook into the process

Apollo3zehn commented 3 weeks ago

It could also be that PureHDF is doing something wrong when reading the file so that it misinterprets little endian data as big endian. Do you have the possibility to send an example file? I am asking because I never encountered big-endian data. I can send you a special mail address if you do not want to share it publicly.

migajek commented 3 weeks ago

@Apollo3zehn I've requested the permission to share the file. I expect there should be no issue with that; please provide me the email address. The file is under 10Mb

Apollo3zehn commented 3 weeks ago

Perfect! Here is the address: purehdf-issue-101@m1.apollo3zehn.net

Apollo3zehn commented 3 weeks ago

Version v1.0.0-beta.24 integrates a workaround. The endianness check still exists but is skipped when the user uses a special overload to read the data. This overload allows to pass a byte[] or Span<byte> buffer to the read method and is intended for raw data access. It is not new but a good match to solve your problem since we want unchanged raw data.

You can use it like this:

var h5File = H5File.OpenRead("<path-to-file>);
var dataset = (NativeDataset)h5File.Dataset("west_east");
var result = new float[dataset.Space.Dimensions[0]];

dataset.Read(buffer: MemoryMarshal.Cast<float, byte>(result.AsSpan()));

Issue #102 tracks the integration of a proper endianness converter. I already created a hardware-accelerated endianness converter in a different project but did not yet find the time to integrate it here (and maybe there were other problems I do not remember anymore).

I hope v1.0.0-beta.24 will solve your problem :-)