aldanor / hdf5-rust

HDF5 for Rust
https://docs.rs/hdf5
Apache License 2.0
310 stars 85 forks source link

Update the example to showcase some new features #178

Closed aldanor closed 3 years ago

aldanor commented 3 years ago

@mulimoen Here's the updated example (I'll add it to the README as well) which showcases several new features (anything else? need to keep it as short as possible though)

Closes #156.

Interestingly enough, it currently fails (!) unless you disable blosc compression (in which case it runs fine), see below for examples.

This is definitely a bug somewhere and needs investigating.

  left: `[[Pixel { xy: (0, 0), color: B }, Pixel { xy: (0, 0), color: B }],
 [Pixel { xy: (0, 0), color: B }, Pixel { xy: (0, 0), color: B }]], shape=[2, 2], strides=[2, 1], layout=Cc (0x5), const ndim=2`,
 right: `[[Pixel { xy: (3, 4), color: G }, Pixel { xy: (4, 5), color: R }],
 [Pixel { xy: (5, 6), color: B }, Pixel { xy: (6, 7), color: G }]], shape=[2, 2], strides=[2, 1], layout=Cc (0x5), const ndim=2`', examples/simple.rs:42:9

And if you replace read_slice with read_2d:

  left: `[[Pixel { xy: (618879123714, -6917529027641081712), color: B }, Pixel { xy: (0, 0), color: B }],
 [Pixel { xy: (0, 0), color: B }, Pixel { xy: (0, 0), color: B }],
 [Pixel { xy: (0, 0), color: B }, Pixel { xy: (0, 0), color: B }]], shape=[3, 2], strides=[2, 1], layout=Cc (0x5), const ndim=2`,
 right: `[[Pixel { xy: (3, 4), color: G }, Pixel { xy: (4, 5), color: R }],
 [Pixel { xy: (5, 6), color: B }, Pixel { xy: (6, 7), color: G }]], shape=[2, 2], strides=[2, 1], layout=Cc (0x5), const ndim=2`', examples/simple.rs:42:9

Am I missing something obvious again? Hmm... 🤔

aldanor commented 3 years ago

Also, how to only run this example on CI with blosc enabled so it doesn't fail to compile? 🤔

aldanor commented 3 years ago

Few more observations: LZF filters work fine. SZip filters couldn't make work due to 'unknown library error', perhaps the dataset is too small or something.

mulimoen commented 3 years ago

It seems blosc can not compress the data on some algorithms and returns 0. This should cause hdf5 to fall back to no compression, but data written is garbled

mulimoen commented 3 years ago

Found and fixed the compression error, was a too eager comparison which overwrote the data buffer. For running in CI I inserted the feature flags, this might not be the easiest to read, but should compile on all platforms and still illustrate the principle

mulimoen commented 3 years ago

Build failure shows btime does not have to equal ctime for LocationInfo

aldanor commented 3 years ago

(While resolving an "add blosc_*() methods" todo, I couldn't help but refactor some impl_builder_stuff!() mess, it will help in the future iteration to work on consistent documentation etc)

Relaxed location info timestamp tests, let's see if it's green now.

I think the next release we should focus 99% on tests and documentation, it kind of got to the point where it's more important than adding/improving new features, at least in the state it's in now.