aldanor / hdf5-rust

HDF5 for Rust
https://docs.rs/hdf5
Apache License 2.0
310 stars 85 forks source link

Add Chunk::Max (chunk_max) and limit chunks to 2GB in auto modes #186

Open aldanor opened 3 years ago

aldanor commented 3 years ago

Since filter values are int32, the maximum safe chunk size is (1 < 31) - 1; e.g., blosc would fail otherwise. Should we limit chunk sizes by 2GB? (at least when using min-kb mode, or when determining chunk size automatically)

It might also be nice to add Chunk::Max which would pick the largest possible chunk size (i.e., shape), but limited by 2GB.