Apollo3zehn / PureHDF

A pure .NET library that makes reading and writing of HDF5 files (groups, datasets, attributes, ...) very easy.
MIT License
47 stars 16 forks source link

Filter pipeline improvements #33

Closed Apollo3zehn closed 1 year ago

Apollo3zehn commented 1 year ago

1.) Filters like deflate do not know uncompressed size. But if they are the last filter in the pipeline, the size is known by the chunk size.

Also if is the second last filter, the following filter might be shuffle which does not change buffer size. So for deflate it is always better to use the chunk size as guess than to use the input buffer size.

2.) All filters should use MemoryPool instead of new byte[].

3.) The last filter can directly write to the resulting array, save one copy operation.

To enable this there should be a FilterInfo structure with currently present fields + MemoryProvider.

This class would have method GetResultMemory() which normally provides a MemoryPool memory. But if the filter is the last one in pipeline, it returns the sliced result buffer.