error reading datasets compressed with dynamic compression filters

[paukel]

Dynamiacally loaded compression filters are not recognized - lzf for example.

[paukel]

Dynamic filters works with h5 library only for a very small files.

I prepared two hdf5 files using h5py with several 1D and 2D datasets varying in size, compression and maxshape. LZF dynamic coimpression filter is built and works both in Python and with h5dump.

from h5py import File
from numpy import arange

def h5_build(n, suffix):
    # create HDF5 file for tests
    N = n**4
    with File('.'.join(('h5_test', suffix)), 'w') as df:
        # <DIM>_<Compression>_<is first dim unlimited>
        #1D
        df.create_dataset('1D_F_F', data=arange(N, dtype='f4')) # 1D no compression
        df.create_dataset('1D_LZF_F', data=arange(N, dtype='f4'), compression='lzf') # 1D lzf 
        df.create_dataset('1D_LZF_T', data=arange(N, dtype='f4'), compression='lzf', maxshape=(None,)) # 1D lzf unlimited 1dim
        #2D
        df.create_dataset('2D_F_F', data=arange(N, dtype='f4').reshape(-1,n**2)) # 1D no compression
        df.create_dataset('2D_LZF_F', data=arange(N, dtype='f4').reshape(-1,n**2), shape=(n**2,n**2), compression='lzf') # 1D lzf 
        df.create_dataset('2D_LZF_T', data=arange(N, dtype='f4').reshape(-1,N), compression='lzf', maxshape=(None,N)) # 1D lzf unlimited 1dim

# tiny file
h5_build(3, 'tiny')
# small
h5_build(9, 'small')

h5 perfectly reads tiny example but it fails while reading lzf compressed dataset from the second file.

> library(h5)
> tf <- h5file('h5_test.tiny')
> tf['2D_LZF_F'][1:3,1:3]
     [,1] [,2] [,3]
[1,]    0    1    2
[2,]    9   10   11
[3,]   18   19   20
> h5close(tf)
> sf<-h5file('h5_test.small')
> sf['2D_F_F'][1:3,1:3]
     [,1] [,2] [,3]
[1,]    0    1    2
[2,]   81   82   83
[3,]  162  163  164
> sf['2D_LZF_F'][1:3,1:3]
ERROR: H5Dread failed in DataSet::read
> h5close(sf)

[mike-lawrence]

Popping in to add a vote for this feature!

[makmanalp]

For what it's worth, me too!