aiidateam / disk-objectstore

An implementation of an efficient "object store" (actually, a key-value store) writing files on disk and not requiring a running server
https://disk-objectstore.readthedocs.io
MIT License
15 stars 8 forks source link

Adding a test for reading hdf5 files from compressed objects #144

Closed zhubonan closed 1 year ago

zhubonan commented 1 year ago

As discussed with @giovannipizzi, we should add a test to check if we can read from a packed and compressed hdf5 file as part of the enhancement to allow whence != 1 when reading from compressed objects #142 .

The snippet below can be used to create and read such a file:

"""
Example for using h5py
"""
from pathlib import Path

import h5py
import numpy as np

def read_write_h5(ref, compression=None):
    """Write and read a h5py file"""

    # Open an h5 file 
    with h5py.File('array.h5', 'w', ) as h5:
        h5.create_dataset('array', data=ref, compression=compression)

    # Read from the file
    with open('array.h5', 'rb') as fh:
        with h5py.File(fh, 'r') as h5:
            # Read the data back, the [:] is needed to return 
            # the numpy array (in memory)
            data = h5['array'][:]

    assert np.allclose(data, ref)
    Path('array.h5').unlink()

if __name__ == '__main__':

    for comp in [None, 'gzip']:
        ref = np.random.rand(100, 100)
        read_write_h5(ref, comp)
        ref = np.zeros((100, 100))
        read_write_h5(ref, comp)
    print('All pass')