search

Photon-HDF5 / phconvert

Convert Beker&Hickl, PicoQuant and other formats to Photon-HDF5
http://photon-hdf5.github.io/phconvert
Other
16 stars 14 forks source link

save_photon_hdf5(): handle struct arrays inputs #7

Closed tritemio closed 8 years ago

tritemio commented 9 years ago

In case of PQ files, the metadata is loaded as struct arrays that have the same layout in memory as the file on disk. These arrays cannot be saved in as "HDF5 arrays" but need to be saved as "HDF5 tables". However, phconvert does not detect and save tables as of yet (tables are not used in any official field).

As a result, currently, the notebook the converts HT3 file, manually saves the metadata (using pytables) which is not very robust and expose complexity to the user.

This metadata should be automatically handled by save_photon_hdf5() instead, as is the case for the BH notebook.

smXplorer commented 9 years ago

I am not sure I understand what the statement above means. Can you clarify? In particular, what would that change for a user using HDFView or any other HDF5 Viewer without a priori knowledge of this user data?

tritemio commented 9 years ago

This issue is about an improvement to phconvert that would simplify it's usage and make file creation a little more robust when saving raw metadata from HT3 and similar files.

The proposed addition would not change in any way the resulting Photon-HDF5 files.

I try to clarify. Some fields in the HT3 metadata are read as numpy structured arrays from the reader function in phconvert. They are arrays of length 1, but the element is a structure with different "fields" (each field has different type or number of bytes). Structured arrays can be saved to HDF5 as "HDF5 tables".

The function save_photon_hdf5() takes as input a dict-based representation of the HDF5 file and saves it to disk. However, currently, it will only handle arrays (which includes strings, because for pytables strings are just a different kind of array). Ideally, save_photon_hdf5() should detect when a field in the input is a structured array and save it as a table (using tables.create_tables()). Not very difficult.

Currently, the HT3 converter notebook, to save this raw metadata, has a cell that calls tables.create_tables() manually, which is a little more fragile because no tests are performed when saving those fields.

tritemio commented 8 years ago

This issue is addressed by 23e1aadd503412884a0141b74a7235cc2e9a96a3. Closing.