search

Ulm-IQO / qudi-iqo-modules

A collection of qudi measurement modules originally developed for experiments on colorcenters in semiconductor materials.
GNU General Public License v3.0
12 stars 29 forks source link

[New Feature] Saving GUI & logic overhaul #150

Open prithviulm opened 3 months ago

prithviulm commented 3 months ago

Feature Description

There is no consistent saving interface across all modules (e.g. some give the option to have custom nametags, some do not). Also, the only way users can add additional metadata is in the filenames which is not ideal (difficult to parse, character limits etc).

We should have a common saving widget (and accompanying logic) that is imported into every GUI for consistent saving behaviour. There should also be an option to add custom metadata to the saved data (that is not appended to the filename but is instead stored within the file). This could simplify the organisation and sorting of large amounts of experimental data without having to solely rely on filename and autogenerated metadata.

Related Problem

No response

Considered Alternatives

This is a longterm overhaul given the scope of the work.

Additional Context

An additional feature could be the use of the HDF5 format (see Wikipedia here and Python library here) which is designed for neatly storing large amounts of (possibly mixed) data. HDF5 natively supports metadata as well. One can then easily sort and filter data based on metadata tags to retrieve only data they are interested in. A GUI tool for opening and accessing the HDF file could also be added to Qudi. There are examples here and here which are not Python-based but give an idea of what is possible.

Contact Details

No response

Neverhorst commented 3 months ago

I agree with adding some drop-in functionality to facilitate saving in GUIs.

However it is not correct that you have no means to store metdata in the current state. The DataStorage objects have this option already. In case of text file saving, this information goes into the commented header. For other formats (e.g. numpy binary) you have an additional metadata file which is fine if you use the loading functionality if DataStorage to read it back into memory.

HDF5 came up several times in the past but was always turned down by most users due to the unfamiliarity and non-human-readable nature. Many people use their own data post-processing scripts and a text file or numpy file simplifies handling. If you really want this format, it should be possible to implement a DataStorage specialisation that writes/reads to/from HDF5 so it can be used interchangeably with other save formats (this was the whole idea behind DataStorage classes)

prithviulm commented 3 months ago

I see. I wasn't aware of the HDF5 history. Also thanks for the suggestion on how it could be implemented.

With regards to the DataStorage objects supporting metadata, I meant the GUIs do not allow users to add metadata to the saved files (at least not that I know of). I wasn't being clear.