When settings / metadata are saved to HDF5 in a DingoDataset, the dictionary is converted to a string. When loading, the string is converted back to a dictionary using ast.literal_eval(). This method was used to ensure that we could handle any data types that might appear in the settings dictionary.
It would be good to instead treat settings / metadata as any other DingoDataset saved attribute (specified by data_keys). In other words, the dictionary would be saved in HDF5 as a set of nested groups and datasets / attributes. It will be important to thoroughly check this to be sure all the values can be properly saved and loaded, and a conversion script should be written to convert the old file format to the new one.
When settings / metadata are saved to HDF5 in a
DingoDataset, the dictionary is converted to a string. When loading, the string is converted back to a dictionary usingast.literal_eval(). This method was used to ensure that we could handle any data types that might appear in the settings dictionary.It would be good to instead treat settings / metadata as any other
DingoDatasetsaved attribute (specified bydata_keys). In other words, the dictionary would be saved in HDF5 as a set of nested groups and datasets / attributes. It will be important to thoroughly check this to be sure all the values can be properly saved and loaded, and a conversion script should be written to convert the old file format to the new one.