The Meta Dataset allows for easy loading of data! See the documentation for more info.
Features include:
Standardized Loading routines for labels
Standardized Loading routines for get_example calls via predefined functions applied to labels as defined in the datasets meta.yaml file or even easier via the label array's name
Custom label to loaded thing routines are also possible, by referencing custom functions in the meta.yaml
It has a nice show method, which probably should be part of the DatasetMixin class, but for now it is here. Try it in a jupyter notebook and be amazed.
Things, that should be done in the near future:
Port the EvalDatafolder and the whole eval pipeline to this new dataset class (This should be relatively easy, as large parts of this class are derived from the EvalDatafolder)
Once the eval pipeline is ported, package the data writing routines, such that we can easily reuse them to export arbitrary datasets.
Write a test for the MetaDataset, which uses a custom function as loader
The Meta Dataset allows for easy loading of data! See the documentation for more info.
Features include:
meta.yamlfile or even easier via the label array's namemeta.yamlshowmethod, which probably should be part of theDatasetMixinclass, but for now it is here. Try it in a jupyter notebook and be amazed.Things, that should be done in the near future:
EvalDatafolderand the whole eval pipeline to this new dataset class (This should be relatively easy, as large parts of this class are derived from theEvalDatafolder)MetaDataset, which uses a custom function as loader