how to save experiment objects on HDD?

[ibah]

Catalog has a dictionary (catalog: models, predictions, evaluations, ...) of dictionaries (each containing trained models, or predictions, or evaluations). How to save these to HDD? There can be a folder for every sub-dictionary (e.g. library/predictions) but then we should determine:

• format of the files
• how to split the data into multiple files so we don't create a single too-large-to-handle file
• how to enable fast retrieving of objects from these files (when requested by Librarian through Catalog)
• what should be the schedule for saving object by Catalog on HDD (to be both efficient and ensure data are stored even when the process crushes)

Also we should pick some very simple version to start with something :)

[dryguz]

I would suggest to simply pickle objects we need to files. https://www.thoughtco.com/using-pickle-to-save-objects-2813661

Structure of folders... that would be easier to figure out when we will have class hierarchy and scope of functionality (does module will heavily transform data or will just consume data prepared outside of module), we need assume how we want this module to be used.

[ibah]

Yes, pickle can be our default solution. It is flexible enough to handle all our needs. We can even pickle multiple objects into one file to avoid having too large number of files. I put some other options in a document on Persistence that I am about to share. To summarize the options:

• pickle
• joblib
• format specific for DataFrames (for data only) like HDF5