Open finmod opened 2 years ago
I appreciate your comment. However, I got mixed feelings about this.
In general, my experience with data tells me basically this: Everything is different every time. So a too rigid struct for data storage would limit the user, since a preprocessing step is needed in advance.
Personally, I like to use JLD2 and - if needed - using DataFrames + CSV. However, I know of outputs in the form of xml, txt, tab in one, two or multiple files.
I could however generate an example for use with CSV + Dataframes when working on the docs over the holidays.
Searching on your proposed solutions, I bumped into this: https://diffeq.sciml.ai/stable/features/io/ . Just showing an example of these I/O in the docs of DataDrivenDiffEq or referring to it would be fine.
Can we have in all your tutorials and the README an intermediate step of writing out the synthetic data generated to an external file of your advocacy and then reading in that dataset from the external file to pursue the model discovery?
Researchers interested in using DataDrivenDiffEq should be told how they should format their experimental or statistical datasets and which Julia app is best for interaction between SciML and the rest of the galaxy. There are also some data precision issues involved and synthetic data should replicate real data acquisition methods.