search

SciML / DataDrivenDiffEq.jl

Data driven modeling and automated discovery of dynamical systems for the SciML Scientific Machine Learning organization
https://docs.sciml.ai/DataDrivenDiffEq/stable/
MIT License
404 stars 57 forks source link

Write and read data files for DataDrivenDiffEq inference #317

Open finmod opened 2 years ago

finmod commented 2 years ago

Can we have in all your tutorials and the README an intermediate step of writing out the synthetic data generated to an external file of your advocacy and then reading in that dataset from the external file to pursue the model discovery?

Researchers interested in using DataDrivenDiffEq should be told how they should format their experimental or statistical datasets and which Julia app is best for interaction between SciML and the rest of the galaxy. There are also some data precision issues involved and synthetic data should replicate real data acquisition methods.

AlCap23 commented 2 years ago

I appreciate your comment. However, I got mixed feelings about this.

In general, my experience with data tells me basically this: Everything is different every time. So a too rigid struct for data storage would limit the user, since a preprocessing step is needed in advance.

Personally, I like to use JLD2 and - if needed - using DataFrames + CSV. However, I know of outputs in the form of xml, txt, tab in one, two or multiple files.

I could however generate an example for use with CSV + Dataframes when working on the docs over the holidays.

finmod commented 2 years ago

Searching on your proposed solutions, I bumped into this: https://diffeq.sciml.ai/stable/features/io/ . Just showing an example of these I/O in the docs of DataDrivenDiffEq or referring to it would be fine.