Planning - Data structure flexibility

[PeterDSteinberg]

Currently elm automates fitting and prediction from ElmStore or xarray data structures (see also elm-readers elm_store.py.

• We have the Phase II goal of supporting a wide variety of data structures in machine learning, such as numpy / dask arrays, pandas / dask dataframes, and xarray Datasets, as well as better support for reading multi-file datasets.
• We are moving file reading and geographic utilities to elm-readers to better separate the ensemble and genetic algorithm ML logic and its requirements from the requirements of the geographic data.

Planning items:

• How can parts of elm.model_selection and elm.pipeline subpackages being rewritten in a way to minimize assumptions about data structures?
• How should transforming functions check data structures for requirements? (Some functions like spatial preprocessors may inherently require a data structure other than a numpy array so that spatial metadata may be conveyed through a pipeline)
• This issue relates in part to #143 (improvement of parallelism) as the data structures we allow in different parts of the pipeline may have implications for our approach to parallelism.
• Related to data structure flexibility, is there any need to refactor the API for earthio / elm.readers, such as making the endpoints more similar to xarray Dataset loading functions?
• Where do utils related to data structures go? If we have a separate earthio / elm.readers and this elm repo that both may need a data structure util, such as a base class, then we can't have the two packages be optionally coupled if one pkg requires the other pkg for an import.

[PeterDSteinberg]

See also https://github.com/mrocklin/sparse for sparse matrix ideas.

[PeterDSteinberg]

This issue largely overlaps with #149, an epic issue that covers the topics and plans in more detail. Closing this 145 issue so we can keep discussion on #149.