supermanzer / NDBC

Repository for housing Python code for fetching, parsing, and loading NDBC data into a local Python object for analysis.
MIT License
13 stars 4 forks source link

Provide data persistence methods #1

Closed supermanzer closed 4 years ago

supermanzer commented 4 years ago

In order to allow this package to be more useful for large or long term data analyses, provide a means of writing data sources out to various file formats (.json, .csv, .nc, etc.).

For this feature to be fully useful the data class should also have the capability to load the saved data again and append new data to previously retrieved data.

supermanzer commented 4 years ago

Due to vulnerabilities for arbitrary code execution, as well as wider interoperability concerns, we will focus on using JSON as the initial format for data persistence.

Additional approaches will involve developing APIs to allow for saving/re-instantiating DataBuoy objects to/from database records. Initial approach likely with SQLite DB but PostgreSQL as well as ORM engines like SQLAlchemy and Django should be examined.

supermanzer commented 4 years ago

Part of the meta-data stored when writing to JSON file should be the orientation of each pandas DataFrame converted to native Python objects using the to_dict() method. This will ensure a re-instantiation method will be able to determine the correct orientation when reading from a .json file.