search

kinverarity1 / python-sa-gwdata

Python package for the Groundwater Data section of the DEW WaterConnect website
https://python-sa-gwdata.readthedocs.io/en/latest/index.html
MIT License
3 stars 0 forks source link

Store data in a local database cache #19

Open kinverarity1 opened 4 years ago

kinverarity1 commented 4 years ago

Is your feature request related to a problem? Please describe. Every time you make a request for some data that you've already done, it re-downloads it.

Describe the solution you'd like A locally stored database cache. Ideally this would be allowed to have a maximum size, and therefore we would benefit from an ORM so that we easily do bulk removes, updates, and inserts. The data model should be based only on what's available via the bulk CSV downloads. The JSON service queries are an extra goodie which could be mapped across later. So SQLAlchemy would be an additional dependency. SQLite would be the obvious choice for the cache itself.

The default behaviour of whether to rely solely on the cached results, or update from Groundwater Data, should be configurable, but an easy rule of thumb would be that for some queries like water_levels and salinities, an update should be run if the query is on a new day. Others like well_summary, and so on, should probably be more like a month. In any case: add a keyword argument to the query methods: update_cache=False/True/"auto".

kinverarity1 commented 1 year ago

FYI #31 adds a framework to add this on to.