search

edgi-govdata-archiving / ECHO-Cross-Program

Jupyter Notebooks for ECHO that use data from multiple EPA programs
https://colab.research.google.com/github/edgi-govdata-archiving/ECHO-Cross-Program/blob/master/ECHO-Cross-Programs.ipynb
GNU General Public License v3.0
8 stars 5 forks source link

Use last_modified timestamp to save database query results #92

Open shansen5 opened 3 years ago

shansen5 commented 3 years ago

With the last_modified of the database tables we can store CSV files of database query results and use them in subsequent runs for faster responses.
When asked for a data set we can check a date stamp on the CSV file and query the Last Modified table to see if our CSV still contains the most current data.
Using data from a local CSV will be faster than downloading results from the database.

Frijol commented 2 years ago

@shansen5 is this still relevant?