search

edgi-govdata-archiving / ECHO-Cross-Program

Jupyter Notebooks for ECHO that use data from multiple EPA programs
https://colab.research.google.com/github/edgi-govdata-archiving/ECHO-Cross-Program/blob/master/ECHO-Cross-Programs.ipynb
GNU General Public License v3.0
8 stars 5 forks source link

Add last modified date #84

Open ericnost opened 3 years ago

ericnost commented 3 years ago

Last-Modified data as returned from the ECHO EPA server

http://apps.tlt.stonybrook.edu/echoepa/?query=select%20*%20from%20%22Last-Modified%22&pg

Perhaps we can add a property "last_modified" to the data set class, something that goes and gets the value from the "modified" field in this table for each table...

shansen5 commented 3 years ago

I see that each table can have a different 'modified' date stamp, so we will get this per dataset. Since it will be the same for all DataSetResults, we don't need to get it each time results are obtained. (I'll put this into a branch of ECHO_modules.)

shansen5 commented 3 years ago

A slight complication in that Last-Modified has the table names (e.g. RCRA_VIOLATIONS), while we are using the views based on them (RCRA_VIOLATIONS_MVIEW). I'll have to add something like base_table_name to each of the DataSet objects when they are created in make_data_sets().

shansen5 commented 3 years ago

A new base_table was added to each DataSet instance identifying the database table that reflects the data set's last modified time. In the DataSet's get_data() function the last_modified time is set if it hasn't already been done.

Frijol commented 2 years ago

What's the use context for this date? How do we hope to incorporate it?