opensatx / sawaterlevel-api

Simple web scraper that grabs water data for sawaterlevel app, serves json
sawaterlevels-api.herokuapp.com
3 stars 2 forks source link

Add APIs for requesting historical data #8

Open waynehartman opened 10 years ago

waynehartman commented 10 years ago

Being able to visualize data can be a powerful way to understand trends over time. Creating an API to request historical data would allow client applications to display data in an interesting and compelling way.

An important aspect of offering these kinds of APIs would be to limit how much clients would tax the system, so offering granular service APIs could be advantageous from a performance perspective.

The Edwards Aquifer Authority makes the historical data available in a CSV format, so doing a bulk export of data that can be cached in a database would be feasible. Getting regular extracts of the data might take a bit more effort in order to keep the data fresh, but is definitely doable.

I propose the following services:

Path: /historical/15minute/past24hours Description: Returns the water level at 15 minute intervals for the past 24 hours. Example return payload:

{
    "levels" : [
        {
            "timestamp" : "2014-07-17T09:15:00+00:00",
            "level" : 645.67
        }, {
            "timestamp" : "2014-07-17T09:30:00+00:00",
            "level" : 645.50
        }, {
             // etc.
        }
    ]
}

Path: /historical/15minute/past7days Description: Returns the water level at 15 minute intervals for the past 7 days. Example return payload: See sample above

Path: /historical/daily/ POST data (contentType=application/json):

{
    "startDate" : "2013-07-16T00:00:00+00:00",
    "endDate" : "2014-07-17T00:00:00+00:00"
}

Description: Returns the official daily water level for the passed date range. Example return payload: See sample above

Allenrocks commented 10 years ago

Wayne I agree the ability to visualize the data on a trend would go well with other forms of weather tracking.

I'll see about putting together some correlation data this week.

On Jul 23, 2014, at 5:25 PM, Wayne Hartman notifications@github.com wrote:

Being able to visualize data can be a powerful way to understand trends over time. Creating an API to request historical data would allow client applications to display data in an interesting and compelling way.

An important aspect of offering these kinds of APIs would be to limit how much clients would tax the system, so offering granular service APIs could be advantageous from a performance perspective.

The Edwards Aquifer Authority makes the historical data available in a CSV format, so doing a bulk export of data that can be cached in a database would be feasible. Getting regular extracts of the data might take a bit more effort in order to keep the data fresh, but is definitely doable.

I propose the following services:

Path: /historical/15minute/past24hours Description: Returns the water level at 15 minute intervals for the past 24 hours. Example return payload:

{ "levels" : [ { "timestamp" : "2014-07-17T09:15:00+00:00", "level" : 645.67 }, { "timestamp" : "2014-07-17T09:30:00+00:00", "level" : 645.50 }, { // etc. } ] } Path: /historical/15minute/past7days Description: Returns the water level at 15 minute intervals for the past 7 days. Example return payload: See sample above

Path: /historical/daily/ POST data (contentType=application/json): { "startDate" : "2013-07-16T00:00:00+00:00", "endDate" : "2014-07-17T00:00:00+00:00" }

Description: Returns the official daily water level for the passed date range. Example return payload: See sample above

— Reply to this email directly or view it on GitHub.

Allen P. Buck IV LinkedIn T. 210.595.9045 F. 1.830.521.0139 M. 327 E. Fest St. San Antonio, TX 78204

Asclepius Marketing Solutions T. 210.679.1559 F. 1.830.521.0139 M. 300 Convent St. Suite 1330 San Antonio, TX 78205

waynehartman commented 10 years ago

I think temperature and precipitation data would be pretty cool to include to include in the data, as well. I would find it interesting to be able to see corollaries between weather and aquifer level data.

davidrleonard commented 10 years ago

Where are you seeing the .csv for aquifer data?

waynehartman commented 10 years ago

Select the J17 data from this page:

http://data.edwardsaquifer.org/histwaterlevels_text.asp