cagov / API

Used to develop Function As A Service Apps in support of Alpha.CA.gov.
5 stars 3 forks source link

Put Human Right to Water testing data into API #19

Closed aaronhans closed 4 years ago

aaronhans commented 4 years ago

HR2W data is published every couple months here https://www.waterboards.ca.gov/water_issues/programs/hr2w/

We should:

- [ ] Create automated process that pulls that dataset regularly and processes it into our API

aaronhans commented 4 years ago

Right now we have the latest data and can refresh it with scripts running locally. Ran into some interesting issues pulling the HR2W datasource with code: https://github.com/nodejs/node/issues/27711 need to run the script with flags to grab an older http parser because the water site is using headers with spacers which break latest node parsers so makes any retrieval method fail including request and server side fetch. I'm not sure yet how to get our script to run regularly on a lambda using these flags so we need to research this. I'd like to split that off into a separate issue and address it when we see a water data update (which also might break scripts if they change structure)

So postponing this issue: "Create automated process that pulls that dataset regularly and processes it into our API"

New issue to solve this: https://github.com/cagov/API/issues/23

abquirarte commented 4 years ago

great documentation- thank you Aaron