Closed aaronhans closed 4 years ago
Right now we have the latest data and can refresh it with scripts running locally. Ran into some interesting issues pulling the HR2W datasource with code: https://github.com/nodejs/node/issues/27711 need to run the script with flags to grab an older http parser because the water site is using headers with spacers which break latest node parsers so makes any retrieval method fail including request and server side fetch. I'm not sure yet how to get our script to run regularly on a lambda using these flags so we need to research this. I'd like to split that off into a separate issue and address it when we see a water data update (which also might break scripts if they change structure)
So postponing this issue: "Create automated process that pulls that dataset regularly and processes it into our API"
New issue to solve this: https://github.com/cagov/API/issues/23
great documentation- thank you Aaron
HR2W data is published every couple months here https://www.waterboards.ca.gov/water_issues/programs/hr2w/
We should:
- [ ] Create automated process that pulls that dataset regularly and processes it into our API