search

azavea / noaa-hydro-data

NOAA Phase 2 Hydrological Data Processing
11 stars 3 forks source link

Save extract of HUC8 #48

Closed lewfish closed 2 years ago

lewfish commented 2 years ago

The queries we want to perform all involve getting a set of reach id's for a HUC. This can be done using the NHDPlus V2 and NHDPlus HR dataset, but that requires a running database, which is not cloud-friendly. Instead, we'd like to load NHD into a database and then save a JSON file for each HUC8. The name of the file will be the HUC ID, and the contents will be a list of reach IDs (ie. the COMID field in NHDPlus V2). We probably don't need the HUC boundary or reach geometries (aside from visualization purposes), so those should be saved in a separate file.

Here is a good starting point: https://github.com/azavea/noaa-hydro-data/blob/master/notebooks/save_nhd_extract.ipynb

This should be done in a notebook for all HUC8s. This will require loading the datasets into a database. See https://github.com/azavea/noaa-hydro-data#local-database-setup

lewfish commented 2 years ago

You can see why we need both NHD database by looking at the comments in https://github.com/azavea/noaa-hydro-data/blob/master/notebooks/huc8_streamflow_query.ipynb

lewfish commented 2 years ago

@vlulla Did you upload the extracts to S3, or is that still in progress? In case not, I made an S3 bucket in the NOAA account: azavea-noaa-hydro-data.

vlulla commented 2 years ago

I uploaded the data on S3.

lewfish commented 2 years ago

I uploaded the data on S3.

Where is the data on S3?

vlulla commented 2 years ago

s3://azavea-noaa-hydro-data/noaa/ ... but this is only for the sample data listed in the README.md !

vlulla commented 2 years ago

All the geojsons are now on s3://azavea-noaa-hydro-data/noaa/huc8-extracts/ . Ought I to close this issue now?

lewfish commented 2 years ago

No, you should put Closes #48 in your PR. Then, when that PR is closed, this issue will be closed automatically. In general, the issue shouldn't be closed until the associated PR has been reviewed and accepted.