WikiWatershed / model-my-watershed

The web application front end for Model My Watershed.
https://modelmywatershed.org
Apache License 2.0
57 stars 31 forks source link

Add NHD stream order attribute #1408

Closed mmcfarland closed 8 years ago

mmcfarland commented 8 years ago

NHD Flowline features are currently loaded into the database. The NHD dataset contains a related table set which includes an attribute called stream order. Produce an nhdflowlines table to replace the current one that include this attribute.

mmcfarland commented 8 years ago

The nhdflowline dataset was updated to have stream_order field.

To test, run ./setupdb.sh -s

mmcfarland commented 8 years ago

Additional test, check that layer still renders with the tiler and that a MapShed run, outside of the DRB, still works.

lewfish commented 8 years ago

+1, tested

mmcfarland commented 8 years ago

Leaving some instructions here about how this was accomplished. NHD datasets can be downloaded by region from http://www.horizon-systems.com/NHDPlus/NHDPlusV2_data.php. There are several dozen regions. When the data is extracted, it forms a nested directory structure and at one of the leafs of the tree will be a PlusFlowlineVAA.dbf file that has attributes for nhdflowlines. A key field called comid relates a VAA row to a flowline feature. The PG driver for ogr2ogr can't append to a table, so I do an intermediate step of collecting the several dozen dbf files into a single dbf file. Simultaneously, I only extract the attribute I care about (streamorde) to keep the table size small:

find . -name 'PlusFlowlineVAA.dbf' -exec ogr2ogr -update -append ./totals.dbf {} -sql "SELECT comid, streamorde from PlusFlowLineVaa" \;

I then copy the new dbf file to postgres:

ogr2ogr -update -append -f "PostgreSQL" PG:"host=localhost user=mmw dbname=mmw password=***" -nln plusflowlinevaa totals.dbf

Then a simple join against the ~3M flowline rows to add the attribute to a new table which forms the basis of the data update:

SELECT f.*, a.streamorde INTO nhdflowline_plus 
FROM nhdflowline f LEFT JOIN plusflowlinevaa a ON f.comid = a.comid;