error at get_huc function

[zexuanxu]

Hi Ethan, I hit an error msg when running the mesh_coeweet.ipynb file at

find what HUC our shape is in, any quick hint?

huc = workflow.find_huc(sources['HUC'], coweeta.exterior(), crs, hint, shrink_factor=0.1) logging.info("Found Coweeta in HUC: {}".format(huc)) 2020-08-10 01:15:55,597 - root - INFO: 2020-08-10 01:15:55,598 - root - INFO: Preprocessing HUC 2020-08-10 01:15:55,599 - root - INFO: ------------------------------ 2020-08-10 01:15:55,599 - root - INFO: Loading level 4 HUCs in 0601. 2020-08-10 01:15:55,621 - root - INFO: Using HUC file "/global/project/projectdirs/m2398/watershed-workflow/examples/data/hydrography/WBD_06_GDB/WBD_06.gdb"

KeyError Traceback (most recent call last)

in 1 # find what HUC our shape is in ----> 2 huc = workflow.find_huc(sources['HUC'], coweeta.exterior(), crs, hint, shrink_factor=0.1) 3 logging.info("Found Coweeta in HUC: {}".format(huc)) /project/projectdirs/m2398/watershed-workflow/workflow/hilev.py in find_huc(source, shape, crs, hint, shrink_factor) 473 hint = workflow.sources.utils.huc_str(hint) 474 --> 475 _, hint_hu = get_huc(source, hint, crs) 476 inhuc = _in_huc(shply_s, hint_hu) 477 if inhuc is not 2: /project/projectdirs/m2398/watershed-workflow/workflow/hilev.py in get_huc(source, huc, crs, digits) 63 """ 64 huc = workflow.sources.utils.huc_str(huc) ---> 65 crs, hu_shapes = get_hucs(source, huc, len(huc), crs, digits) 66 assert(len(hu_shapes) == 1) 67 return crs, hu_shapes[0] /project/projectdirs/m2398/watershed-workflow/workflow/hilev.py in get_hucs(source, huc, level, crs, digits) 102 logging.info("Loading level {} HUCs in {}.".format(level, huc)) 103 --> 104 profile, hus = source.get_hucs(huc, level) 105 logging.info(' found {} HUCs.'.format(len(hus))) 106 for hu in hus: /project/projectdirs/m2398/watershed-workflow/workflow/sources/manager_nhd.py in get_hucs(self, huc, level) 107 with fiona.open(filename, mode='r', layer=layer) as fid: 108 hus = [hu for hu in fid if hu['properties']['HUC{:d}'.format(level)].startswith(huc)] --> 109 profile = fid.profile 110 profile['always_xy'] = True 111 return profile, hus /project/projectdirs/m2398/watershed-workflow/workflow/sources/manager_nhd.py in (.0) 107 with fiona.open(filename, mode='r', layer=layer) as fid: 108 hus = [hu for hu in fid if hu['properties']['HUC{:d}'.format(level)].startswith(huc)] --> 109 profile = fid.profile 110 profile['always_xy'] = True 111 return profile, hus KeyError: 'HUC4'

[daniellivingston]

@zexuanxu , were you able to get this resolved? I get the same error (Python 3.7.8, watershed_workflow_dev conda env).

[zexuanxu]

Ah, not yet, maybe it's a good time to @ecoon

[daniellivingston]

FYI I get this error with bin/mesh_hucs.py too:

$ python bin/mesh_hucs.py 060102080102 mytest.exo
2020-08-19 16:45:52,806 - root - INFO:
2020-08-19 16:45:52,807 - root - INFO: Meshing HUC: 060102080102
2020-08-19 16:45:52,807 - root - INFO: ==============================
2020-08-19 16:45:52,807 - root - INFO: Target projection: "epsg:5070"
2020-08-19 16:45:52,807 - root - INFO:
2020-08-19 16:45:52,807 - root - INFO: Preprocessing HUC
2020-08-19 16:45:52,807 - root - INFO: ------------------------------
2020-08-19 16:45:52,807 - root - INFO: Loading level 12 HUCs in 060102080102.
2020-08-19 16:46:00,691 - root - INFO: Unzipping: "/Users/livingston/playground/watershed-workflow/data/hydrography/WBD_06_GDB/raw/WBD_06_HU2_GDB.zip"
2020-08-19 16:46:00,691 - root - INFO:        to: "/Users/livingston/playground/watershed-workflow/data/hydrography/WBD_06_GDB/raw"
2020-08-19 16:46:01,088 - root - INFO: Moving: "/Users/livingston/playground/watershed-workflow/data/hydrography/WBD_06_GDB/raw/WBD_06_HU2_GDB.gdb"
2020-08-19 16:46:01,088 - root - INFO:     to: "/Users/livingston/playground/watershed-workflow/data/hydrography/WBD_06_GDB/WBD_06.gdb"
2020-08-19 16:46:01,088 - root - INFO: Using HUC file "/Users/livingston/playground/watershed-workflow/data/hydrography/WBD_06_GDB/WBD_06.gdb"
Traceback (most recent call last):
  File "bin/mesh_hucs.py", line 69, in <module>
    hucs, rivers, triangulation = mesh_hucs(args)
  File "bin/mesh_hucs.py", line 43, in mesh_hucs
    crs, hucs = workflow.get_split_form_hucs(sources['HUC'], args.HUC, crs=args.projection)
  File "/Users/livingston/playground/watershed-workflow/workflow/hilev.py", line 154, in get_split_form_hucs
    crs, hu_shapes = get_hucs(source, huc, level, crs, digits)
  File "/Users/livingston/playground/watershed-workflow/workflow/hilev.py", line 104, in get_hucs
    profile, hus = source.get_hucs(huc, level)
  File "/Users/livingston/playground/watershed-workflow/workflow/sources/manager_nhd.py", line 109, in get_hucs
    hus = [hu for hu in fid if hu['properties']['HUC{:d}'.format(level)].startswith(huc)]
  File "/Users/livingston/playground/watershed-workflow/workflow/sources/manager_nhd.py", line 109, in <listcomp>
    hus = [hu for hu in fid if hu['properties']['HUC{:d}'.format(level)].startswith(huc)]
KeyError: 'HUC12'

$ which python && python --version
/anaconda3/envs/watershed_workflow_dev/bin/python
Python 3.7.8

[pinshuai]

I have just submitted a pull request that can resolve this issue. The problem is that HUC field in the downloaded WBD is in lower case. You need to change the upper case HUC to huc in the following line.

hus = [hu for hu in fid if hu['properties']['HUC{:d}'.format(level)].startswith(huc)]

You may need to do this for other lines that have hu['properties']['HUC{:d}'.format(level)].startswith(huc)].

[ecoon]

Yeah, sorry, for some reason my email settings seem to be burying notifications from GitHub for this repo, so this is the first I saw the ticket. USGS seems to have change the standard in their NHD files at least from HUCX (where X is the level) to hucX.

I just accepted Pin's pull request that fixes this to hucX, but it may break old downloads (I have ~100GB of existing NHD files that I don't want to re-download just to change this dictionary key). So I guess the "right" fix is to try hucX and if it fails fall back on looking for HUCX.

[ecoon]

fixed