Closed RobHanna-NOAA closed 1 month ago
I was able to replicate the issue on this branch. The error being thrown is obviously due to the absence of the crosswalked catchments dataset gw_catchments_reaches_filtered_addedAttributes_crosswalked_8954000027.gpkg
, but it appears that there is some sort of "quiet error" upstream. I bet it has to do with the recent changes to add_crosswalk.py
in #1205.
Yes, it's because add_crosswalk.py
is exiting here https://github.com/NOAA-OWP/inundation-mapping/blob/dev/src/add_crosswalk.py#L92. It should be handled via a fim_exit code
.
Yes, it's because
add_crosswalk.py
is exiting here https://github.com/NOAA-OWP/inundation-mapping/blob/dev/src/add_crosswalk.py#L92. It should be handled via afim_exit code
.
Optional... If you want it logged somewhere, you can create a new fim_exit_codes and then add that code to process_branch so it knows how to log or drop it. If you even want to log it. Just putting it to screen will never get seen. At a min, just put a note in the code about it.
There's some abnormal stuff happening that I'm still looking into, but ultimately none of the midpoints of the DEM-derived reaches (red) are within 100 meters of the NWM stream lines (blue) so no streams are returned in the crosswalk
How is the stream being routed uphill there??
That's part of the abnormality I'm looking into...it's an example of reverse flow which we hadn't seen in a while. You can see the burned in streamline exiting in the bottom of the image.
It looks like the flowdir
grid for that branch gets corrupted in gagewatershed
which affects the DEM-derived reaches (red) and the gw_catchments
(purple lines in flowdir). The DEM appears to be correct. Interesting that we can replicate this problem and that it only seems to happen for select branches.
DEM
flowdir
The strangeness in the flowdir is being caused by a dry lake/valley bed (gray area) being raised to a constant elevation in the depression-filling process. Blue lines are NWM level paths and end at sinks (to=0
).
Ah, that makes sense. I don't think this is impactful or widespread enough to try and fix the routing, but we should definitely take care of the exit 1 status for this branch. @RobHanna-NOAA Was this the only one in the whole BED that had this problem?
It sounds like there are lots (the original issue mentions > 500 branches). I'm not surprised this happens since if (when) the DEM reach midpoint isn't spatially aligned within 100 meters of the NWM stream (and we know that does happen) then the crosswalk will be empty and result in this error being triggered later in the pipeline.
I'll add NO_VALID_CROSSWALKS
to FIM_exit_codes
No. This bug, as well as the new card from this morning 1215 src optimization error for hydro ID, both showed up in a test UAT run done on UCS3 yesterday. It was based on the new 4.5.2.5 code.
During a full BED of 4.5.2.5, over 500 branches triggered the following error:
./logs/unit/16040203_unit.log:8074:fiona._err.CPLE_OpenFailedError: /fim_temp/fim_4_5_2_5/16040203/branches/8954000027/gw_catchments_reaches_filtered_addedAttributes_crosswalked_8954000027.gpkg: No such file or directory
Log details:
This was detected even before running post processing on that output folder, but running the command line script grep scanning logs for errors. ie) grep -H -r -i -n "error" ./logs/ > ./all_errors_from_logs.log. That error file results are in the 4.5.2.5 output folder.
I will now run post-processing to see if other types of errors are discovered.