NOAA-OWP / inundation-mapping

Flood inundation mapping and evaluation software configured to work with U.S. National Water Model.
Other
95 stars 30 forks source link

[8pt] Split CatFIM output library file to group by HUC2's #1331

Open RobHanna-NOAA opened 2 weeks ago

RobHanna-NOAA commented 2 weeks ago

During testing of CatFIM 2.1, it was proven that the CatFIM output library files are too big. In fim 4.5.2.11, stage based library file was one giant csv file and was ~10 GiB and it was very difficult to load based on it's size. This also made it pretty hard and slow to copy around, load in GPKG and debug. Even in the program, merging the 1,000's of individual site gpkgs was very hard on the server as it was loading all of that into memory in order to create one file.

Currently in the gpkg output folder, every single site, magnitude and interval creates a new file.

With the bug from 4.4.0.0/4.5.2.11 for stage for intervals, we are now expecting more interval and an even larger outputs. Split the final output file in HUC2.

Plan:

RobHanna-NOAA commented 2 weeks ago

Put on hold. Adding this feature create more problems than expected. It all means we would have problem with our new catfim comparison tool which an compare versions of outputs (ie.. 4.4.0.0 to 4.5.2.11, etc)