During testing of CatFIM 2.1, it was proven that the CatFIM output library files are too big. In fim 4.5.2.11, stage based library file was one giant csv file and was ~10 GiB and it was very difficult to load based on it's size. This also made it pretty hard and slow to copy around, load in GPKG and debug. Even in the program, merging the 1,000's of individual site gpkgs was very hard on the server as it was loading all of that into memory in order to create one file.
Currently in the gpkg output folder, every single site, magnitude and interval creates a new file.
With the bug from 4.4.0.0/4.5.2.11 for stage for intervals, we are now expecting more interval and an even larger outputs. Split the final output file in HUC2.
Plan:
create a new subfolder in the root dir of the catfim file called "outputs"
put all files including the csv;s into that folder. It will represent all files / folder that are final outputs and are already grouped to go to HydroVIS.
Using the gpkg output folder, load and roll up all files into their respective HUC2 gpkg.
Put on hold. Adding this feature create more problems than expected. It all means we would have problem with our new catfim comparison tool which an compare versions of outputs (ie.. 4.4.0.0 to 4.5.2.11, etc)
During testing of CatFIM 2.1, it was proven that the CatFIM output library files are too big. In fim 4.5.2.11, stage based library file was one giant csv file and was ~10 GiB and it was very difficult to load based on it's size. This also made it pretty hard and slow to copy around, load in GPKG and debug. Even in the program, merging the 1,000's of individual site gpkgs was very hard on the server as it was loading all of that into memory in order to create one file.
Currently in the gpkg output folder, every single site, magnitude and interval creates a new file.
With the bug from 4.4.0.0/4.5.2.11 for stage for intervals, we are now expecting more interval and an even larger outputs. Split the final output file in HUC2.
Plan: