hdf5 file sizes are enormous

[rburghol]

See my issue on RESPEC https://github.com/respec/HSPsquared/issues/94 When using hsp2 import all uci files and supporting wdm data are imported into an h5 database. Before even running the model, the resulting h5 file is nearly 200x larger than the component files. See Test 1-4 below for file details and commands.

Questions:

• Does the wdm/uci file source matter?
  • Test 1: 135x inflation - test10.h5 from respec repo yields 27M from 200k of source files. See Test 1 below for details.
  • Test 2: 142x inflation - cbp Phase 5.3.2 river RCHRES only. See Test 2 below for details.
  • Test 3: 20x inflation - cbpPhase 5.3.2 land PERLND only. See Test 3 below for details.
  • Conclusion: yes, wd,/ucifile source may matter, as 1 of 3 tests had significantly less inflation.
• Is this due to the installed version? See: https://github.com/respec/HSPsquared/issues/36
  • Compiled 1.13.1 from source on deq1:
  • repeated import_uci with CBP model (999M) and test10 (27M) from RESPEC- Does the UCI/WDM of interest matter?
  • wdm + uci file are ~200k, resulting h5 file is 27M, so increase of 135x
  • Conclusion: No.
• Does running the model matter?
  • No. All models were run after initial tests and h5 file sizes were generally less than 1% increased in size.

Test 1: Importing test10.uci with hdf5 version 1.10.4

hsp2 import_uci test10.uci test10.h5
ls -lhrt
total 29M
-rw-rw-r-- 1 rob rob 160K Jun 17 13:00 test10.wdm
-rw-rw-r-- 1 rob rob  36K Jun 17 13:00 test10.uci
-rw-rw-r-- 1 rob rob  27M Jun 17 13:02 test10.h5

Test 2: Importing UCI from Chesapeake Bay Model 5.3.2 single RCHRES only (no PERLND).

rm OR1_7700_7980.h5
hsp2 import_uci OR1_7700_7980.uci
ls -lhrt
total 1006M
-rwxrwxr-x 1 rob rob 480K Jun 13 20:40 prad_A51037.wdm
-rwxrwxr-x 1 rob rob 4.3M Jun 13 20:40 met_A51037.wdm
-rwxr-xr-x 1 rob rob  80K Jun 13 20:41 ps_sep_div_ams_p532sova_2021_OR1_7700_7980.wdm
-rw-rw-r-- 1 rob rob  11K Jun 15 16:19 OR1_7700_7980.uci
-rwxr-xr-x 1 rob rob 1.7M Jun 15 20:24 OR1_7700_7980.wdm
-rw-rw-r-- 1 rob rob 999M Jun 17 11:52 OR1_7700_7980.h5

Test 3: Importing UCI from Chesapeake Bay Model 5.3.2 single PERLND only (no RCHRES).

rm forA51800.h5
hsp2 import_uci forA51800.uci
ls -lhrt 
total 825M
-rwxrwxr-x 1 rob rob 4.3M Jan 14 15:56 met_A51800.wdm
-rwxrwxr-x 1 rob rob 520K Jan 14 15:56 prad_A51800.wdm
-rwxr-xr-x 1 rob rob  14M Jan 14 15:57 forA51800.wdm
-rw-rw-r-- 1 rob rob 7.0K Jan 18 14:43 forA51800.uci
-rw-rw-r-- 1 rob rob 383M Jan 21 17:59 forA51800.h5

Test 4: Importing test10.uci with hdf5 version 1.13.1 - same file size as with version 1.10.4.

rm test10.h5
hsp2 import_uci test10.uci test10.h5
ls -lhrt 
total 29M
-rw-rw-r-- 1 rob rob 160K Jun 17 13:00 test10.wdm
-rw-rw-r-- 1 rob rob  36K Jun 17 13:00 test10.uci
-rw-rw-r-- 1 rob rob 2.0M Jun 17 13:00 TEST10_hsp2_compare.ipynb
-rw-rw-r-- 1 rob rob  27M Jun 17 13:02 test10.h5