search

CH-Earth / summa

Structure for Unifying Multiple Modeling Alternatives:
http://www.ral.ucar.edu/projects/summa
GNU General Public License v3.0
80 stars 105 forks source link

Improve output configuration #382

Closed arbennett closed 4 years ago

arbennett commented 4 years ago

This pull request adds 2 new features. The first, is the ability to add outPrecision to the output control file to specify the output precision as either single or double precision. This can be set as:

outputPrecision | single
outputPrecision | float
outputPrecision | double

where single and float both are single-precision floating point numbers and double is double precision floating point numbers.

Additionally this defines the chunking for output variables in an explicit way. This helps to speed up writeout for smaller simulations as well as maintains a reasonable file size for larger simulations. As far as my testing went, setting the maximum chunksize for any variable type to ~1000 worked well, but this might be explored further in the future.

andywood commented 4 years ago

Nice work, Andrew. Having also explored both of these factor (float v double output and chunksizes), I think it's great to make these changes. That said, it would be good to get some more info/benchmarking on the choices for settings (eg 480, 960 etc) for how things are chunked in this commit. For the small runs the increase in time chunksize makes a huge difference, but I've also seen it be problematic in file sizes for really large hru-dim runs.

Aside from this I'd suggest making the the time & hru/gru global chunksizes into params that could be set in the 'localParamInfo' (ie 'globalHruParamInfo') in the same section as the balance convergence tolerances. To optimize for a given application (eg small scale runs vs large domain runs), a user could just change those settings.

Also, btw, if 'single' and 'float' lead to the same outcome, why not just make it 'single' versus' 'double' as options? or 'float' versus 'double'?

arbennett commented 4 years ago

I think it's probably worth doing a bit of benchmarking on a really large run before this is merged, and think we could do some balancing if the number of HRU or GRU is > 1000. In that case we might do something like equally balance the timeChunk and gruChunk/hruChunk so that both have the same number of chunks. That way both timeseries and spatial operations are somewhat balanced in terms of processing time.