Open CyprienBosserelle opened 8 months ago
Added a new variable savebyblk
input parameter. default is on
if set to off
it will save per level which is a lot faster but that require enough memory to allocate the full high res level.
No testing has been done for zoneoutput
This is on-hold until testing with new bnd complete
Save to file takes too long for many blocks
The problem
The current dev branch save each block independantly to the netcdf file. The bottleneck is the netcdf write function overhead which make writing large model (20000 blocks and more) quite slow.
The solution
To alleviate this needs a new function that combines all blocks in a single array (for each level/variable) and write the whole array at once to the netcdf file.
Hold on
While this will be faster, it is likely inefficient when there are few arrays in a level. Typically when the highest level has only a few block we would still be writing a full array.
Memory is cheap?
This will also be impossible if going to very high level where we won't be able to allocate memory to hold the highest level array.
The real solution
Something that combine both option above and decide what is the best compromise.
Collapse array
This may be a good time to introduce collapsing level to a single array and saving it to disk.
Other things
This branch also includes: