search

metno / snap

Severe Nuclear Accident Programme
GNU General Public License v3.0
23 stars 6 forks source link

netcdf SWMR compatible writing #108

Open heikoklein opened 1 year ago

heikoklein commented 1 year ago

SNAP currently opens the output netcdf-file for each time-step in write-append mode and closes it again after the output-timestep is written. With netcdf3 and early netcdf4 implementations this allowed files to be read while the model was still running.

The new hdf5.10 SWMR (single write multiple read) implementation https://support.hdfgroup.org/HDF5/docNewFeatures/SWMR/Design-HDF5-FileLocking.pdf explicitly states that files which are opened for reading can no longer be opened for writing. This leads to the SNAP model crashing when one starts to analyze the model-results while the model is still running.

The updated implementation should keep the write file-handle of the output netcdf-file open for the complete time the model is running. It should also ensure that the output is synced to disk after the output-timestep so all information is available to all readers. The SWMR mode should be the default, while the old writer should still be kept available.

magnusuMET commented 1 year ago

See progress for this feature in netcdf-c here: https://github.com/Unidata/netcdf-c/pull/1448

heikoklein commented 1 year ago

The implementation of this feature can already start independently of the availability in netcdf-c (though full functionality requires netcdf-c)