search

geoschem / geos-chem

GEOS-Chem "Science Codebase" repository. Contains GEOS-Chem science routines, run directory generation scripts, and interface code. This repository is used as a submodule within the GCClassic and GCHP wrappers, as well as in other modeling contexts (external ESMs).
http://geos-chem.org
Other
168 stars 165 forks source link

[FEATURE REQUEST] Remove buffering in history NetCDF output #482

Open ltmurray opened 4 years ago

ltmurray commented 4 years ago

I've noticed that the History diagnostic — at least on my systems — buffers its output when writing to disk using GEOS-Chem Classic 12.9.3.

For example, if I request monthly restart files for a simulation beginning Jan 1, then on Feb 1, the model creates the Feb 1 restart file with its associated coordinate variables, but does not actually populate the file with the variables until it creates the next restart file on Mar 1. The exception is when the model gracefully stops, all outstanding data is written to disk. See an example as evidenced by the file size from an ongoing run below.

[prompt] ls -ltr GEOSChem.Restart*
 13293123 Oct 14 11:00 GEOSChem.Restart.19500101_0000z.nc4
 13261488 Oct 14 11:45 GEOSChem.Restart.19500201_0000z.nc4
 13261342 Oct 14 11:58 GEOSChem.Restart.19500301_0000z.nc4
    44529 Oct 14 11:58 GEOSChem.Restart.19500401_0000z.nc4

Would it be possible to get the model to write to disk the full file when the averaging interval is complete? The same happens for the rest of the history diagnostics (e.g., Feb monthly mean SpeciesConc data is not written to disk until April 1), but does not happen for the HEMCO diagnostics.

yantosca commented 1 year ago

This has been done in PR #1464, which went into GEOS-Chem 14.0.2. We can close out this issue now.

lizziel commented 1 year ago

I am reopening this feature request since it is still relevant for all history collections except Restart. I am running into this problem now as I try to view daily average files as soon as the simulation gets past the time-averaging period. For example, I am trying to view a daily average for July 1, 2019 when the simulation reaches July 2, 2019 01:00:00. Because of the issue it is not possible to view the data until July 3, 2019 00:00:00.