Open ndkeen opened 5 months ago
@jayeshkrishna do you know how big of an impact we'd have if we flushed the output file after every write? I'm assuming it's non negligible, but maybe still relatively small?
Edit: I don't mean "after each write_darray
call", but rather "after all the write_darray
and put_var
calls within a timestep"...
@bartgol , @jayeshkrishna I want to bring this issue back to life. We had a discussion about this in the eval call today.
@crterai can you comment briefly on how this impacted the CESS sims?
We had portions of the Cess production run that we're having to re-run because we are missing outputs from certain periods. We got this when the model crashed pretty close to a restart write and one of the output files was still filling up but hadn't flushed. And example is in
/lustre/orion/cli115/proj-shared/noel/e3sm_scratch/cess-oct2/cess-control.ne1024pg2_ne1024pg2.F2010-SCREAMv1.cess-oct2/run
2020-04-20-03600
the output.scream.Cess.hourly2DVars.
output stream started writing a new file. 2020-04-22-00000
a restart was written, but the output.scream.Cess.hourly2DVars.
output stream wasn't flushed. 2020-04-22 17:21:40
the model crashed and output.scream.Cess.hourly2DVars.INSTANT.nhours_x1.2020-04-20-03600.nc
remained empty because the output hadn't flushed. 2020-04-22-00000
. At that point, we started writing to a new file output.scream.Cess.hourly2DVars.INSTANT.nhours_x1.2020-04-22-03600.nc
. That left us missing the data for 2020-04-20-03600
to 2020-04-22-00000
.
I recently had a case running (happened to be PPE member) that was beyond second day and I needed to cancel it, thinking we already had the data written for that second day. Afterwards, looking at data, the file is there, but empty.
In atm.log, it does indicate we are "done" with the file:
@bartgol explains that it might be scorpio not flushing and we have some control by adding
flush_frequency: 1
to the yamls.There must be some perf impact of doing this, but unless it's severe, I would think we would generally want this? Could it actually explain why some of the data from Cess sims are missing?