search

GEOS-ESM / MAPL

MAPL is a foundation layer of the GEOS architecture, whose original purpose is to supplement the Earth System Modeling Framework (ESMF)
https://geos-esm.github.io/MAPL/
Apache License 2.0
27 stars 17 forks source link

Can we chunk checkpoints? #1988

Open mathomp4 opened 1 year ago

mathomp4 commented 1 year ago

Recently, @aoloso tried to run ncks on a C2880 restart (compress) with a command like:

ncks \
  -L 1 \
  -O DYAMONDv2_c2880_L181.moist_import_rst.e20200206_21z.GEOSgcm-v10.16.3.Icarus-NLv3_Ostia_CF2880x6C_CF2880x6C.nc4 \
  output/DYAMONDv2_c2880_L181.moist_import_rst.e20200206_21z.GEOSgcm-v10.16.3.Icarus-NLv3_Ostia_CF2880x6C_CF2880x6C.nc4

and he got:

nco_def_var_chunking(): ERROR Total requested chunk size = 36229939200 exceeds netCDF maximium-supported chunk size = 4294967295
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_def_var_chunking()
nco_err_exit(): ERROR Error code is -127. Translation into English with nc_strerror(-127) is "NetCDF: Bad chunk sizes."
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)

My suggestion was to add some chunking with ncks:

--cnk_dmn lat,720 --cnk_dmn lon,720 --cnk_dmn lev,1

so that ncks would be happy.

But I wondered: do we have the ability to chunk checkpoints like we do history output? If not, maybe we might want to add that as we increase resolution?

@bena-nasa would know if this would be doable in MAPL2 or if it might be a MAPL3 thing for @tclune to plan for..

bena-nasa commented 1 year ago

No, we currently do not support chunking of checkpoints, no technical reason in principle we could not, but someone would have to implement it and figure out how to express the desire to do this in the AGCM.rc file. MAPL3 won't solve this, I'm not aware of any plans to rewrite the checkpoint/restart IO layer in MAPL3...

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. If there are no updates within 7 days, it will be closed. You can add the "long term" tag to prevent the Stale bot from closing this issue.