How to best handle big input files in SCREAM

[jgfouca]

I think it's a no brainer that we want to use the E3SM input data file server so that we don't have to put big binary files directly in the SCREAM repo.

The next question is, how do we want to manage these large files in the build directories of SCREAM?

Current approach: Download (if needed) and copy the file from SCREAM_INPUT_ROOT to ${CMAKE_CURRENT_BINARY_DIR}/data

Pros: each test dir uses its own files, so the user could tweak these files without impacting other tests. Cons: many of the tests use the same files, so there's duplicate copies in many places in the build which increases disk usage

Potential change 1: Have a data directory that all of SCREAM uses, ${CMAKE_BINARY_DIR}/data

Pros: this would remove all duplicates, reducing dis usage of builds Cons: you'd lose the ability to tweak files for individual tests (unless you used a new file name)

Potential change 2: Same as current approach except use symlinks instead of copies.

Pros: This should reduce disk usage the most of all the approaches and preserves the users ability to customize files for specific tests. Cons: Sometimes compute nodes will not mount the file system that contains SCREAM_INPUT_ROOT, so the links would not work.

[bartgol]

I think the drawback you pointed out for option 2 might be a deal breaker. Option 1 seems like a good compromise, imho.

[jgfouca]

@bartgol , yeah, option 2 would probably require a SCREAM_NO_INPUT_SYMLINK machine cache setting.

[PeterCaldwell]

I think E3SM and CESM solve this by just having a big public space where all their inputdata is stored, right? So "Potential change 1" is just doing what the rest of E3SM does?

Also, I can't think of a case where one would want to "tweak files for individual tests" - can one of you? Or do you mean that we'd never be able to use an input file which isn't in its standard location? I think E3SM handles this by grabbing from the default inputdata location unless your namelist/yaml file explicitly provides an absolute path and filename to instead grab the file... in which case we should just read from that location rather than making a copy anywhere.

Does this all make sense? Am I missing something? I think inputdata file handling is actually one of the things that CESM/E3SM does best.

[jgfouca]

@PeterCaldwell , @bartgol , even after all these years, I still get mixed up on how some aspects of CIME work.

It looks like CIME does not copy the input files into the case as I thought:

Buildconf>% cat cpl.input_data_list 
atm2ocn_fmapname = /sems-data-store/ACME/inputdata/cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gx1v6_aave.130322.nc
atm2ocn_smapname = /sems-data-store/ACME/inputdata/cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gx1v6_blin.130322.nc
atm2ocn_vmapname = /sems-data-store/ACME/inputdata/cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gx1v6_patc.130322.nc
ocn2atm_fmapname = /sems-data-store/ACME/inputdata/cpl/gridmaps/gx1v6/map_gx1v6_TO_fv1.9x2.5_aave.130322.nc
ocn2atm_smapname = /sems-data-store/ACME/inputdata/cpl/gridmaps/gx1v6/map_gx1v6_TO_fv1.9x2.5_aave.130322.nc
atm2ice_fmapname = /sems-data-store/ACME/inputdata/cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gx1v6_aave.130322.nc
atm2ice_smapname = /sems-data-store/ACME/inputdata/cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gx1v6_blin.130322.nc
atm2ice_vmapname = /sems-data-store/ACME/inputdata/cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gx1v6_patc.130322.nc
ice2atm_fmapname = /sems-data-store/ACME/inputdata/cpl/gridmaps/gx1v6/map_gx1v6_TO_fv1.9x2.5_aave.130322.nc
ice2atm_smapname = /sems-data-store/ACME/inputdata/cpl/gridmaps/gx1v6/map_gx1v6_TO_fv1.9x2.5_aave.130322.nc
rof2ocn_liq_rmapname = /sems-data-store/ACME/inputdata/cpl/cpl6/map_rx1_to_gx1v6_e1000r300_090318.nc
rof2ocn_ice_rmapname = /sems-data-store/ACME/inputdata/cpl/cpl6/map_rx1_to_gx1v6_e1000r300_090318.nc

^ This is the input file list for the coupler for a simple case. You can see that it loads the files directly out of the DIN_LOC_ROOT (SCREAM_INPUT_ROOT for us). I've looked through the code and confirmed that under no circumstances does CIME ever copy these kinds of files.

I think we could mimic this approach in standalone SCREAM if you guys are fine with it.

[bartgol]

That's great. This would completely eliminate space concerns.

[PeterCaldwell]

Yeah, I also vote for mimicking E3SM for inputdata. Thanks Jim!

[bartgol]

This was fixed in #1391.