Supergrid input buffer size does not match read_data buffer in MOM OM4_025 test case

[wrongkindofdoctor]

Describe the bug The first call to the new read_data routines via the MOM wrappers from set_grid_metrics_from_mosaic /MOM_grid_initialize.F90#L273) when using the MOM6-examples/ICE_OCEAN_SIS2/OM4_025 test with the new IO generates the following error: FATAL from PE 256: NetCDF: Start+count exceeds dimension bound This occurs because MOM is passing an array 'TMPZ' that is the size of the data domain. To Reproduce Compile MOM6+new IO, SIS2+new IO, FMS 2020.01, FMScoupler 2020.01 with intel 18.0.6 in debug on gaea, and run the OM4_025 test experiment with 480 pes. The model should crash when trying to read in 'x' from INPUT/ocean_hgrid.nc

Expected behavior The data should be read into the buffer via the 2D domain-decomposed read_data interfaces. This is not an issue with small pe runs in the ocean_only cases and simple SIS2 cases.

System Environment Describe the system environment, include:

• OS: Linux 4.4.143-94.47.1.16060.2.PTF.1107299-default x86_64
• Compiler(s): Intel 18.0.6
• MPI type, and version: Cray-mpi
• netCDF Version:4.4.0
• Configure options: -Duse_libMPI -Duse_netCDF -DSPMD -Duse_AM3_physics -D_USE_LEGACYLAND Additional context stack trace:

  00000000054BEC1D  mpp_mod_mp_mpp_er          69  mpp_util_mpi.inc
  MOM6               0000000004D819BB  fms_io_utils_mod_         145  fms_io_utils.F90
  MOM6               0000000004D0BA74  netcdf_io_mod_mp_         236  netcdf_io.F90
  MOM6               0000000004D39779  netcdf_io_mod_mp_         304  netcdf_read_data.inc
  MOM6               000000000522ABB8  fms_netcdf_domain         217  domain_read.inc
  MOM6               0000000002C41C74  mom_io_mp_mom_rea        2157  MOM_io.F90
  MOM6               0000000001756D63  mom_grid_initiali         273  MOM_grid_initialize.F90
  MOM6               000000000174C365  mom_grid_initiali          89  MOM_grid_initialize.F90
  MOM6               00000000021DDA6D  sis_fixed_initial          65  SIS_fixed_initialization.F90
  MOM6               0000000003A521B0  ice_model_mod_mp_        2075  ice_model.F90
  MOM6               0000000000B75AC1  coupler_main_IP_c        1764  coupler_main.F90
  MOM6               0000000000B6190E  MAIN__                    614  coupler_main.F90

This is another gitlab-CI case, so contact me offline for run directories, executables, etc... @marshallward @menzel-gfdl

[wrongkindofdoctor]

To add a bit of context, set_grid_metrics_from_mosaic has 4 temporary arrays (tmpU, tmpV, tmpZ tmpT) that are allocated as 2*(size of data domain) +/- some offset depending on the type of grid (C, B, etc...) they are mapping to. To avoid performance issues, this routine reads 'x', from ocean_hgrid.nc into the tmpZ , and maps the values to to G%geoLonT, G%geoLonBu, G%geolonCV, G%geolonCu instead of doing 4 separate read_data calls with different sized buffers. Similar procedures exist for 'y', 'dx', 'dy'.

In lieu of futzing with the MOM_io wrappers and adding more conditionals for this case (and others that may arise in CM and ESM configurations), one possible solution could simply be to read in the global data array using the non-domain-decomposed interfaces, and map them to the data domain indices in the temporary array, but I would need assistance from @marshallward or @nikizadehgfdl to do this correctly since I am not the expert on MOM's indexing conventions.

I'll add that the prevailing sentiment is that fms-io should "just know" what users want based on the domain description and array size. This is possible provided users pass in standard sized data buffers and let the mpp routines handle the indexing, pe lists, etc... However, data arrays that have user-specified mpp procedures "pre-applied" so-to-speak will cause the interfaces to choke as is the case here. Moving forward, developers should aim to read/write data into a simple arrays (e.g., see how tmpGlbl is used here) , THEN slice-and-dice as needed in their own routines if necessary. looping @menzel-gfdl into the discussion, as well.

[menzel-gfdl]

The I/O is only set up to do domain-decomposed reads into buffers that are either the size of the compute domain, or the size of the compute domain + size of the halos. I don't think 4 reads vs. 1 read is really anything to worry about performance-wise, unless these reads are happening over and over again.

[marshallward]

I'm sorry that I haven't answered, as this isn't something that I have looked at before or know much about. But I am guessing the issue is because you are reading x on a supergrid into tmpZ, and the new IO is tripping up because the grid is an unexpected shape, is that right?

Is it possible to just read in the local segment as ungridded data and then fill in the tmp* variables?

[wrongkindofdoctor]

@marshallward Correct. I can read in the segment instead of the whole array since the new IO handles start and count arguments, but I'll probably need help getting the indexing correct. Stay tuned.