division by zero in ocn_time_integration_split_init

[xylar]

When using gcc/7.4.0 on Grizzly (see full details below), I'm getting a division by zero error on this line: https://github.com/MPAS-Dev/MPAS-Model/blob/ocean/develop/src/core_ocean/mode_forward/mpas_ocn_time_integration_split.F#L2156

Full stack trace

``` #0 0x2ad2ea4093ff in ??? #1 0x8e7fbe in __ocn_time_integration_split_MOD_ocn_time_integration_split_init at /users/xylar/climate/mpas_work/compass/compass/MPAS-Model/ocean/develop/src/core_ocean/mode_forward/mpas_ocn_time_integration_split.F:2272 #2 0x8e421d in __ocn_forward_mode_MOD_ocn_forward_mode_init at /users/xylar/climate/mpas_work/compass/compass/MPAS-Model/ocean/develop/src/core_ocean/mode_forward/mpas_ocn_forward_mode.F:265 #3 0xa89389 in __ocn_core_MOD_ocn_core_init at /users/xylar/climate/mpas_work/compass/compass/MPAS-Model/ocean/develop/src/core_ocean/driver/mpas_ocn_core.F:76 #4 0x40c321 in __mpas_subdriver_MOD_mpas_init at /users/xylar/climate/mpas_work/compass/compass/MPAS-Model/ocean/develop/src/driver/mpas_subdriver.F:331 #5 0x4090af in mpas at /users/xylar/climate/mpas_work/compass/compass/MPAS-Model/ocean/develop/src/driver/mpas.F:14 #6 0x409110 in main at /users/xylar/climate/mpas_work/compass/compass/MPAS-Model/ocean/develop/src/driver/mpas.F:10 ```

Additionally please enter this information if applicable:

• Compiler and version: gcc/7.4.0
• MPI implementation and version: openmpi/2.1.2
• NetCDF version: netcdf-h5parallel/4.7.3
• Parallel NetCDF version: pnetcdf/1.11.2
• Parallel IO version: scorpio/1.1.5
• Output of uname -a: Linux gr-fe1.lanl.gov 3.10.0-1127.18.2.1chaos.ch6.x86_64 #1 SMP Thu Jul 30 12:53:28 PDT 2020 x86_64 x86_64 x86_64 GNU/Linux
• The make line used to build the executable:

  export CORE=ocean
  export USE_PIO2=true
  export AUTOCLEAN=true
  export  HDF5_USE_FILE_LOCKING=FALSE
  export LD_LIBRARY_PATH=${NETCDF}/lib:${LD_LIBRARY_PATH}
  make gfortran DEBUG=true

• The line used to run the executable: (compass nightly regression suite on compass/master)
• Test case used: ocean/baroclinic_channel/10km/default/forward

[xylar]

I spent some time today trying to figure this one out. It seems like the division by zero is not the source of the problem. Instead, cellsOnEdge seems to be messed up (full of nCells + 1 entries) and this leads to numerous other issues. It may be that other fields are also not correct. It will be hard to know if this is a SCORPIO, compiler, MPI or other issue.

I'm not good enough with the MPAS framework to make much headway once the problem seems to be there.