bug: quad_utils_mod out-of-range negative for MOM6

[hkershaw-brown]

:bug: Your bug may already be reported! Please search on the issue tracker before creating a new issue.

Describe the bug

1. Run perturb_single_instance with MOM6 global grid TL319_t232
2. perturb a single file to get ensemble
3. error in static_init_model -> setup_interpolation -> quad_utils_mod

Error Message

 ERROR FROM:
 source : $URL$
 routine: update_reg_list
 message: unable to find right box
 message: ...  index_x may be out-of-range:            1 -2147483647         18
 0
  message: ...  index_y may be out-of-range:            1 -2147483647         18
 0

Looks like an integer overflow. Ran grid T62_g16 (lower res) to completion.

https://github.com/NCAR/DART/blob/924182f29ce23f2a4e4c53aa57cc7d00fdccf374/models/utilities/quad_utils_mod.f90#L1323-L1329

Which model(s) are you working with?

MOM6

Version of DART

Which version of DART are you using? You can find the version using git describe --tags

v11.5.0-1-g4e624120f

Have you modified the DART code?

Yes, changed perturb_single_instance to do a perturb (rather than calling model_mod.f90 perturb routine or exiting).

https://github.com/hkershaw-brown/DART/tree/perturb-for-testing

Build information

Please describe:

1. Derecho
2. intel

[hkershaw-brown]

slide aside: the model interpolation setup might be better lazily called when the first call to model_interpolate is made. There are many programs that call static_init_model but don't do any interpolation so no need to setup these interpolation structures.

[hkershaw-brown]

the problem seems to be before quad-utils, in subroutine nc_get_double_2d

Not sure if this is the calling code or netcdf funkyness.

The ocean geometry file looks ok:

[hkershaw-brown]

scratch that, I don't think geolong should have so many zeros in. geolon from the static file has missing values -> 1.e+20 which is blowing up quad-utils.

 float geolon(yh, xh) ;
                geolon:_FillValue = 1.e+20f ;
                geolon:missing_value = 1.e+20f ;

data:

 geolon =
  _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, 
    _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, 
    _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, 
    _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, 

closing since I think, unfortunately, this is not a dart bug.

[hkershaw-brown]

Reopening since MOM6 has missing values in the grid in cesm3_0_alpha03a

Note ocean_geometry.nc 'wet'

[hkershaw-brown]

Also need to sanitize [0,360) longitude for dart.

We have files with longitude > 360 but also now, TL319_t232 has:

min(geolon) = -286.9998 max(geolon) = 72.9998

[hkershaw-brown]

If you set the missing values to a random land point, you end up filling in the regular box with that point and using a bunch of memory.

617 ! mom6 has missing values in the grid
618 where (geolon == 1.0e+20) geolon = 75.51
619 where (geolon_u == 1.0e+20) geolon_u = 75.51
620 where (geolon_v == 1.0e+20) geolon_v = 75.51

PE 0: init_irreg_interp  to determine (minimum) max_reg_list_num values for new
  grids ...
 PE 0: init_irreg_interp ...  interp_handle%ii%grid_num is        61449
 PE 0: init_irreg_interp  to determine (minimum) max_reg_list_num values for new
  grids ...
 PE 0: init_irreg_interp ...  interp_handle%ii%grid_num is        61453
 PE 0: init_irreg_interp  to determine (minimum) max_reg_list_num values for new
  grids ...
 PE 0: init_irreg_interp ...  interp_handle%ii%grid_num is        61448

[hkershaw-brown]

Is the mask the same for geo{lon|lat}, geo{lon|lat}_u, geo{lon|lat}_v?

geo{lon|lat}, geo{lon|lat}_u, geo{lon|lat}_v seem to have missing values in the same place, however these do not match ocean_geometry.nc (wet).