bug: MOM6 obs too deep

[hkershaw-brown]

:bug: 🐘 Your bug may already be reported! Please search on the issue tracker before creating a new issue.

Describe the bug

There is no catch after the find layer loop to check whether the observation is too deep.

https://github.com/NCAR/DART/blob/502af7865e258530cba7c7daff6aed4390cbb649/models/MOM6/model_mod.f90#L286-L328

There is an OBS_TOO_DEEP status, but it is never used: https://github.com/NCAR/DART/blob/502af7865e258530cba7c7daff6aed4390cbb649/models/MOM6/model_mod.f90#LL120C22-L120C35

1. List the steps someone needs to take to reproduce the bug.
   MOM6 with WOD data base obs 3 ensemble member CESM run here: /glade/scratch/hkershaw/c.T62_g16.Alper.ens3/run note the ensemble members are identical.
2. What was the expected outcome? Correct forward operator result and status
3. What actually happened?
   Obs too deep not caught.

Error Message

Please provide any error messages. You get errors trying to access incorrect memory, e.g. PE 0: filter trace: Before computing prior observation values

mlx5: r5i3n3: got completion with error:
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000e9 00000000 00000000 00000000
00000000 00008813 100036c4 03421ad2

or

forrtl: severe (408): fort: (3): Subscript #1 of the array PE_TO_TASK_LIST has value -278 which is less than the lower bound of 1

Image              PC                Routine            Line        Source             
filter             0000000000BEB1FF  Unknown               Unknown  Unknown
filter             0000000000649C66  ensemble_manager_         966  ensemble_manager_mod.f90
filter             00000000005CBDDB  distributed_state         104  distributed_state_mod.f90
filter             00000000005CB635  distributed_state          79  distributed_state_mod.f90
filter             00000000005CAE90  distributed_state          54  distributed_state_mod.f90
filter             000000000066FECF  model_mod_mp_mode         342  model_mod.f90
filter             0000000000475E2F  obs_def_mod_mp_ge        1342  obs_def_mod.f90
filter             00000000004167C0  forward_operator_         447  forward_operator_mod.f90
filter             000000000040EAB1  forward_operator_         283  forward_operator_mod.f90
filter             00000000004E6152  filter_mod_mp_fil         864  filter_mod.f90
filter             00000000005ECC73  MAIN__                     20  filter.f90

Which model(s) are you working with?

MOM6

Screenshots

The obs depth is greater than the total depth. Note I think the total depth might be incorrect (edit will update later see comment below), but still the obs_too_deep should be caught.

Version of DART

Which version of DART are you using? You can find the version using git describe --tags
v10.7.2-2-g5f3235bbc

Have you modified the DART code?

Yes/No
If your code changes are available on GitHub, please provide the repository. https://github.com/NCAR/DART/tree/mom6-scripting

Build information

Please describe:

1. Cheyenne
2. intel

[hkershaw-brown]

note on obs_depth. This is an obs that is failing because it is too deep:

obs_depth is 450m, model depth is ~284m

So I believe the depth calculated in model_mod is correct. Is it a large difference in depth (obs vs. model) because of the low model resolution I'm using?

caveat this todo for the u and v grid variables: https://github.com/NCAR/DART/blob/502af7865e258530cba7c7daff6aed4390cbb649/models/MOM6/model_mod.f90#L283

cat obs_seq.out
 obs_sequence
obs_kind_definitions
          13
          15 FLOAT_SALINITY                 
          16 FLOAT_TEMPERATURE              
          23 GLIDER_SALINITY                
          24 GLIDER_TEMPERATURE             
          27 MOORING_SALINITY               
          28 MOORING_TEMPERATURE            
          30 BOTTLE_SALINITY                
          31 BOTTLE_TEMPERATURE             
          32 CTD_SALINITY                   
          33 CTD_TEMPERATURE                
          43 XBT_TEMPERATURE                
          46 APB_SALINITY                   
          47 APB_TEMPERATURE                
  num_copies:            1  num_qc:            1
  num_obs:       1   max_num_obs:        1
WOD observation                                                 
WOD QC                                                          
  first:            1  last:        1
 OBS            1
  3.433300018310547E-002
  0.000000000000000E+000
       -1       -1          -1     
obdef
loc3d
     2.247111411942699        0.5542118456182733         425.0000000000000      3      
kind
          32     
 67392     151245 
  2.500000000000000E-007