BUMP sub-grid visible in the increment

guillaumevernieres commented 1 year ago

Description

We need to use a higher order interpolation between the analysis grid and the bump sub-grid, this may fix the issue illustrated in the figure below (courtesy of @liuxiao37k ):

BUMP sub-grid is clearly seen in the EnVAR increment (bottom panel).

Possible solutions

Some pointers from @danholdaway and Benjamin M:

nicas:
  interpolation type:
  - variables: [common]
    type: si

si stands for smooth interpolation and it went in with this PR: https://github.com/JCSDA-internal/saber/pull/357 You can have a look here: https://github.com/JCSDA-internal/saber/pull/389/files though and search on “: si” to see where it is actually used in the YAMLs. There are two places it’s used: sampling interpolation and nicas interpolation. I think you want nicas.

More from @benjaminmenetrier .To save the nicas grid: _You can activate the key "write nicas grids: true" in the "drivers" section.

bump:
  io:
    data directory: bump
  drivers:
    multivariate strategy: specific_univariate
    compute nicas: true
    write local nicas: true
    write nicas grids: true
    [...]

You'll get an additional file for each MPI task ("nicas_grids" pattern in the file name), with the lon/lat/lev of each subgrid point. You might be interested in the points owned by each task (halo A in BUMP conventions), so look for variables "lon_sa", "lat_sa" and "levsa".

From Benjamin:

[...] But the real reason behind this issue is probably that the resolution of the NICAS subgrid is too small compared to the desired length-scale. You could increase both "nicas/resolution" and "nicas/max horizontal grid size". It is important to check the effective resolution in the BUMP log, because it can be smaller than "nicas/resolution" if "nicas/max horizontal grid size" is too small.

Tasks

[ ] Change the interpolation type to si
[x] increase both "nicas/resolution" and "nicas/max horizontal grid size"
[x] 3dvar with the same scales/parameters used in the localization
[x] EnVAR with no localization

guillaumevernieres commented 1 year ago

Task #2 above is being tested here:

/scratch2/NCEPDEV/ocean/Guillaume.Vernieres/runs/staticb-test

using resolution: 10

guillaumevernieres commented 1 year ago

Update from @liuxiao37k : Testing the envar with no localization to make sure the issue is isolated to bump ...

guillaumevernieres commented 1 year ago

Figures to come soon, but increasing c1max and resolution did not do the trick (work done by @liuxiao37k ). parameters used:

nc1max: 50000
resol: 10

danholdaway commented 1 year ago

@guillaumevernieres does this grid imprinting happen when you run a 3DVar? Or is it only seen with EnVar with localization turned on? What about hybrid?

liuxiao37k commented 1 year ago

@guillaumevernieres does this grid imprinting happen when you run a 3DVar? Or is it only seen with EnVar with localization turned on? What about hybrid?

@danholdaway Good question. I'll run a cycle of 3DVar with the same Obs, ICs, BUMP resol, and do a zoom-in comparison of increments side by side with EnVar. Thanks. I have not tried hybrid methods yet.

guillaumevernieres commented 1 year ago

@guillaumevernieres does this grid imprinting happen when you run a 3DVar? Or is it only seen with EnVar with localization turned on? What about hybrid?

@danholdaway , we've seen the issue with the 3dvar as well.

guillaumevernieres commented 1 year ago

After a quick chat with @benjaminmenetrier , we are probably not going to resolve this issue without: 1 - increasing the horizontal scales 2 - increasing nc1max so we have enough points in the delaunay grid to resolve the scales from above 3 - put the resolution back to something around 6 (~6 points to represent the scales above), which should be sufficient

liuxiao37k commented 1 year ago

I think it works!

nc1max: 150000
resol: 6
length scale: 10x before, about 300km-1000km

guillaumevernieres commented 1 year ago

Mos excelent @liuxiao37k and thanks for pointing us the right direction @benjaminmenetrier ! How long did the initialization take @liuxiao37k ?

liuxiao37k commented 1 year ago

Mos excelent @liuxiao37k and thanks for pointing us the right direction @benjaminmenetrier ! How long did the initialization take @liuxiao37k ?

@guillaumevernieres 15 mins, so I think we have room to increase resol and/or decrease length scales. nc1max=150000 didn't seem to place a constraint because estimated nc1 was around 40000 under this setting and didn't get reset to nc1max.

--- Block: tocn-tocn
       Component: 1
          Compute adaptive sampling, vertical
             Compute vertical subset L1
             Effective levels:   1
          Compute adaptive sampling, horizontal
             Compute vertically dependent horizontal subsampling:
                Sublevel   1/  1 (level   1):
                   Horizontal support radius:     645.19 km (    218.11 km -     993.50 km)
                   Estimated nc1 from horizontal support radius:    36075
                   Final nc1:    36075
                   Compute horizontal subset Sc1:
                   Decimate full grid, at least    36075 points required
                     290617 valid points found

          Parameters for processor #     1
             nc0 =         1555200
             nc0a =           3240
             nl0 =              75
             nc1(  1) =      36075
             ns =            36075
             nsa =               0
             nsb =               0
             nsc =               0
             interp_c1b_to_c0a(  1)%n_s =        0
             v%n_s =             1
             c%n_s =             0

guillaumevernieres commented 1 year ago

From @benjaminmenetrier about saving the grid:

    drivers:
      multivariate strategy: specific_univariate
      read local nicas: true
      write nicas grids: true

_You can activate the key "write nicas grids: true" in the "drivers" section. You'll get an additional file for each MPI task ("nicas_grids" pattern in the file name), with the lon/lat/lev of each subgrid point. You might be interested in the points owned by each task (halo A in BUMP conventions), so look for variables "lon_sa", "lat_sa" and "levsa".

guillaumevernieres commented 1 year ago

bump subgrid for the scales that we want to use for localization: nicas-subgrid using these parameters for the scales:

scales:
  vert layers: 5 # in units of layer
  socn:
    rossby mult: 1.00
    min grid mult: 2.0

and these nicas parameters:

  model:
    do not cross mask boundaries: false
  nicas:
    resolution: 3
    max horizontal grid size: 300000

NOAA-EMC / godas