Issue with ADTs during cycling

[Dooruk]

I keep encountering an issue while cycling with a 2/3-deg setup we have where it ends up crashing after 3 weeks of simulation time. I tried multiple GEOS settings but after taking a closer look at the increments I figured it might be SOCA related. I'm at a bit of a loss here though.

For these particular plots, I'm using a 24-hr DA window, 2/3-deg tripolar grid, ocean-only assimilation (surface observations only), and using GDAS YAMLs.

I'm calling "unhealthy" increments a the ones behave as spikes in the J_o plots.

A healthy increment (2021-06-09):

An unhealthy increment (2021-06-10) (notice the difference in color scale):

I compared hofx_0 and hofx_1 vs. obs value plots to take a closer look

2021-06-09 (left, healthy) 2021-06-10(right, unhealthy)

So hofx_1 in the unhealthy plot shows analysis is having 2-3m difference and this behavior eventually leads to blow up.

One clear solution is perhaps activating an obs - analysis filter in the ADT YAMLs but I would like to ensure this is not a systemic issue first. It is just puzzling to me this doesn't happen during 1/4-deg runs but I notice some spikes there as well. I will investigate them closely next.

Our first instinct was some sort of a land mask issue but that doesn't seem to be the case, you can see the location of the issue.

I also suspect the correlation lengths might be the culprit here. I used StaticB BUMP for these results and I just used the same correlation lengths from the 1/4-deg setup, obviously had to create new BUMP files for the 2/3-deg grid.

However I also looked at the ADT ufo operator and I was wondering if this might have an impact on the analysis?

https://github.com/JCSDA-internal/ufo/blob/d7a8918003ca5da787c4e459f2928a87e6e50882/src/ufo/operators/marine/adt/ObsADT.cc#L77

P.S.: I see same outliers with SST observations as well but maybe the obs. count prevent the same issue from happening (272k for SST vs. ~9k with ADTs)

FYI @guillaumevernieres

[guillaumevernieres]

Thanks for sharing @Dooruk . We recently changed the adt obs yamls and the way we assimilate the rads retrieved adt. I'm not sure this is related to your problem though, but here's what changed: 1 - We're no longer blacklisting adt obs in the southern ocean, instead we compute the unbalanced term of ssh at every cycle. 2 - We only have one obs space for adt. This was done to improve the reference that we estimate in the ufo.

A large ssh increment translates into a large temperature and salinity increment. Did you look at those? I suspect something is wrong in your background error for temperature.

[travissluka]

A couple of related things. 1) Guillaume mentions background error. @Dooruk you're using the BkgErrGodas filter for generating parametric background error, right? It's next on my list to rewrite that filter into a proper saber block (https://github.com/JCSDA-internal/soca/issues/1004) and while I'm at it have it optionally save the calculated error so we can look at it. 2) The line in the ADT operator you note is a problem, but I don't think it's the problem here. An outlier ob would just shift the global bias slightly.

Since the adt hofx_0 looks fine, but hofx_1 is weird, I suspect it is not because of the ADT obs but rather one of the other surface obs, as Guillaume said you should look at what the T/S increments look like at that location

[Dooruk]

@guillaumevernieres

We're no longer blacklisting adt obs in the southern ocean, instead we compute the unbalanced term of ssh at every cycle.

~Sorry, calculating the unbalanced part and incorporating it back is not clear to me. Is this something that can be done via the BalanceSOCA operator? Or by changing h/ssh in the increment file?~ you already responded to this, calculated offline, I remember now!

Combining ADTs is something I will try, at some point..

@travissluka Yes, I use the BkgErrGodas. Sounds good on saving errors.

The line in the ADT operator you note is a problem, but I don't think it's the problem here. An outlier ob would just shift the global bias slightly.

Yeah, that makes sense.. I will look at increments closely and conduct more dirac tests while at it. Thanks for the responses, I will look into this issue again in the near future.