Background of all observations over time

[ctroupin]

User question

Quick question about computing the mean background of all observations over time, in one timestep:

I did not completely understand the documentation I think. Had the idea that background was computed by divand in some standard way, if nothing is specified, but now I think, maybe not. This is our current diva3d call.

Will this in any way use a background of all the observations, to improve the coverage of the interpolated results? If not, I plan to compute a background .nc file and activate that by parameters to diva3d. Did this in the old diva, but it was lost / not implemented in our code when moving to divand.

@time dbinfo = diva3d((lonr,latr,depthr,TS),
    (obslon,obslat,obsdepth,obstime), obsval,
    len, epsilon2,
    filename,varnames[varindex],
    bathname=bathname,
    plotres = plotres,
    mask = nothing,#mask_edit,
    fitcorrlen = false,
    fitvertcorrlen = true,
    niter_e = 2,
    ncvarattrib = ncvarattrib,
    ncglobalattrib = ncglobalattrib,
    surfextend = true,
    minfield = 0.0 # avoid negative values, make sure this is correct for all parameters
    );

[ctroupin]

To create the background, you can run diva3d with for example:

TSbackground = DIVAnd.TimeSelectorYearListMonthList(
    yearlist, [1:12])

which means that observations from all the years (depending on your period of interest, for instance yearlist= [1970:2020]) and all the months are taken to compute the background.

Another thing that is frequently done is to increase the correlation length and the noise-to-signal ratio to ensure that the background you get is smooth.

You have to call diva3D like this:

@time dbinfo = diva3d((lonr,latr,depthr,TSbackground),
    (obslon,obslat,obsdepth,obstime), obsval,
    len, epsilon2,
    filenamebackground,varnames[varindex],
    bathname=bathname,
    plotres = plotres,
    mask = nothing,#mask_edit,
    fitcorrlen = false,
    fitvertcorrlen = true,
    niter_e = 2,
    ncvarattrib = ncvarattrib,
    ncglobalattrib = ncglobalattrib,
    surfextend = true,
    minfield = 0.0 
    );

so background=nothing by default.

Once the background is computed (and stored in the file filenamebackground), then you can perform the analysis that use it:

1. Read the variable from the background file:

   background = DIVAnd.backgroundfile(filenamebackground, varname, TSbackground)

2. Run diva3d: @time dbinfo = diva3d((lonr,latr,depthr,TS), (obslon,obslat,obsdepth,obstime), obsval, len, epsilon2, filename,varnames[varindex], bathname=bathname, background=background, plotres = plotres, mask = nothing,#mask_edit, fitcorrlen = false, fitvertcorrlen = true, niter_e = 2, ncvarattrib = ncvarattrib, ncglobalattrib = ncglobalattrib, surfextend = true, minfield = 0.0 # avoid negative values, make sure this is correct for all parameters );

[jmbeckers]

Can you clarify if one can create a time-independent background field and include that as background in a time-dependent analysis (assuming the spatial grid is exactly the same) ?

is the call time dbinfo = diva3d((lonr,latr,depthr,TS) for the background using the same (lonr,latr,depthr,TS) as the later analysis ? TSbackground seems only used when reading the file ?

[ctroupin]

I think TSbackground also has to be in the first diva3d call in my reply for the calculation of the background, i.e.

@time dbinfo = diva3d((lonr,latr,depthr,TS),...

should be

@time dbinfo = diva3d((lonr,latr,depthr,TSbackground),...

I'll edit my reply accordingly. @Alexander-Barth can you also check?

[qcazenave]

Although default is background=nothing, there is still a background variable in the dbinfo output. is it correct to consider that it is the average of all the data points corresponding to the current depth and time indices ?

[jmbeckers]

If you have data in dbinfo[:background_profile] I think that is little bit more than data average at a given depth, since it also includes a vertical correlation to get a smooth vertical profile.

[ctroupin]

Hello @qcazenave, shall we close this issue, or there is something we should make clearer (here and/or in the doc)? Many thanks for the feeback!

[qcazenave]

Hi @ctroupin,

In my opinion, it is always better to have a complete documentation (although easier said than done). For example, information are missing about the output of the diva3d function when using the command ?diva3d: No mention of :used or :background_profile, for example. I don't really understand why only :residuals and :qc_values are mentioned. Unless you decided not to be exhaustive and provide further information in tutorials ? And if there is a default background profile, I would like to know how it is created.

Aside from that, I think this issue can be closed. Thank you very much for your help and your time.

[ctroupin]

Thanks! yes you are right, we should improve the doc, I'll try to do that soon.

[kawess]

Hi - since this issue is not closed, I put a question here:

I wonder if there is a difference between using a background = nothing and to use a background = background where the background is calculated with a background nc file that is produced with the same year and month selection as the final analyse? Or is it in fact the same?

For example, for my coastal product I do one diva field for the entire time period and per each season. Is it then not needed to do a specific background? Will the default one be the same?

[ctroupin]

Hi Karin, I am not sure I understood properly the background you are using now: is it a seasonal background (using all the years), or a unique background taking all the months and years?

Thanks

[kawess]

Hi, my test was with a seasonal background using all the years.

[ctroupin]

OK, thanks for the information. I'm not sure of the answer so I will dig into the source code to see if I can find it, and let you know quickly.

[ctroupin]

As far as I understand:

if no background is defined (i.e. background=nothing) DIVAnd.DIVAnd_averaged_bg is applied to the observation anomalies; these anomalies are computed by subtracting the spatial mean.

DIVAnd_averaged_bg works as in DIVAndrun except that it includes all dimensions before averaging.

Which means that using a seasonal background (as you did) is not exactly the same as setting background=nothing. https://github.com/gher-uliege/DIVAnd.jl/blob/fbf6867703b589a9395ccc07d46a369e39e51dac/src/diva.jl#L420

[kawess]

Thank you for the reply! For now I do not use a specific background for the coastal product, there were no obvious difference. But for the regional maps I use a background based on same season and 6y-period as the diva-run.

[ctroupin]

Yes I think it really depends on the data coverage, as after all the background will determine the value of the gridded field in regions without data, so it's trying to get the "least bad" values.