add "first guess" and "difference between First guess and Analysis"

[sebvi]

Branch

https://github.com/wmo-im/GRIB2/tree/issue-56

Summary and purpose

This issue propose to add 2 new entries in Code table 4.3 - "Type of generating process"

Action proposed

The team is requested to approve the content of this proposal for inclusion with the next update of the WMO Manual on Codes

Discussions

ECMWF would like to request 2 new entries in Code table 4.3: "First guess" and "Difference between first guess and analysis". The "first guess" is the term commonly used in NWP to name the background data field used in the assimilation step to be blended with the observations to produce the analysis field. It is usually taken from a previous forecast run valid at the appropriate date and time. The difference between the first guess and the analysis is used to evaluate how good the assimilation system is performing and to assess the quality of the background field.

At ECMWF, we store the first guess, the analysis and the difference between the two in our long term archive and we disseminate them to our member state.

Detailed proposal

ADD in Code Table 4.3 - Type of generating process

Octet Meaning 19 First guess 20 Difference between first guess and analysis 21-191 Reserved

[SebastianB2]

Dear Sebastien,

Thank you for raising this issue, this is of interest for us at the DWD, too.

We code analysis increments (i.e. the difference: Analysis - First Guess) with a local type of generating process. Therefore, we would prefer the difference AN - FG instead of FG - AN, but of course this is a matter of convention and taste, and could be handled by changing the sing of the values accordingly.

Another comment: Maybe a third additional entry, the difference: Initialized Analysis - Analysis, could be useful, too.

To summarize, we would suggest the following modification of your issue for discussion:

Octet Meaning 19 First guess 20 Difference between analysis and first guess 21 Difference between initialized analysis and analysis 22-191 Reserved

(In order to avoid any misunderstanding with regard to the sign convention: Would it be of help, if a note would be added to the entry 20 (or the entries 20 and 21)? Maybe something like: 'The wording: "Difference between A and B" means "A minus B" here.' ?).

[amilan17]

from Nov minutes discuss with DWD about the sign of difference (adding two entries is also an option) and the meaning of "initialized analysis," branch will be created when ready

[sebvi]

I have checked with my colleagues at ECMWF and we are doing the same processing than DWD, i.e. we use difference between Analysis and First Guess and we also call it "Analysis Increments" which makes me wonder if we should simply call entry 20 "Analysis increments"!

I have a question for @SebastianB2 regarding "initialized analysis": is this the same thing as "initialization" (entry 1 in the same table). If it is then we should be consistent with the terms and use "initialization" instead of "initialized analysis". Furthermore, since the difference is similar to entry 20 we could maybe call it "initialization increments"?

[SebastianB2]

Thanks for considering our suggestion!

We consulted colleagues on our part and would formulate the relation between "initialization" and "initialized analysis" in the following way (though other definitions are well possible):

"Initialization" means a processing of data to minimize meteorological noise (e.g., by some kind of filtering). It can be applied to analyses (A) yielding an "initialized analysis" (IA), or forecasts (first guess, FG), yielding an "initialized first guess" (IFG). The term "initialization increment" seems to really exist. It stands for IA - A, but maybe also for IFG - FG
(but we are not entirely sure about that, the latter might also be called "filter increment").

Maybe the coding perspective provides another approach to this issue (in ecCodes notation):	Case	typeOfProcessedData (T1.4)	typeOfGeneratingProcess (T4.3)	forecast time/step
A	0 (analysis)	0 (analysis)	0
FG	1 (forecast)	2 (forecast)	step > 0
IA	1	1 (initialization)	0
IFG	1	1	step > 0

So IA and IFG would differ only in the forecast time.

We like your idea of calling the entries "analysis increment" and "initialization increment". We think, however, that some info on what the increment is composed of could be helpful. So what's about the following compromise:

Octet Meaning 19 First guess 20 Analysis increment (analysis minus first guess) 21 Initialization increment (initialized minus uninitialized analysis or forecast, see Note 2) 22-191 Reserved

Note 2: Here "initialization" means a method to eliminate noise from the forecast or analysis (e.g., by some kind of filtering).

would that work for you?

[sebvi]

Thanks for your reply!

Would it make sense to separate the initialization increments for AN and FC in 2 entries. I see in the same table that Analysis error and forecast error are 2 separate entries and not "error in analysis or forecast", same with post-processed analysis and forecast, and so on.

I always found redundant the use of Code Table 1.4 and 4.3 and still today I don't really understand the purpose to have both. If someone knows the purpose/difference of the 2 tables, I am happy to learn :)

[SebastianB2]

Your are right with entry 21. We would use entry 21 only for the case IA - A, but to keep it general, we decided for the formulation "analysis or forecast". We would agree to split entry 21 into:

21 Initialization increment for analysis (initialized minus uninitialized analysis, see Note 2) 22 Initialization increment for forecast (initialized minus uninitialized forecast, see Note 2)

(where "forecast" should include "first guess" as a special case.) However, we are no experts on this subject. Maybe it is the best option at this point, to keep the revised entry 21 and to drop entry 22, since neither we nor probably you would use entry 22 and we are doubtful about its reasonableness.

Regarding your second question: We felt the same. It was confusing for us, too. But now, we use it in a way that

• Entry in Tab. 1.4: Kind of overall description ("headline")
• Entry in Tab. 4.3: Detailed description (specialization)
• And for "Multi GRIBS": As section 1 never will be repeated, entry from Tab. 1.4 stands for ALL gribs in the multi grib message, and repeated entry 4.3 only for one single grib in the multi grib message.

That is our understanding of the entries coming from Tab. 1.4 and 4.3. So they are not really redundant

[sebvi]

Can we please create a branch for this issue? thanks

[sebvi]

I have update the branch with 3 new entries:

Octet Meaning 19 First guess 20 Analysis increment (see Note 2) 21 Initialized analysis increment (see Note 3) 22-191 Reserved

the Notes should be modified, the current NOte should be renamed "Note 1", then add Note 2 and 3: 2) Analysis increment represents the difference Analysis - First Guess 3) Initialized analysis increment represents the difference Initialized Analysis - Analysis

Is it fine with you @SebastianB2 ?

[SebastianB2]

Thank you for wrapping up and updating the branch.

Overall, we like your suggestion. We are bit uncertain about entry 21 for IA - A. The wording may be read as "(Initialized analysis) increment", but it might be misinterpreted as "Initialized (analysis increment)", i.e. I(A - FG) = IA - IFG, as well. We would feel a bit more comfortable with the formulation:

21 Initialization increment for analysis (see Note 3)

However, if you prefer "Initialized analysis increment" this is fine with us, for your Note 3 gives the intended definition of the entry.

[sebvi]

@SebastianB2 I am happy to change with your suggestion. It is always better to be as specific as possible and avoid any sort of possible confusion. My experience is that notes are rarely read so it is better to capture the best possible wording in the entry itself.

I will update the git branch accordingly.

[amilan17]

@sebvi 

Can you verify that these editorial amendments don't change the meaning in the notes? Or is it supposed to mean analysis minus first guess? and initialized analysis minus analysis? 

20	Analysis increment	Analysis increment represents the difference Analysis - First Guess Analysis increment represents the difference between analysis and first guess
21	Initialization increment for analysis	Initialized analysis increment represents the difference Initialized Analysis - Analysis Initialized analysis increment represents the difference between initialized analysis and analysis

[sebvi]

@amilan17 I guess it is fine, the reason it was written that way is because the way the difference is produced matters: (AN-FG) = -(FG-AN)

The way it is done has been discussed in the comments above.

[amilan17]

ah, I'm reading the comments more closely now. how about:

Analysis increment represents analysis minus first guess

Initialized analysis increment represents initialized analysis minus analysis

[sebvi]

@amilan17 yes I think this makes it less ambiguous. :)