Surface (CANARI) Timeseries plot of "Number of Observations" with number of observations that have been used (with anflag 1) only

[marikenh]

or in addition to the total number of observations that is currently showed in the plot.

[paulovcmedeiros]

Hi @marikenh ,

I'm a bit reluctant to make a new "Number of Used Observations" plot using just anflag == 1 as condition. This is because, in obsmon's source code, I see that any anflag > 0 may result in an "Active(2)" label, except for anflag == 4 (Rejected) and anflag == 8 (Blacklisted).

Would you be able to point me to some documentation about how these various anflag values are determined and what they really mean? Or maybe explain it to me here?

Many thanks in advance.

[ewhelan]

From ODB documentation: https://www.ecmwf.int/file/25476 7.2.3 Report/datum status flags:	Member name	Bit range	Description
active	0	Active flag
passive	1	Passive flag
rejected	2	Rejected flag
blacklisted	3	Blacklisted flag
		Not for database CCMA:
		Blacklist failcode bits, fails in ...
monthly	4	... monthly monitoring part of Blacklist
constant	5	... constant part
experimental	6	... experimental part
whitelist	7	Whitelisted (i.e. un-Blacklisted)
	4–31	Unused for database CCMA
	8–31	Unused for database ECMA/ECMASCR

Table 7.2.2 also has some nice CANARI datum_flag bits!

[paulovcmedeiros]

Thanks @ewhelan !

This, however, doesn't really match what @trygveasp originally wrote in obsmon. In fact, it seems to be in direct contradiction: In obsmon, anflag==0 could potentially have the corresponding data entry labelled as Rejected instead of Active, as the table indicates. See, e.g., line 205 of the src/plots/plots_maps.R file.

Since no one has ever complained about this, I'm reasonably confident that Trygve's code is correct. I also assume that the table is correct, so I'm left with the suspicion that I'm reading this wrong. The question then is: What is it I'm missing? Sorry, but I've got to ask since I'm not entirely familiar with these conventions.

[ewhelan]

I am not sure what situation anflag would be set to zero.

In prepObsmon @trygveasp extracts the anflag active/rejected/passive/blacklisted bits (value=[0,1]) rather than the value of anflag in src/odb/ddl/obsmon_conv.sql

[paulovcmedeiros]

But the code does extract datum_anflag.final which, according to Table 7.2.2 in the document you pointed out earlier, has a "bit" range 0-3. Table 7.2.3 then goes to explain this range, as you've already pointed out:

0: Active
1: Passive
2: Rejected
3: Blacklisted 

I'm not convinced that picking anflag == 1 is the way to accomplish what @marikenh wishes (or perhaps the given explanation and the value of the flag are not really compatible). Would it make sense to pick active == 1 instead?

[marikenh]

Thank you for looking at this issue!

The original CANARI documentation from 2002 (available on http://www.umr-cnrm.fr/gmapdoc/spip.php?article3 ) gives a definition of the analysis flag coded over 29 bits: I had a look at those values more than 10 years ago, extracting some information.

But in the later years I have extracted only datum_anflag.final from odb, for monitoring the surface analysis, and found (as far as I remember) only 3 different values which it is meaningful to interpret as 1: used in the surface analysis 4: Rejected in quality control 8: Blacklisted

I think datum_anflag.final is somehow created from anflag, but cannot give you any documentation of that.

[paulovcmedeiros]

Thanks, @marikenh !

The values you present look indeed compatible with what is expected in the current obsmon code. Thanks for the clarification.

I think it is possible then to do something more general, taking into account some other quality control flags (that may apply to other types of observations) instead of just selecting anflag == 1. I'll make a new "Number of Active Observations" timeseries plot which, I believe, will work as you expect for CANARI but could maybe also be useful for other observation types.

Once again, thanks @marikenh and @ewhelan for the discussion.

[paulovcmedeiros]

Done. A new "Number of Active Observations" plot has been added in obsmon v4.1.0.