Populate QCP and QCF variables

[cgrdn]

Need to indicate tests that were performed/failed/passed. Refer to Argo manual / cookbooks to check what values should be.

If variables do not already exist in VMS files, need to ask Anh where to properly store them.

[cgrdn]

CHLA tests:

• Test 1: Manufacturer DARK_CHLA compared to LAST_DARK_CHLA (same pass, different fail)
• Test 2: Last observation depth is sufficiently below MLD (deep profile pass, shallow profile fail)
• Test 3: Is the newly observed dark count more than 20% different than the manufacturer DARK_CHLA (similar pass, different fail)
• Test 4: Is the new dark count the same as the last dark count (same pass, different fail??? This seems harsh for fail, the dark count is going to fluctuate slightly)
• Test 5: range test (inside range pass, any values outside range, fail)
• Test 6: spike test (no spikes, pass, any spikes, fail)
• Test 7: Test 2 again?? Check FLAG_MLD, which is raised if test 2 fails. If that is true, fail this test and do not perform test 8.
• Test 8: NPQ correction, but manual defines test 8 as positive spikes (pass if no spikes, fail if any positive spikes)

[cgrdn]

BBP is less clear, as are the naming conventions (per https://archimer.ifremer.fr/doc/00491/60262/63668.pdf):

• Test "6": global range test (pass all in range, fail any out of range)
• Test "9": spike test (pass no spikes, fail any spikes)

[cgrdn]

QCP & QCF go in data['PR_STN']['SURF_CODES'], which is a list of ordered dicts. It should be as simple as appending the following to them:

OrderedDict([
    ('PCODE', 'QCP$'),
    ('CPARM':'EE0H'), # or whatever hex code is appropriate/calculated
    ('Q_PARM':'0')
])

I am not sure if each variable is meant to have their own QCP/QCF? They do have dimension N_PARAM, but also I don't see where I would put that in the VMS data.

[cgrdn]

Helpful from Anh:

It’s one QCP and QCF$ variable for all variables in the files. So, for the tests that are common with core profile (TEMP, PSAL, DOXY), the QCP$ will set to 1 if the test was performed. For QCF$, if the variable passed the test, QCF$ for that test is 0. If it failed the QCF$ will set to 1.

I read bbp and chla python file, it looked like you do regional, global range tests plus other tests specific to BBP and CHLA. So in your code, before you set QCP$, you should check to see whether or not it was set to 0 or 1. If it’s 0, then you set it to ‘1’. Similar to QCF$, if it’s 0 and the variable didn’t pass the BBP and CHLA test, you’ll set it to 1. Looking at all the tests you have to do now, you will for sure need to set QCP$ for test #58 to 63 if the variable passed through these tests.

[cgrdn]

Finally, tests from the BGC manual. Unclear if I should populate tests 6 and 9 but I will ask Anh.

Test number	Test name	Type of parameter involved (‘c’, ‘i’, ‘b’)
1	Platform Identification	all
2	Impossible Date Test	all
3	Impossible Location Test	all
4	Position on Land Test	all
5	Impossible Speed Test	all
6	Global Range Test	all
7	Regional Range Test	‘c’ and ‘i’
8	Pressure Increasing Test	‘c’
9	Spike Test	all
10	Top and Bottom Spike Test : removed	n/a
11	Gradient Test	all
12	Digit Rollover Test	‘c’ and ‘i’
13	Stuck Value Test	all
14	Density Inversion	‘c’
15	Grey List	all
16	Gross salinity or temperature sensor drift	‘c’ and ‘i’
17	Visual QC	all
18	Frozen profile	‘c’ and ‘i’
19	Deepest pressure test	all
20	Questionable Argos position test	all
21	Near-surface unpumped CTD salinity test	‘c’
22	Near-surface mixed air/water test	all
23	Real-time Quality Control Flag Scheme for float data deeper than 2000 dbar	all
...	24 to 56 Not used	...
57	DOXY specific test	‘b’
58	CDOM specific test	‘b’
59	NITRATE specific test	‘b’
60	PAR specific test	‘b’
61	IRRADIANCE specific test	‘b’
62	BBP specific tests	‘b’
63	CHLA specific tests	‘b’

[cgrdn]

This is becoming more about populating QCP and QCF (which is also important). Going to change the title and migrate their creation to a new issue.

[cgrdn]

Variables are now created. I am thinking the structure of Flag will be good for this, since if a test has failed already it should not be marked as passed, so an update_safely() method would be appropriate.

[cgrdn]

Done and with 4c81484 the new longer QCP/QCFs can be properly written.