flag function ideas

[richardsc]

From today's f2f

• [x] initializeFlagScheme()
• [x] initializeFlags()
• [x] setFlags()
• [x] handleFlags()

[dankelley]

I think this is working in the "dk" branch commit fa160dea90420eaeea096d8f5e7f9de809dca8cb. I will likely have a more careful look at the vignette this afternoon ... all I've done there, and in the various function documentation entries, is to rename setFlagScheme to the new name initializeFlagScheme, and to add the initializeFlags calls as needed. Oh, and of course I did the coding, and added tests.

[dankelley]

I am going to rework setFlagScheme() today, because I forgot to create an argument that names the scheme (as we had agreed in the f2f).

From the user's point of view, this will just mean the deletion of one arg (scheme) and the addition of two (name and mapping). That seems like a small thing, but there are lots of interlocking pieces and so some fiddling will be required to get all the code working again. Every fiddle requires a build, which is slow, but today is a good day for that because I have some reading work to do.

Given all of this, I will move my planned merge of "dk" into "develop" from tomorrow (Saturday) to the upcoming Wednesday.

[richardsc]

Sounds good.

I assume that you mean initializeFlagScheme() though?

[dankelley]

Yes, initializeFlagScheme() was what I meant. I'm building in lots of tests. In the docs I refer to argo codes at http://www.oceannetworks.ca/data-tools/data-quality but I would like to find something more official and general. I'd prefer a website that doesn't require someone to download a big .pdf and search for a certain page or section. Do you have any suggestions?

Also, another thing, if you have a second: I am a bit divided on naming the codes. Quite a few use value 2 to indicate things that were tested but not found faulty, and that might incline me to try to use a similar word for each, but I decided that I'd prefer to honour the codes as stated for the particular setup. I am not taking direct quotes from tables of meanings, though, because e.g. argo says "Data probably bad. Failed minor tests." for value 2, and that's an excessively long name!

Below are the names I'm using (from the not-yet-commited test suite). Can I get your OK on the names? (They matter, because I have to synch them between code, doc, and test.)

test_that("predefined flag schemes", {
          data(ctd)
          ## Check error message for unknown scheme
          a <- expect_error(initializeFlagScheme(ctd, "unknown"), "name \"unknown\" is not pre-defined")
          ## Check predefined schemes
          a <- initializeFlagScheme(ctd, "WHP CTD")
          expect_equal(a[["flagScheme"]],
                       list(name="WHP CTD",
                            mapping=list(not_calibrated=1, acceptable=2, questionable=3,
                                         bad=4, not_reported=5, interpolated=6,
                                         despiked=7, missing=9)))
          a <- initializeFlagScheme(ctd, "WHP bottle")
          expect_equal(a[["flagScheme"]],
                       list(name="WHP bottle",
                            mapping=list(no_information=1, no_problems_noted=2, leaking=3,
                                         did_not_trip=4, not_reported=5, discrepency=6,
                                         unknown_problem=7, did_not_trip=8, no_sample=9)))
          a <- initializeFlagScheme(ctd, "argo")
          expect_equal(a[["flagScheme"]],
                       list(name="argo",
                            mapping=list(not_assessed=0, passed_all_tests=1, probably_good=2,
                                         probably_bad=3, bad=4, averaged=7,
                                         interpolated=8, missing=9)))
})

[dankelley]

Actually, for consistency with the lower-case nature of classes in oce, I think I ought to rename "WHP CTD" to "WHP ctd". (I'm already renaming from "WHP CTD exchange", so I figure I have some latitude.)

[EOGrady21]

@dankelley If you are still looking for flag schemes, this is the BODC standard which is part of SeaDataNet, this list can also be easily exported to .csv if that is helpful at all. I can't confirm for sure but I think this will likely be part of the standard used in some of the new NetCDFs I'm working on at BIO. I think 1-9 would likely be all the codes needed, avoided the A and Q character flags.

url

[dankelley]

Done in "dk" branch commit b950cc7abb6b5746136763993f8df81aec9797df

Thanks, @Echisholm21. Those 'A' and 'Q' codes are quite weird. I don't really understand what the 'A' one means, and the 'Q' one reflects what I consider to a bad idea. Hm, I dropped my glasses, so I can't see whether there's a person standing at X metres away, so I'll say there is someone there, at my detection limit. Nutty.

Thanks!!

PS. you did the URL in the reverse way. The first part, in square brackets, is the text to be shown, and the URL is in round brackets thereafter. I altered your comment accordingly. Usually, though, I just put the URL directly, because it's less work, and it lets the reader see the URL, in case the link fails and the reader wants to try similar URLs to find the new location.

[dankelley]

MEDS has yacs (yet another coding scheme) which might be relevant to @Echisholm21 's comment above. The URL for this MEDS scheme is http://www.dfo-mpo.gc.ca/science/data-donnees/code/list/014-eng.html and for convenience I've copied the table below. Frankly, I find this scheme a bit odd, e.g. the definition of "acceptable" doesn't really sound too acceptable to me (I wonder, is "inconsistent" a typo, when they mean "consistent???"), especially because other agencies use "acceptable" as the translation of a numerical code of 2, which often means "good".

0 Blank - No quality control (QC) has been performed

1 Good - QC has been performed: record appears correct

3 Doubtful - QC has been performed: record appears doubtful

4 Erroneous - QC has been performed: record appears erroneous

5 Changes - The record has been changed as a result of QC

6 Acceptable - QC has been performed: record seems inconsistent with other records

7 Off Position - There is a problem with the buoy position or mooring. Data may still be useful.

8 Reserved

9 Reserved - indicates missing elements

[dankelley]

Closing, since this has been addressed in the "develop" branch, commit 3ccd35cc70479d69ec78abb0570342a7715346d7, which was merged from the "dk" branch earlier today.