Closed dankelley closed 1 year ago
In "develop" commit c91ff8648c356eb70f4ccd6a7ed47c043443a748 I have a beginning to this. I decided to shove the _qc
items into flags, but actually I think the qartod
items might actually be the things to record, at least if the test file shown below provides guidance.
An advantage of this addition is that the data
slot is now less full of things that ought to go into metadata
.
NOTE: I have not slurped in the flag scheme because oce assumes there is just one scheme per file, and I don't know if that is the case for these datasets. Figuring this out should be a high priority because the flag-handling architecture of oce
requires knowledge of the scheme.
Code
library(oceanglider)
f<- "sandbox/cl/01downloadFromIOOSERDDAP/ga_538-20151124T1730-delayed.nc"
d<-read.glider.netcdf.ioos(f)
summary(d)
Snippet of output
...
* Data-quality Flags
conductivity: "NA" 14155
density: "NA" 14155
depth: "NA" 14155
latUv: "NA" 14155
...
I decided to go ahead and scan the flagScheme
from the file, because that's a bit tricky to do (since it has to match oce
in construction). I am just about done that but there is something odd that will need to be checked: in the netcdf file it says as follows
library(ncdf4)
f <- "sandbox/cl/01downloadFromIOOSERDDAP/ga_538-20151124T1730-delayed.nc"
o <- nc_open(f)
ncatt_get(o,"salinity_qc")
yields as below. Notice that there are 8 meanings, but supposed the numbers range from 0 to 9. So, um, do they start at 0 or 1? I am assuming the answer is 1 because that's what other CTD-like data seem to use.
...
$flag_meanings
[1] "no_qc_performed good_data probably_good_data bad_data_that_are_potentially_correctable bad_data value_changed interpolated_value missing_value"
$flag_values
[1] 0 1 2 3 4 5 6 7 8 9
...
Now, "develop" commit c8822dc9f7d100de6117d8d3019d6b9965261ba8 infers the flag scheme from attributes in the data. It assumes that all variables have the same scheme (and oce
needs that). It also assumes that the flag values start at 1, not at 0. I guess we'll know if that's correct when we get data with non-NA flags.
f<- "sandbox/cl/01downloadFromIOOSERDDAP/ga_538-20151124T1730-delayed.nc"
summary(d<-read.glider.netcdf.ioos(f))
produces as in the snippet below
* Data-quality Flag Scheme
name "IOOS"
mapping list(no_qc_performed=1, good_data=2, probably_good_data=3, bad_data_that_are_potentially_correctable=4, bad_data=5, value_changed=6, interpolated_value=7, missing_value=8)
default c(1, 3, 4, 5, 6, 7, 8)
PS to coauthors -- this is the last I'll be doing on this over the weekend. I think we need more test-file information to code much more.
This seems to have been done. On the stated test file, I get as shown below (click Details). I also see from
ncdump sandbox/cl/01downloadFromIOOSERDDAP/ga_538-20151124T1730-delayed.nc|less
that the flags are all missing-values.
Although I have no data that can let me test the problem, I have no reason to think there is a problem. Therefore, I'm closing the issue. If we get a file that has flags, and if the package cannot read the data, then perhaps we can get a new test file (and a user who cares), which will make it easier to find a way to fiddle with this.
I don't know whether the
_qc
items or theqartod
are flags; perhaps both are, in which case the usualoce
convention will need to be extended somehow. In any case, I know the internal workings ofoce
flags, so I am going to hack in something preliminary, which might be useful if @clayton33 wants to do the work properly later (perhaps recognizingqartod
as the real flag, or inventing two schemes for qc-related items).