search

dankelley / oceglider

R package for processing ocean glider data
https://dankelley.github.io/oceglider/
3 stars 1 forks source link

should read flags for IOOS files #95

Closed dankelley closed 1 year ago

dankelley commented 1 year ago

I don't know whether the _qc items or the qartod are flags; perhaps both are, in which case the usual oce convention will need to be extended somehow. In any case, I know the internal workings of oce flags, so I am going to hack in something preliminary, which might be useful if @clayton33 wants to do the work properly later (perhaps recognizing qartod as the real flag, or inventing two schemes for qc-related items).

dankelley commented 1 year ago

In "develop" commit c91ff8648c356eb70f4ccd6a7ed47c043443a748 I have a beginning to this. I decided to shove the _qc items into flags, but actually I think the qartod items might actually be the things to record, at least if the test file shown below provides guidance.

An advantage of this addition is that the data slot is now less full of things that ought to go into metadata.

NOTE: I have not slurped in the flag scheme because oce assumes there is just one scheme per file, and I don't know if that is the case for these datasets. Figuring this out should be a high priority because the flag-handling architecture of oce requires knowledge of the scheme.

Code

library(oceanglider)
f<- "sandbox/cl/01downloadFromIOOSERDDAP/ga_538-20151124T1730-delayed.nc"
d<-read.glider.netcdf.ioos(f)
summary(d)

Snippet of output

...
* Data-quality Flags

    conductivity: "NA" 14155
    density:      "NA" 14155
    depth:        "NA" 14155
    latUv:        "NA" 14155
...
dankelley commented 1 year ago

I decided to go ahead and scan the flagScheme from the file, because that's a bit tricky to do (since it has to match oce in construction). I am just about done that but there is something odd that will need to be checked: in the netcdf file it says as follows

library(ncdf4)
f <- "sandbox/cl/01downloadFromIOOSERDDAP/ga_538-20151124T1730-delayed.nc"
o <- nc_open(f)
ncatt_get(o,"salinity_qc")

yields as below. Notice that there are 8 meanings, but supposed the numbers range from 0 to 9. So, um, do they start at 0 or 1? I am assuming the answer is 1 because that's what other CTD-like data seem to use.

...
$flag_meanings
[1] "no_qc_performed good_data probably_good_data bad_data_that_are_potentially_correctable bad_data value_changed interpolated_value missing_value"

$flag_values
 [1] 0 1 2 3 4 5 6 7 8 9
...
dankelley commented 1 year ago

Now, "develop" commit c8822dc9f7d100de6117d8d3019d6b9965261ba8 infers the flag scheme from attributes in the data. It assumes that all variables have the same scheme (and oce needs that). It also assumes that the flag values start at 1, not at 0. I guess we'll know if that's correct when we get data with non-NA flags.

f<- "sandbox/cl/01downloadFromIOOSERDDAP/ga_538-20151124T1730-delayed.nc"
summary(d<-read.glider.netcdf.ioos(f))

produces as in the snippet below

* Data-quality Flag Scheme

    name    "IOOS"
    mapping list(no_qc_performed=1, good_data=2, probably_good_data=3, bad_data_that_are_potentially_correctable=4, bad_data=5, value_changed=6, interpolated_value=7, missing_value=8)
    default c(1, 3, 4, 5, 6, 7, 8)

PS to coauthors -- this is the last I'll be doing on this over the weekend. I think we need more test-file information to code much more.

dankelley commented 1 year ago

This seems to have been done. On the stated test file, I get as shown below (click Details). I also see from

ncdump sandbox/cl/01downloadFromIOOSERDDAP/ga_538-20151124T1730-delayed.nc|less

that the flags are all missing-values.

Although I have no data that can let me test the problem, I have no reason to think there is a problem. Therefore, I'm closing the issue. If we get a file that has flags, and if the package cannot read the data, then perhaps we can get a new test file (and a user who cares), which will make it easier to find a way to fiddle with this.

```R R version 4.3.0 (2023-04-21) -- "Already Tomorrow" Copyright (C) 2023 The R Foundation for Statistical Computing Platform: x86_64-apple-darwin20 (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details. Natural language support but running in an English locale R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications. Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R. > library(oceglider) Loading required package: oce Loading required package: gsw > f <- "sandbox/cl/01downloadFromIOOSERDDAP/ga_538-20151124T1730-delayed.nc" > g <- read.glider.netcdf.ioos(f) FIXME: store "latitude" unit "degrees_north" FIXME: store "longitude" unit "degrees_east" FIXME: store "instrumentCtd" unit "1" FIXME: store "latUv" unit "degrees_north" FIXME: store "lonUv" unit "degrees_east" FIXME: store "platformMeta" unit "1" FIXME: store "preciseLat" unit "degrees_north" FIXME: store "preciseLon" unit "degrees_east" FIXME: store "qartodConductivityFlatLineFlag" unit "1" FIXME: store "qartodConductivityGrossRangeFlag" unit "1" FIXME: store "qartodConductivityPrimaryFlag" unit "1" FIXME: store "qartodConductivityRateOfChangeFlag" unit "1" FIXME: store "qartodConductivitySpikeFlag" unit "1" FIXME: store "qartodDensityFlatLineFlag" unit "1" FIXME: store "qartodDensityGrossRangeFlag" unit "1" FIXME: store "qartodDensityPrimaryFlag" unit "1" FIXME: store "qartodDensityRateOfChangeFlag" unit "1" FIXME: store "qartodDensitySpikeFlag" unit "1" FIXME: store "qartodMonotonicPressureFlag" unit "1" FIXME: store "qartodPressureFlatLineFlag" unit "1" FIXME: store "qartodPressureGrossRangeFlag" unit "1" FIXME: store "qartodPressurePrimaryFlag" unit "1" FIXME: store "qartodPressureRateOfChangeFlag" unit "1" FIXME: store "qartodPressureSpikeFlag" unit "1" FIXME: store "qartodSalinityFlatLineFlag" unit "1" FIXME: store "qartodSalinityGrossRangeFlag" unit "1" FIXME: store "qartodSalinityPrimaryFlag" unit "1" FIXME: store "qartodSalinityRateOfChangeFlag" unit "1" FIXME: store "qartodSalinitySpikeFlag" unit "1" FIXME: store "qartodTemperatureFlatLineFlag" unit "1" FIXME: store "qartodTemperatureGrossRangeFlag" unit "1" FIXME: store "qartodTemperaturePrimaryFlag" unit "1" FIXME: store "qartodTemperatureRateOfChangeFlag" unit "1" FIXME: store "qartodTemperatureSpikeFlag" unit "1" Could not read "row", proceeding to next variable" Could not read "trajectoryStrlen", proceeding to next variable" Could not read "wmoIdStrlen", proceeding to next variable" Could not read "crsStrlen", proceeding to next variable" Could not read "instrumentFlbbStrlen", proceeding to next variable" Could not read "instrumentOxygenStrlen", proceeding to next variable" Could not read "radiationWavelengthStrlen", proceeding to next variable" Could not read "sourceFileStrlen", proceeding to next variable" > summary(g) Glider Summary -------------- * Input file: sandbox/cl/01downloadFromIOOSERDDAP/ga_538-20151124T1730-delayed.nc * Type: ioos * Time ranges from 2016-08-01 04:48:07.473 to 2016-08-01 17:23:21.881 with 14155 samples and mean increment 3.201527 s * Data Overview: Min. Mean Max. Dim. OriginalName backscatter [m^-1] 0.0012879 0.04244 0.045785 14155 backscatter chlorophyll [μg/l] 0.18 1.6288 5.7168 14155 chlorophyll conductivity [S/m] 0 3.2473 3.6075 14155 conductivity crs NA NA NA 14155 crs ctdTimestamp 0 1469849956 1470075690 14155 ctd_timestamp density [kg/m³] 999.84 1029.4 1032 14155 density depth [m] 0 478.24 967.97 14155 depth dissolvedOxygen [μmol/kg] 170.07 214.72 309.97 14155 dissolved_oxygen instrumentCtd NA NA NA 14155 instrument_ctd instrumentFlbb NA NA NA 14155 instrument_flbb instrumentOxygen NA NA NA 14155 instrument_oxygen latitude -43.366 -43.325 -43.278 14155 latitude latUv -43.331 -43.293 -43.251 14155 lat_uv longitude -43.827 -43.823 -43.819 14155 longitude lonUv -43.823 -43.818 -43.812 14155 lon_uv oxygenSaturation [%] 50.874 68.142 84.575 14155 oxygen_saturation pitch [°] -42.3 -34.778 17.5 14155 pitch platformMeta NA NA NA 14155 platform_meta preciseLat -43.374 -43.325 -43.272 14155 precise_lat preciseLon -43.83 -43.823 -43.818 14155 precise_lon preciseTime 1.47e+09 1.47e+09 1470075691 14155 precise_time pressure [dbar] 0 482.84 978.03 14155 pressure profileId 1.47e+09 1.47e+09 1470068713 14155 profile_id qartodConductivityFlatLineFlag 1 1.0016 3 14155 qartod_conductivity_flat_line_flag qartodConductivityGrossRangeFlag 1 1 1 14155 qartod_conductivity_gross_range_flag qartodConductivityPrimaryFlag 1 1.0022 3 14155 qartod_conductivity_primary_flag qartodConductivityRateOfChangeFlag 1 1.0005 3 14155 qartod_conductivity_rate_of_change_flag qartodConductivitySpikeFlag 1 1.0005 3 14155 qartod_conductivity_spike_flag qartodDensityFlatLineFlag 1 1 1 14155 qartod_density_flat_line_flag qartodDensityGrossRangeFlag 1 1 1 14155 qartod_density_gross_range_flag qartodDensityPrimaryFlag 1 1.0005 3 14155 qartod_density_primary_flag qartodDensityRateOfChangeFlag 1 1.0003 2 14155 qartod_density_rate_of_change_flag qartodDensitySpikeFlag 1 1.0005 3 14155 qartod_density_spike_flag qartodMonotonicPressureFlag 1 1.0046 3 14155 qartod_monotonic_pressure_flag qartodPressureFlatLineFlag 1 1 1 14155 qartod_pressure_flat_line_flag qartodPressureGrossRangeFlag 1 1 1 14155 qartod_pressure_gross_range_flag qartodPressurePrimaryFlag 1 1.0046 3 14155 qartod_pressure_primary_flag qartodPressureRateOfChangeFlag 1 1.0003 2 14155 qartod_pressure_rate_of_change_flag qartodPressureSpikeFlag 1 1 1 14155 qartod_pressure_spike_flag qartodSalinityFlatLineFlag 1 1 1 14155 qartod_salinity_flat_line_flag qartodSalinityGrossRangeFlag 1 1 1 14155 qartod_salinity_gross_range_flag qartodSalinityPrimaryFlag 1 1.0005 3 14155 qartod_salinity_primary_flag qartodSalinityRateOfChangeFlag 1 1.0005 3 14155 qartod_salinity_rate_of_change_flag qartodSalinitySpikeFlag 1 1.0005 3 14155 qartod_salinity_spike_flag qartodTemperatureFlatLineFlag 1 1.0019 3 14155 qartod_temperature_flat_line_flag qartodTemperatureGrossRangeFlag 1 1 1 14155 qartod_temperature_gross_range_flag qartodTemperaturePrimaryFlag 1 1.0024 3 14155 qartod_temperature_primary_flag qartodTemperatureRateOfChangeFlag 1 1.0003 2 14155 qartod_temperature_rate_of_change_flag qartodTemperatureSpikeFlag 1 1.0005 3 14155 qartod_temperature_spike_flag radiationWavelength [nm] NA NA NA 14155 radiation_wavelength roll [°] 5.6 6.5245 8.6 14155 roll salinity [PSS-78] 0 34.273 34.527 14155 salinity sourceFile NA NA NA 14155 source_file temperature [°C, ITS-90] 0 4.3348 8.3214 14155 temperature timeUv 1.47e+09 1470065641 1470088529 14155 time_uv trajectory NA NA NA 14155 trajectory u [m/s] -0.02226 0.014154 0.055192 14155 u v [m/s] 0.18461 0.20846 0.22962 14155 v wmoId NA NA NA 14155 wmo_id * Data-quality Flag Scheme name "IOOS" mapping list(no_qc_performed=1, good_data=2, probably_good_data=3, bad_data_that_are_potentially_correctable=4, bad_data=5, value_changed=6, interpolated_value=7, missing_value=8) default c(1, 3, 4, 5, 6, 7, 8) * Data-quality Flags conductivity: "NA" 14155 density: "NA" 14155 depth: "NA" 14155 latUv: "NA" 14155 latitude: "NA" 14155 lonUv: "NA" 14155 longitude: "NA" 14155 preciseLat: "NA" 14155 preciseLon: "NA" 14155 preciseTime: "NA" 14155 pressure: "NA" 14155 salinity: "NA" 14155 temperature: "NA" 14155 time: "NA" 14155 timeUv: "NA" 14155 u: "NA" 14155 v: "NA" 14155 * Processing Log - 2023-08-10 16:41:10.559 UTC: `create 'glider' object` > > ```