Include uncertainty in NetCDF files

[ggalibert]

Some comments about uncertainty information:

What about adding a uncertainty of 0.4 C (from the manufacture’s sheet) would also indicate the data quality.

-- @petejan

As a user of ADCP temperature variables: Shouldn't the manufacture uncertainty be added as an attribute (or as variable if applicable) to all ANMN (and AODN) data? Final users would be happy to avoid digging that info in a per sensor/instrument basis (not only for ADCP). Also, they can take this into account in their analysis if wanted. Although the end variable uncertainty for a lot of instruments is less than the actual instrument one (say variable are burst average in the instrument itself), some info regarding the uncertainty is VERY useful. Maybe something like "nominal_uncertainty" attribute, or even like precision and/or accuracy would be useful.

-- @ocehugo

For me the uncertainty should always be added to variables in a netCDF file, that however is not that easy to achieve.

http://imos.org.au/fileadmin/user_upload/shared/IMOS%20General/documents/IMOS/IMOS_Data_streams_and_Their_Uncertainties_3.1.pdf

-- @petejan

[ggalibert]

From the .pdf document above:

Physical parameters

Instrument	Real time clock	Temperature	Conductivity	Pressure	Oxygen	Fluorescence	Turbidity
WQM	± 5 seconds per month	± 0.002 °C + 0.002 °C per year (95% c.i.)	± 0.0003 S/m + 0.004 S/m per year (95% c.i.)	± 0.1% FS + 0.1% FS per year	± 2% + 2% per 1000 hours operation	±1.5% + 1% per year	±1.5% + 1% per year
SBE37	± 5 seconds per month	± 0.002 °C + 0.002 °C per year (95% c.i.)	± 0.0003 S/m + 0.004 S/m per year (95% c.i.	± 0.1% FS + 0.1% FS per year
SBE39	± 5 seconds per month	± 0.002 °C + 0.002 °C per year (95% c.i.)		± 0.1% FS + 0.1% FS per year
RBR DR1050	?			± 0.05% FS, yearly drift uncertain
Aqualogger 520	unspecified	± 0.05 °C annual drift rates unknown		± 0.2% FS annual drift rates unknown
FSI NXIC CTD BIO	± 50 seconds per month	± 0.002 °C + 0.006 °C per year (95% c.i.)	± 0.0005 S/m + 0.0006 S/m per year	?
FSI NXIC CTD AUTO 7000M	± 50 seconds per month	± 0.002 °C + 0.006 °C per year (95% c.i.)	± 0.0002 S/m + 0.0006 S/m per year	?
FLNTU (with FSI NXIC CTD)						±1.5% + 1% per year	±1.5% + 1% per year
SBE19+	?	± 0.002 °C + 0.002 °C per year (95% c.i.)	± 0.0003 S/m + 0.0003 S/m per year (95% c.i.)	± 0.1% FS
SBE43 (with SBE19+)					± 2% + 2% per 1000 hours operation

NOTES:

1. The figures provided for FSI sensors are based directly on the manufacturer’s specification sheet. Experience in the CMAR Calibration Facility supports the stated drift rates for the temperature channels. However observation of the calibration stability for a cohort of more than two dozen FSI instruments has indicated that the conductivity sensor drift is often around 0.02 S/m per year. Admittedly the instruments observed all utilise the older style “external field sensor” rather than the NXIC sensor, so the stability issues may not apply to these NXIC sensors. It will be valuable to follow a careful recalibration protocol with these new sensors to discover the stability of the calibration when used in the field. It is recommended that at least a subset of the instruments are calibrated on a yearly cycle so that knowledge of the sensor stability can be verified and quantified.
2. It will be possible to correct the conductivity sensor on the SBE19+ by using the salinity bottle samples that are collected by the coincident biogeochemical sampling program for the National Reference Station work. For this reason a reduced uncertainty figure is suggested for these instruments.

Current measurement

Instrument	Real time clock	Pressure	Velocity	Tilt	Compass
RDI 300kHz Workhorse	?	?	± 0.5%	± 0.5°	± 2°
RDI 75kHz Long Ranger	?	± 5 m	± 1%	± 0.5°	± 2°
RDI 150kHz Quartermaster	?	± 5 m	± 1%	± 0.5°	± 2°
Nortek 190kHz Continental	± 5 seconds per month	± 0.5%	± 1%	± 0.2°	± 2°

There are some instruments currently used that are not covered but this could be the kind of information that could be included in the NetCDF files.

Can we assume this information can be hard coded in the parsers or can it vary for the same instrument from deployment to deployment and in that case needs to be stored somewhere else (for example DDB)?

[sspagnol]

If it helps this is our collection of notes on ADCP TEMP sensors.

RDI

RDI Workhorse/LongRanger/Quartermaster TEMP sensor reported nominal specifications are uncertainty ±0.4° C and resolution 0.01°. Not yet known response time.

From Long Ranger & QuarterMaster Operation Manual

This sensor is embedded in the transducer head and is not field replaceable.

RDI Workhorse User Guide has a statement about effects of salinity on velocity, namely

The WorkHorse records velocity data in units of mm/s. Calibration depends on how well the WorkHorse knew the speed of sound (which it computed based on its measured temperature and the salinity value it was given). A salinity error of 5 ppt introduces less than 0.5% velocity error.

And Long Ranger & QuarterMaster Operation Manual states

TRDI recommends return every two to three years for Factory calibration

And and interesting fact to remember (last sentence)

If the PA test fails the sensor test, run PC2 to isolate the problem. The ambient temperature sensor is mounted on the receiver board. This sensor is imbedded in the transducer head, and is used for water temperature reading. The attitude temperature sensor is located on the High Power Amplifier Board under the compass. The ADCP will use the attitude temperature if the ambient temperature sensor fails.

Nortek

Information collated from various Nortek instrument manuals.

AWAC

Specs for AWAC TEMP sensor

Temperature (thermistor embedded in head)

Range: –4 °C to +30 °C

Accuracy/Resolution: 0.1 °C/0.01 °C

Time response: Approximately 10 min.

Construction note

The temperature sensor, standard on all AWACs, is mounted inside the sensor head and has a conductive (titanium) contact to the water.

Sensor check note

Temperature should be close to the ambient temperature, assuming that the AWAC has been exposed to a constant temperature for at least an hour. The temperature sensor is located inside the sensor head

Continental

Specs for Continental TEMP sensor

Temperature (thermistor embedded in head)

Range: -5°C to 40°C

Accuracy/Resolution: 0.1°C/0.01°C

No data for response time is given.

Construction note

The temperature sensor, standard on all Continentals, is mounted internally in the sensor head.

Sensor check note

Temperature should be close to your room temperature, assuming the Continental has been in the room for a while.

Aquadopp Profiler

Specs for Aquadopp Profiler TEMP sensor

Temperature Type Thermistor embedded

Range -4°C to 30°C

Accuracy/resolution 0.1°C/0.01°C

Time response 10-15 min (depending on which manual or product sheet you read)

Construction note

The temperature sensor, standard on all Aquadopps, is mounted internally in the sensor head.

Sensor check note

Temperature should be close to your room temperature, assuming the Aquadopp has been in the room for a while.

Given the lighter housing "a while" could maybe mean half an hour?

EasyQ

Specs for EasyQ TEMP sensor

Temperature (thermistor embedded in head)

Range: -4°C to 30°C

Accuracy: 0.1°C

No data is given for response time.

Construction note

The temperature sensor, standard on all EasyQ’s, is mounted internally in the sensor head.

Sensor check note

Temperature should be close to your room temperature, assuming the EasyQ has been in the room for a while.

[mhidas]

FYI There is a group putting together a proposal to extend the CF conventions to better represent uncertainties. It proposes to always store uncertainty values in variables (not attributes), and allows for the distinction between random, systematic, and total uncertainty. For more details see the draft document here (they may still be open to comments & suggestions).

[ocehugo]

Yes, I agree that variables are the way to go -> It shouldn't be hidden in a variable and/or global attribute.

By being a variable the uncertainty is explicit -> the user will not miss it, otherwise the value is rather hidden in an attribute (and implicit)...

[petejan]

Hmm, read through the document and comments, and only justification I could see from not using the existing structure defined by CF (aux variable, standard_error name modifier) is that its hard for users to understand. I don't see that document on the CF conventions Trac as a proposal either, so might be some way off.

[ggalibert]

Looks like the current CF convention can handle total uncertainty (systematic + random) with the standard_error standard name modifier: http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html#standard-name-modifiers . This is probably enough for what we would like to achieve to start with and would store uncertainty in an ancillary variable.