search

iobis / env-data

ENV-DATA related issues and documentation
2 stars 0 forks source link

QC flags for abiotic data #1

Open Daphnisd opened 7 years ago

Daphnisd commented 7 years ago

Abiotic measurements often have a QC flags associated to them, e.g good value, bad value,... . This information is relevant and we should look into finding a way to store this in IPT.

The eMoF collumn measurementAccuracy is not suitable for this information.

One option is to extend the eMoF with a field measurementQuality and/or a measurementQualityID field. The IODE Primary Level quality flags could be used for this http://vocab.nerc.ac.uk/collection/L34/current/. Adding both measurementQuality and measurementQualityID fields does seem like overkill. Easy mapping of IODE flags provide with other schema's is available https://www.nodc.noaa.gov/oceanacidification/support/MG54_3.pdf

While meaningful, solely including measurementQuality does not provide any information regarding the tests which were carried out on the reading to derive the Quality flag. Which tests passed, which tests failed? A secondary flag system seems needed for this.

Any idea's or comments are welcome.

meliezer commented 6 years ago

QC flags are very important. I would add only an optional measurementQualityID, which will be something like SDN::L34:3 or SDN::L20:3, as long as it's a vocabulary that can be mapped to IODE flags. In measurementRemarks the user is free to state the flag meaning, explain why, use another flagging system or even link to a web service which reads a code and provides a user-friendly test results html report. The QC procedures should be mentioned in the EML file under qualityControl.

Daphnisd commented 6 years ago

A QF may be only needed to indicate whether a measurement is bad or suspect. It's unlikely that we would have other information. The abiotic data would come from biologists. If it's physicists or chemists who have the data, likely it would go to other repositories.

"Good" might only be needed in case a value was suspected but checked and found to be correct. So: I would opt for http://vocab.nerc.ac.uk/collection/L34/current/, and only add the URI, not a text qualifier. MeasurementRemarks can contain information on why we think the measurement is bad or suspect.

L20 seems to extensive for what we need.

meliezer commented 6 years ago

@Daphnisd Do you mean for example http://vocab.nerc.ac.uk/collection/L34/current/3/ as a new term measurementQualityID?

meliezer commented 6 years ago

The abiotic data would come from biologists. If it's physicists or chemists who have the data, likely it > would go to other repositories.

It can certainly come from a national data centre which has both types of data and knows how to evaluate the quality of both types.

"Good" might only be needed in case a value was suspected but checked and found to be correct.

How would a user know that no flag means a good value?

Daphnisd commented 6 years ago

@Daphnisd Do you mean for example http://vocab.nerc.ac.uk/collection/L34/current/3/ as a new term measurementQualityID?

yes

How would a user know that no flag means a good value?

It doesn't mean good value. It means the the quality is has not been accessed. If the quality is assessed, then we would need a way to say what was assessed. Why do you say it's good? I think it's more important for OBIS to flag data as questionable or bad. I fear that promoting a "good" flag would invite malpractice and unclarity on what the statement good would be based on.

meliezer commented 6 years ago

It doesn't mean good value. It means the the quality is has not been accessed. Leaving an empty value creates confusion. There is a flag for this case: http://vocab.nerc.ac.uk/collection/L34/current/2/

I understand your concerns, as I've written: "In measurementRemarks the user is free to state the flag meaning, explain why, use another flagging system or even link to a web service which reads a code and provides a user-friendly test results html report. The QC procedures should be mentioned in the EML file under qualityControl."