search

aodn / content

Tracks AODN Portal content and configuration issues
0 stars 0 forks source link

as a User I would like to know the endpoint url of all quality control documents from the netcdf global attributes #396

Closed ggalibert closed 5 years ago

ggalibert commented 5 years ago

From @ocehugo :

I believe the user should be directed to the actual file link of the quality control procedure for every IMOS file.

Currently, the direction to the QC document is soft:

For example, file http://thredds.aodn.org.au/thredds/dodsC/IMOS/ACORN/gridded_1h-avg-current-map_QC/BONC/2014/02/05/IMOS_ACORN_V_20140205T010000Z_BONC_FV01_1-hour-avg.nc said the following for the "file_version_quality_control" global attribute:

'Data in this file has been through the quality control procedure as described in the document: Offline QC procedures for ACORN radars.  Every data point in this file has an associa
ted quality flag.'

I believe this should be hardwired -> point to the actual document link pdf (best) or the actual Facility documentation page which describe the quality control for that particular dataset. say for example: 

http://imos.org.au/files/acorn_qc_procedure-seasonde-v1.5.pdf

or bare minimum:

http://imos.org.au/facilities/oceanradar/oceanradar-documentation/

In fact, I could not easily find the QC document for SeaSonde. Is this the Proposed QC for Seasonde data back in 2014 in the oceanradar-documentation page!?

I found frustrating have to google around and try to find this document. This also creates a problem with people possibly getting outdate information or even wrong (say an outdate pdf). So I went to do what everyone does:

  1. I googled "Offline QC procedures for ACORN radars".
  2. was happy to get the first result to an IMOS web page (good).
  3. The first link is a WERA QC page. There is no pdf link in that page (even for WERA).
  4. I went backwards in the tree to the "data" section. Some papers refs, but no pdf document. A link to WERA QC was present, but is a dead link (I'm looking for Seasonde...)!
  5. I went backwards one more step in the tree "Ocean Radar", and them I found the "documentation" link.
  6. Documentation does not have a QC document for Seasonde, only a proposal. Should I assume this document has all the procedures? Even though the document does not specify all the tests I believe.

The idea is to ease the burden of the above search.

ggalibert commented 5 years ago

This comes from a good intention and should be what we try to plan for when creating new files.

In reality, when you have to update several millions of files like for ACORN non QC radials (3 637 649) to put this information back in the file this becomes a big challenge.

As a first step I think we should aim for this document to be linked to the collection metadata record.

atkinsn commented 5 years ago

On an unrelated track, I am about to assign a DOI to the "QC procedures for ACORN radars manual". I am going to place this document on content.aodn, and also suggest to the IMOS office to upload this to the website (I will create a link to the "latest" file, so this link will always point to the latest version). Simone is also "I have already modified the global parameters in the RT data stream to point to the manuals and will soon do the same for the DM data" - and this will point to the DOI URL, which resolve to the metadata record about the manual, which then links to the files.

atkinsn commented 5 years ago

I have added a link to the QC document to all radar collection level metadata records.

ggalibert commented 5 years ago

@scosoli let us know when you have implemented the change in the NetCDF files so that they include information about the QC document and where it can be found and we can close this issue.

scosoli commented 5 years ago

Changes are operational for both RT and DM products already. the link to the doi is included in the metadata. IMOS has requested a few additional edits to the document (the facility name has changed from ACORN to IMOS Ocean Radar Facility) and I am going through the text to change it accordingly

ggalibert commented 5 years ago

Great!

ggalibert commented 5 years ago

@scosoli it doesn't seem to be the case for SeaSonde files yet though. Is this intended? See http://thredds.aodn.org.au/thredds/dodsC/IMOS/ACORN/radial/LANC/2019/03/28/IMOS_ACORN_RV_20190328T000000Z_LANC_FV00_radial.nc.html

scosoli commented 5 years ago

yes, we do not yet perform custom RT QC on seasonde radials for the WA systems and run only manufacturer tests as we need to upgrade site computers, operating system, processing software and license keys first. I have developed tools for DM QC (and planning reprocessing all the historical dataset and running this custom QC). this is being ported for RT application in the NSW codar systems where we run the latest OS and proprietary software.

scosoli commented 5 years ago

at the moment the NSW systems are down - we'll update the document when the RT QC is performed there