search

ioos / glider-dac

The IOOS Glider DAC site/scripts/tools
http://gliders.ioos.us/providers/
9 stars 13 forks source link

Scheduling the application of QARTOD tests for real-time data sets #279

Open kerfoot opened 1 year ago

kerfoot commented 1 year ago

There is some inconsistency/significant latency in when QARTOD results are made available in the real-time data sets (#268, #274). This has downstream consequences for the use of these results to determine suitability for GTS release.

Request the following action items:

  1. Examine the current mechanism for identifying files that need to be queued for QC for real-time data sets.
  2. Run a series of test to quantify how long it takes to run QC on a file and examine resource allocation.
  3. Use the answers from 1 and 2 to develop a plan for scheduling the application of all QARTOD tests to real-time active data sets only in order to decrease the latency with which the results are available.
  4. Ensure that only the valid results/variables of the QARTOD tests are made available in the ERDDAP data sets asap (#278).

The corresponding plan for application of QARTOD to delayed mode data sets will be addressed in a separate issue.

benjwadams commented 1 year ago

Currently implementing in branch https://github.com/benjwadams/glider-dac/tree/event_driven_qartod

benjwadams commented 1 year ago

283 greatly reduced latency for 3) -- files are QCed quite often now.