search

ioos / glider-dac

The IOOS Glider DAC site/scripts/tools
http://gliders.ioos.us/providers/
8 stars 13 forks source link

NCEI Archival Package Creation is Broken #233

Open kerfoot opened 1 year ago

kerfoot commented 1 year ago

The current NCEI archival package creation is poorly understood and is broken. No new packages have been created and archived by NCEI since July 2022.

Questions

  1. What is the current (broken) archival package process?
  2. How is it scheduled?

Tasks

  1. Understand the current (broken) archival package creation process
  2. Build an automated archival package process
  3. Determine archival package creation schedule. Potentially different schedules for real-time and delayed-mode processing.
  4. Provide a manual process for creating archival packages in the event the automated process breaks again.
  5. Full documentation on all of the above posted the the wiki
benjwadams commented 1 year ago

I'm not sure on the status of this now.

Basically, archival takes place if the dataset is completed, marked for submittal to NCEI, and variables pass CF standard name checks. There were some earlier issues with faulty logic causing some submissions not to run the compliance check properly, but some commits have addressed those issues.

kbailey-noaa commented 10 months ago

(Note: Aaron and Kathy were testing the Glider DAC Project Board)

kerfoot commented 3 months ago

Per email communication with John Relph (NCEI) on April 17, 2024:

We normally check for new NGDAC data once a day. However, we have not seen any new data since June 2023.

So it appears that the archival package creation is no longer functional. This should be addressed when time and funding permits.

benjwadams commented 2 months ago

This has now been addressed and I will await feedback from NCEI. We had to restore the Glider DAC server from a previous snapshot which had different configuration setup in the YAML files. This setup was missing configuration variables for the archival script. Going forward it'd be good to possibly ship out these archival logs and set up reasonable metrics for monitoring.