NCEI Archival Package Creation is Broken

[kerfoot]

The current NCEI archival package creation is poorly understood and is broken. No new packages have been created and archived by NCEI since July 2022.

Questions

1. What is the current (broken) archival package process?
2. How is it scheduled?

Tasks

1. Understand the current (broken) archival package creation process
2. Build an automated archival package process
3. Determine archival package creation schedule. Potentially different schedules for real-time and delayed-mode processing.
4. Provide a manual process for creating archival packages in the event the automated process breaks again.
5. Full documentation on all of the above posted the the wiki

[benjwadams]

I'm not sure on the status of this now.

Basically, archival takes place if the dataset is completed, marked for submittal to NCEI, and variables pass CF standard name checks. There were some earlier issues with faulty logic causing some submissions not to run the compliance check properly, but some commits have addressed those issues.

[kbailey-noaa]

(Note: Aaron and Kathy were testing the Glider DAC Project Board)

[kerfoot]

Per email communication with John Relph (NCEI) on April 17, 2024:

We normally check for new NGDAC data once a day. However, we have not seen any new data since June 2023.

So it appears that the archival package creation is no longer functional. This should be addressed when time and funding permits.

[benjwadams]

This has now been addressed and I will await feedback from NCEI. We had to restore the Glider DAC server from a previous snapshot which had different configuration setup in the YAML files. This setup was missing configuration variables for the archival script. Going forward it'd be good to possibly ship out these archival logs and set up reasonable metrics for monitoring.

[sarinamann-noaa]

Ben confirmed that this issue is still a priority after hurricane season; NCEI process still experiencing issues. Ben will review the process