search

fathomnet / community-feedback

3 stars 0 forks source link

Finalize FathomNet to NCEI/MSU pipeline #167

Open hohonuuli opened 7 months ago

hohonuuli commented 7 months ago

This is stage 1 of the pipeline (with stage 2 being the periodic migration from MSU to NOAA).

Status

  1. The upload page has been updated to accept zip files (https://fathomnet.org/fathomnet/#/upload). This will repackage them and move them to the staging location at https://fathomnet.org/static/staging/ to be copied by NCEI to their server.
  2. A scanner is running on quasar.shore.mbari.org under my account every hour to look for new packages that have been unpacked on NCEI.
0 * * * * docker rm fv3 > /dev/null 2>&1; docker run --name fv3 mbari/fathomnet-support archive-scanner -a <super secret key> "https://oer.hpc.msstate.edu/FathomNet/staging/" > /u/brian/Library/Logs/quasar-archive-scanner.log     2>&1

TODO

Notes

Links to related issues: #136, #137, #139, #141, #165, fathomnet-support pull request 6,

hohonuuli commented 7 months ago

upload-scanner

This app looks for new imageset uploads and updates the database that a notification has been sent and then sends a notification. I've deployed it on quasar.shore.mbari.org to run every 30 minutes via cron. The cron entry is:

# every 30 minutes scan for new image packages and notify users
*/30 * * * * docker rm fv4 > /dev/null 2>&1; docker run --name fv4 mbari/fathomnet-support upload-scanner -a <api key > /u/brian/Library/Logs/quasar-upload-scanner.log 2>&1