Periodic tasks for Docker VM services (backup, database dumps, cleanup, etc.)

[nomoon]

Summary

This issue is about organizing the various periodic processes that the system needs to perform—daily dumps of database or parts thereof, cleaning up of Collections files, other "Cron" jobs—under the purview of one Docker container.

TODOs:

• [x] Find/enumerate all the different periodic task scripts that currently exist.
• [x] Build Docker container image for orchestrating periodic tasks.
• [x] Move, port, and adjust scripts as necessary to operate under Docker.
• [ ] Create any critical new scripts for periodic tasks if necessary.
• [x] Integrate orchestrator container into Docker network infrastructure/deploy configuration.

Other links or relevant information

TODO: List found scripts here.

::: Automatic :::

• [ ] Daily copy of FM backups from 158 to 154//Volumes/cdli_archivalfiles /databases/fmpro_all/FM16_auto_bkp (158 to 154)
  • [x] to core : A daily safe copy of the catalogue that is overwritten daily in cdlicore/catalogue; cdli_catalogue.csv is the file and the script safe_catalogue_dump.php which is run through cron
  • [x] to core : A daily dump of all transliteraqtions in ATF format, stored in cdlicore/transliterations, the script is atfbackup.php, run through cron. The script dumps both a daily copy of all atf and a copy of public atf, and pushes older version to folder "datesBackups" with dated versions.
  • [x] to core : A daily conversion of data and push to Github of the most recent version of both the catalogue data and the atf from the pre-cited scripts, the script is cdlicore/make_data_public.sh and the local copy of the processed result is saved at Library/WebServer/Documents/cdli/bulk_data
  • [x] to core : the 3 most recent mysql dumps in cdlicore/mysql_dumps, not sure where the dump script is.
  • [x] to core : Weekly dump overwriting the last one of version history of the transliterations. script and data in cdlicore/version_histories
  • [x] to core : Daily dump of the databases to cdlicore (156 to 156)[ ]
  • [ ] Daily rsync copies of 154 to Qnap, lacie, Oxford, Berlin, CC (to be set up but ready). (These should the Qnap as a source ASAP).
  • [ ] CDLI 154 to QNAP rsync script is in /Users/cdli/CDLI-test-rsync-from154-toQnap.sh, on the CDLI 154 server

:::Should be automatic::: (not blocking)

• [ ] CDLI publication files in cdlicore/publications (not sure where the original is, as I think there is more than just html and pdf)
• [ ] Monthly copy to archival: cdlicat.csv (exported from FM?)
• [ ] Monthly copy to archival: copy of the mysql dump from cdlicore
• [ ] Monthly copy to archival: copy of cdliatf (so the copy of atf dump that has all the atf not just the public)
• [ ] Monthly copy to archival: revhistory.log.zip
• [ ] Monthly copy to archival: all publications to varia

:::To create:::

• [ ] Backup strategy for VM data volumes (DL + other webroot)
• [ ] Cleanup of collections (not blocking)
• [x] Cleanup of any temp data ( eg upload atf files and logs ?) (not blocking)
• [ ] Daily dump of a csv file with list of all public images/files (not blocking)
• [ ] backup a copy of code releases for Framework (not blocking)
• [ ] Backup the project google drive (not blocking)

Roadmap Data

🗓 Start Date: 2018-06-05

🗓 Expected Date:

💪 Label:

📈 Progress (0-1): 0

[nomoon]

@epageperron Who else should I tag if I'm looking to make a list of all the existing periodic tasks to integrate?

[epageperron]

Unfortunately except Prashant, me and Bob...

[epageperron]

This is a draft, will be updated soon.

Qnap new folder structure suggestion

• glacier The glacier folder is the long term backup folder where no file should be updated manually. All files present are created automatically by a backup script
  • static_files
    • dl
    • webroot
  • databases
    • FMPro
    • MariaDB
  • data
  • varia
• raw_images_new
• archival_files_new
• varia
• collections_new Do we still need the collections folder on the Qnap or should that be a mount on the Production VM 157? If we have the collections folder only on Production VM 157, when transferring the content from archival 154 or Qnap to the Production VM, or if the Production VM 157 has to be rebuilt, only the structure needs to be preserved.

Orchestrator

Dumps

• [ ] Daily dump of the whole database
• [ ] Daily dump of the whole_text field of all artifact entries, one file with public entries only and one file with all the texts
• [ ]

Clean-ups

147

rsync of 157 mounts to Qnap

Daily copy of DL and webroot to /glacier/static_files/ Daily copy of dumps/database to glacier/databases/MariaDB Daily copy of dumps/data to /glacier/data Destination: Other info:

rsync of 158 dump to Qnap

Daily rsync of the 158 db dump folder to /databases/FMPro

rsync of Qnap to partners

Daily rsync of Qnap to partners

[epageperron]

Ok I think we need to seriously look over all of this and implement our new backup strategy. 154's os disk is always full for some reason, the varia drive has failed so this task should be solved before spring @danielatucla2010 @nomoon

[epageperron]

https://gitlab.com/cdli/framework/issues/102