Backup strategy

[markmac99]

What to back up

• single and matched parquet files (100MB + 350MB)
• RMSCorrelate: (120GB)
  • raw ftpdetect, config and platepars files
  • trajectory pickles
• fireballs (11GB)
• consolidated (1GB)

Historical ftpdetect, platepars_all and .config from Archive folders.

[markmac99]

One-off copy made (excl archive folders), but need strategy to perform incrementals without scanning the entire bucket. I think a bit of python should be good for this.

[markmac99]

ukmda-shared backup sorted out. Website is more interesting, but can largely be re-created from the data. Images are the only thing we can't easily recreate - consider how much data this would be.

[markmac99]

Answer: lots ! at least 5-8GB/month. Need to consider whether there's any benefit to saving images of faint meteors. Might be worth building something to collect images and mp4s for the brighter events though.

[markmac99]

ok, so i will backup matched data brighter than mag -3, whch is still a LOT of data. i did consider backing up single-station detections brighter than mag 0 but thats a HUGE quantity of data, some of it not even meteors,

[markmac99]

mostly done, but i need to add some cronjobs :)

[markmac99]

done