Recover from /disks/themisdata drive failures

[jameswilburlewis]

"new justice" suffered a series of disk failures over the weekend. It is now online, in a somewhat degraded state, as justice2. "old justice" has been put back online after being idle for about 2 weeks. We now need to fill in the last two weeks of data products, and continue routine processing until justice2 is repaired and ready to be put back into service.

The recovery plan:

• [x] Export /disks/themisdata from old justice and clear any stale NFS handles on processing servers
• [x] Run bfds_archive to copy predicted ephemerides from /home/bfds/Products/THEMIS to /disks/themisdata/bfds
• [x] Reprocess V00 state CDFs to fill the gap
• [x] Copy any updated FGM cal files from justice2 to justice
• [x] Compile a list of VC files that need to be reprocessed, copy them to somewhere on /dsks/themisdata
• [x] Copy the VC files to the vc_drop ready2process directory and let the automated processing pick them up
• [x] Check that the expected L0 and L1 products have been produced and that the VC files have been added to the appropriate vc_archive directories
• [x] Notify Bryce to start sending VC files that accumulated since the server failure
• [x] Check that the L0 and L1 files and vc_archive directories are now up to date
• [x] Adjust cron job times so that L2 files and summary plots for the most recent dates will be produced ASAP
• [x] Notify SPDF, SSL, NASA, SPEDAS, and PySPEDAS users that data is back online at themis.ssl.berkeley.edu
• [x] Manually reprocess any L2 files or summary plots outside our nominal look-back periods
• [x] Check with Cindy to see if automated processing has filled gaps in GMAG products for rsync-based and wget-based scripts
• [x] Check with Nick to see if automated processing has filled gaps in ASI products
• [x] Update GMAG exclusion list, reprocess affected AE index and summary plots to remove some spikes
• [x] Reprocess ASI mpeg files to mp4
• [x] Check for any other products that might need to be restored from justice2 or other archives

[nickssl]

1. Reprocessed missing data and plots for ASI, KOMPSAT, GOES, POES.

2. Ongoing rerun of the ASI mosaic movies conversion, from mpg to mp4 format. This script requires more than a week to complete, because it has to convert movies from 2007 to now.