Added script to assess processing progress

[awhoward]

Here's the doc string:

# Script Name: check_fits_files.sh
# Description:
#   This script searches through /data/kpf/L0/YYYYMMDD subdirectories for L0 
#   files matching the pattern KP.YYYYMMDD.NNNNN.NN.fits. It records the most 
#   recent modification date of any L0 file and checks if corresponding 
#   KP.YYYYMMDD.NNNNN.NN_2D.fits, KP.YYYYMMDD.NNNNN.NN_L1.fits, and 
#   KP.YYYYMMDD.NNNNN.NN_L2.fits files in respective directories have a file 
#   modification date after the L0 file. For any missing 2D, L1, and L2 files, 
#   the script checks the 'TRIGTARG' keyword in the FITS header of the L0 file 
#   and excludes the file from the missing count if the keyword does not 
#   contain 'Green' or 'Red'. The script outputs a summary for each YYYYMMDD 
#   directory, showing the count of such files and the most recent L0 
#   modification date. The script takes a starting date (YYYYMMDD) as an 
#   argument and optionally an end date and a flag to print missing files.
#
# Usage:
#   ./check_fits_files.sh YYYYMMDD [YYYYMMDD] [--print_missing]
#
# Example:
#   ./check_fits_files.sh 20231114 20231231 --print_missing

[awhoward]

Here's some sample output. @bjfultn and @howardisaacson, you may find this script to be useful.

shrek% ./scripts/kpf_processing_progress.sh 20231120

DATECODE | LAST L0 MOD DATE | 2D PROCESSING  | L1 PROCESSING  | L2 PROCESSING 
------------------------------------------------------------------------------
20231120 | 2023-11-20 16:00 |  571/571  100% |  563/571   98% |  532/571   93%
20231121 | 2023-11-24 13:59 |  422/439   96% |  398/439   90% |  371/439   84%
20231122 | 2023-11-24 14:25 |  496/496  100% |  496/496  100% |  469/496   94%
20231123 | 2023-11-24 15:42 |  145/524   27% |  137/524   26% |  124/524   23%
20231124 | 2023-11-24 16:00 |  186/245   75% |  183/245   74% |  181/245   73%
20231125 | 2023-11-25 01:00 |   50/138   36% |   41/138   29% |   39/138   28%
------------------------------------------------------------------------------

[awhoward]

Updated doc string with new features:

# Script Name: kpf_processing_progress.sh
#
# Description:
#   This script searches through /data/kpf/L0/YYYYMMDD subdirectories for L0 
#   files matching the pattern KP.YYYYMMDD.NNNNN.NN.fits. It records the most 
#   recent modification date of any L0 file and checks if corresponding 
#   KP.YYYYMMDD.NNNNN.NN_2D.fits, KP.YYYYMMDD.NNNNN.NN_L1.fits, and 
#   KP.YYYYMMDD.NNNNN.NN_L2.fits files in respective directories have a file 
#   modification date after the L0 file. For any missing 2D, L1, and L2 files, 
#   the script checks the 'TRIGTARG' keyword in the FITS header of the L0 file 
#   and excludes the file from the missing count if the value of TRIGTARG does 
#   not contain 'Green' or 'Red'. The script outputs a summary for each 
#   YYYYMMDD directory, showing the count of such files and the most recent 
#   L0 modification date. The script takes a starting date (YYYYMMDD) as an 
#   argument and optionally an end date and flags to print missing files and
#   touch the base L0 files of missing 2D/L1/L2 files.
#
# Options:
#   --help           Display this message
#   --print_missing  Display missing file names
#   --touch_missing  Touch the base L0 files of missing 2D/L1/L2 files
#
# Usage:
#   ./check_fits_files.sh YYYYMMDD [YYYYMMDD] [--print_missing]
#
# Example:
#   ./check_fits_files.sh 20231114 20231231 --print_missing
#

[awhoward]

@bjfultn, I think this is ready for merging. It's slow, but that may be because the file system on shrek is really slow right now.

[awhoward]

@bjfultn - this will be ready to merge when it passes the CI tests running now. I added the two to-do items that we discussed as a comment in the script.