Skip Partially Downloaded Files from Processing

[astrobokonon]

Looks like a partially downloaded file snuck in to the processing loop at some point:

2021-02-13 05:58:21,573 INFO     Reading: ./outputs/radar//raws/KFSX20210212_164326.8F400797
2021-02-13 05:58:21,826 ERROR    /opt/conda/lib/python3.7/site-packages/pyart/io/nexrad_archive.py:177: UserWarning: Fixed_angle(s) greater than 180 degrees present. Assuming angle to be negative so subtrating 360
  UserWarning)
2021-02-13 05:58:21,896 INFO     Done reading!
2021-02-13 05:58:21,898 ERROR    Traceback (most recent call last):
2021-02-13 05:58:21,898 ERROR      File "looper_nexradaws.py", line 157, in <module>
2021-02-13 05:58:21,898 ERROR    forceDown=forceDownloads, forceRegen=forceRegenPlot)
2021-02-13 05:58:21,898 ERROR      File "looper_nexradaws.py", line 99, in main
2021-02-13 05:58:21,898 ERROR    forceRegen=forceRegen)
2021-02-13 05:58:21,898 ERROR      File "/home/lig/Codes/NightShift/nightshift/radar/plot.py", line 192, in makePlots
2021-02-13 05:58:21,899 ERROR    tend = dt.strptime(fullts, "%Y%m%d_%H%M%S")
2021-02-13 05:58:21,899 ERROR      File "/opt/conda/lib/python3.7/_strptime.py", line 577, in _strptime_datetime
2021-02-13 05:58:21,899 ERROR    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
2021-02-13 05:58:21,899 ERROR      File "/opt/conda/lib/python3.7/_strptime.py", line 362, in _strptime
2021-02-13 05:58:21,899 ERROR    data_string[found.end():])
2021-02-13 05:58:21,899 ERROR    ValueError
2021-02-13 05:58:21,899 ERROR    :
2021-02-13 05:58:21,900 ERROR    unconverted data remains: .8F400797
2021-02-13 05:58:25,627 ERROR    /home/lig/Codes/ligmos/ligmos/utils/confparsers.py:30: DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in future versions. Use ConfigParser directly instead.
  config = conf.SafeConfigParser()

[astrobokonon]

This is entirely related to issue #15 since I still haven't had time to restructure and make sure the files are complete before processing them...but this particular issue wouldn't have made it to that point so they're still distinct issues.

[astrobokonon]

So to fix this one:

An easy check would be to regex the expected file name

KFSX20210212_145129 stationNameYYYYMMDD_HHMMSS

I could also just wrap the strptime() to parse the datetime in a try/except block and kick the file out if it fails. But that might not catch and clear the file during the file age check, so I should check on things over there too.

[astrobokonon]

Still a problem in radar:

2021-06-29 06:45:59,819 INFO     Reading: ./outputs/radar//raws/KFSX20210628_113909.C6C4d5ab
2021-06-29 06:46:00,631 ERROR    /opt/conda/lib/python3.8/site-packages/pyart/io/nexrad_archive.py:178: UserWarning: Fixed_angle(s) greater than 180 degrees present. Assuming angle to be negative so subtrating 360
  warnings.warn("Fixed_angle(s) greater than 180 degrees present."
2021-06-29 06:46:00,778 INFO     Done reading!
2021-06-29 06:46:00,781 ERROR    Traceback (most recent call last):
2021-06-29 06:46:00,781 ERROR      File "looper_nexradaws.py", line 156, in <module>
2021-06-29 06:46:00,781 ERROR    main(outdir, creds, sleep=90.,
2021-06-29 06:46:00,781 ERROR      File "looper_nexradaws.py", line 97, in main
2021-06-29 06:46:00,781 ERROR    nplots = plot.makePlots(dout, pout, mapcenter, cmap=gcmap,
2021-06-29 06:46:00,782 ERROR      File "/home/lig/Codes/NightShift/nightshift/radar/plot.py", line 192, in makePlots
2021-06-29 06:46:00,782 ERROR    tend = dt.strptime(fullts, "%Y%m%d_%H%M%S")
2021-06-29 06:46:00,782 ERROR      File "/opt/conda/lib/python3.8/_strptime.py", line 568, in _strptime_datetime
2021-06-29 06:46:00,782 ERROR    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
2021-06-29 06:46:00,782 ERROR      File "/opt/conda/lib/python3.8/_strptime.py", line 352, in _strptime
2021-06-29 06:46:00,783 ERROR    raise ValueError("unconverted data remains: %s" %
2021-06-29 06:46:00,783 ERROR    ValueError
2021-06-29 06:46:00,783 ERROR    :
2021-06-29 06:46:00,783 ERROR    unconverted data remains: .C6C4d5ab

Caught it only by monitoring the docker usage and wondering why my load % spiked and stayed there, and my container uptime tanked.