Open astrobokonon opened 3 years ago
This is entirely related to issue #15 since I still haven't had time to restructure and make sure the files are complete before processing them...but this particular issue wouldn't have made it to that point so they're still distinct issues.
So to fix this one:
An easy check would be to regex the expected file name
KFSX20210212_145129 stationNameYYYYMMDD_HHMMSS
I could also just wrap the strptime()
to parse the datetime in a try/except block and kick the file out if it fails. But that might not catch and clear the file during the file age check, so I should check on things over there too.
Still a problem in radar:
2021-06-29 06:45:59,819 INFO Reading: ./outputs/radar//raws/KFSX20210628_113909.C6C4d5ab
2021-06-29 06:46:00,631 ERROR /opt/conda/lib/python3.8/site-packages/pyart/io/nexrad_archive.py:178: UserWarning: Fixed_angle(s) greater than 180 degrees present. Assuming angle to be negative so subtrating 360
warnings.warn("Fixed_angle(s) greater than 180 degrees present."
2021-06-29 06:46:00,778 INFO Done reading!
2021-06-29 06:46:00,781 ERROR Traceback (most recent call last):
2021-06-29 06:46:00,781 ERROR File "looper_nexradaws.py", line 156, in <module>
2021-06-29 06:46:00,781 ERROR main(outdir, creds, sleep=90.,
2021-06-29 06:46:00,781 ERROR File "looper_nexradaws.py", line 97, in main
2021-06-29 06:46:00,781 ERROR nplots = plot.makePlots(dout, pout, mapcenter, cmap=gcmap,
2021-06-29 06:46:00,782 ERROR File "/home/lig/Codes/NightShift/nightshift/radar/plot.py", line 192, in makePlots
2021-06-29 06:46:00,782 ERROR tend = dt.strptime(fullts, "%Y%m%d_%H%M%S")
2021-06-29 06:46:00,782 ERROR File "/opt/conda/lib/python3.8/_strptime.py", line 568, in _strptime_datetime
2021-06-29 06:46:00,782 ERROR tt, fraction, gmtoff_fraction = _strptime(data_string, format)
2021-06-29 06:46:00,782 ERROR File "/opt/conda/lib/python3.8/_strptime.py", line 352, in _strptime
2021-06-29 06:46:00,783 ERROR raise ValueError("unconverted data remains: %s" %
2021-06-29 06:46:00,783 ERROR ValueError
2021-06-29 06:46:00,783 ERROR :
2021-06-29 06:46:00,783 ERROR unconverted data remains: .C6C4d5ab
Caught it only by monitoring the docker usage and wondering why my load % spiked and stayed there, and my container uptime tanked.
Looks like a partially downloaded file snuck in to the processing loop at some point: