Open astrobokonon opened 3 years ago
I can put bandaids on this, but I really should restructure to make sure all the bits are in the actual datafile before I do anything to it. Otherwise I'll be playing whack-a-mole forever.
After 38ef3ba05d2d88d4e8f04970bb608666d46b3398,
2020-09-04 05:17:54,984 ERROR Traceback (most recent call last):
2020-09-04 05:17:54,984 ERROR File "looper_goes16aws.py", line 186, in <module>
2020-09-04 05:17:54,984 ERROR forceDown=forceDownloads, forceRegen=forceRegenPlot)
2020-09-04 05:17:54,984 ERROR File "looper_goes16aws.py", line 119, in main
2020-09-04 05:17:54,985 ERROR forceRegen=forceRegen)
2020-09-04 05:17:54,985 ERROR File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 292, in makePlots
2020-09-04 05:17:54,985 ERROR pCoeff=pCoeff)
2020-09-04 05:17:54,985 ERROR File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 126, in crop_image
2020-09-04 05:17:54,985 ERROR old_grid, imgdata = G16_ABI_L2_ProjDef(dat)
2020-09-04 05:17:54,985 ERROR File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 62, in G16_ABI_L2_ProjDef
2020-09-04 05:17:54,985 ERROR satH = proj_var.perspective_point_height
2020-09-04 05:17:54,985 ERROR AttributeError
2020-09-04 05:17:54,985 ERROR :
2020-09-04 05:17:54,986 ERROR 'NoneType' object has no attribute 'perspective_point_height'
After a54f733776f9a1c0084f95abfc18ee23da463c58,
2020-09-04 05:26:35,159 INFO Reading: ./outputs/goes//raws/20202471648536_C13.nc
2020-09-04 05:26:35,184 INFO NetCDF: HDF error
2020-09-04 05:26:35,184 ERROR Traceback (most recent call last):
2020-09-04 05:26:35,184 ERROR File "looper_goes16aws.py", line 186, in <module>
2020-09-04 05:26:35,185 ERROR forceDown=forceDownloads, forceRegen=forceRegenPlot)
2020-09-04 05:26:35,185 ERROR File "looper_goes16aws.py", line 119, in main
2020-09-04 05:26:35,185 ERROR forceRegen=forceRegen)
2020-09-04 05:26:35,185 ERROR File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 293, in makePlots
2020-09-04 05:26:35,185 ERROR pCoeff=pCoeff)
2020-09-04 05:26:35,185 ERROR File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 127, in crop_image
2020-09-04 05:26:35,185 ERROR old_grid, imgdata = G16_ABI_L2_ProjDef(dat)
2020-09-04 05:26:35,186 ERROR File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 103, in G16_ABI_L2_ProjDef
2020-09-04 05:26:35,186 ERROR return old_grid, imgdata
2020-09-04 05:26:35,186 ERROR UnboundLocalError
2020-09-04 05:26:35,186 ERROR :
2020-09-04 05:26:35,186 ERROR local variable 'old_grid' referenced before assignment
After 39c01f30bd37db0539d67a4277847076c3c06fea,
2020-09-04 05:31:43,904 INFO Reading: ./outputs/goes//raws/20202471648536_C13.nc
2020-09-04 05:31:43,927 INFO NetCDF: HDF error
2020-09-04 05:31:43,927 INFO 2.989587593880576
2020-09-04 05:31:43,928 INFO 3.5251001155878576
2020-09-04 05:31:43,937 INFO Existing grid information not found! Bad file?
2020-09-04 05:31:43,938 INFO Image data not found! Bad file?
2020-09-04 05:31:43,938 INFO Old projection information: None
2020-09-04 05:31:43,938 INFO NEW projection information: Area ID: lcc_otf
Description: On-the-fly lcc area
Projection: {'ellps': 'WGS84', 'lat_0': '34.7443', 'lat_1': '34.7443', 'lat_2': '34.7443', 'lon_0': '-111.4223', 'proj': 'lcc'}
Number of columns: 1552
Number of rows: 1655
Area extent: (-334576.2194, -331922.8127, 334557.235, 337079.5669)
2020-09-04 05:31:43,939 INFO (-334576.2193889556, 334557.2349663073, -331922.812666423, 337079.56686652114)
2020-09-04 05:31:43,939 INFO 669133.4543552629
2020-09-04 05:31:43,939 INFO 669002.3795329442
2020-09-04 05:31:43,939 INFO 1.0001959257938817
2020-09-04 05:31:43,939 INFO (7.0, 7.0)
2020-09-04 05:31:44,087 ERROR Traceback (most recent call last):
2020-09-04 05:31:44,087 ERROR File "looper_goes16aws.py", line 186, in <module>
2020-09-04 05:31:44,087 ERROR forceDown=forceDownloads, forceRegen=forceRegenPlot)
2020-09-04 05:31:44,088 ERROR File "looper_goes16aws.py", line 119, in main
2020-09-04 05:31:44,088 ERROR forceRegen=forceRegen)
2020-09-04 05:31:44,088 ERROR File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 338, in makePlots
2020-09-04 05:31:44,088 ERROR cmap=cmap)
2020-09-04 05:31:44,088 ERROR File "/opt/conda/lib/python3.7/site-packages/matplotlib/pyplot.py", line 2684, in imshow
2020-09-04 05:31:44,089 ERROR None else {}), **kwargs)
2020-09-04 05:31:44,089 ERROR File "/opt/conda/lib/python3.7/site-packages/cartopy/mpl/geoaxes.py", line 1123, in imshow
2020-09-04 05:31:44,089 ERROR result = matplotlib.axes.Axes.imshow(self, img, *args, **kwargs)
2020-09-04 05:31:44,090 ERROR File "/opt/conda/lib/python3.7/site-packages/matplotlib/__init__.py", line 1599, in inner
2020-09-04 05:31:44,090 ERROR return func(ax, *map(sanitize_sequence, args), **kwargs)
2020-09-04 05:31:44,090 ERROR File "/opt/conda/lib/python3.7/site-packages/matplotlib/cbook/deprecation.py", line 369, in wrapper
2020-09-04 05:31:44,090 ERROR return func(*args, **kwargs)
2020-09-04 05:31:44,091 ERROR File "/opt/conda/lib/python3.7/site-packages/matplotlib/cbook/deprecation.py", line 369, in wrapper
2020-09-04 05:31:44,091 ERROR return func(*args, **kwargs)
2020-09-04 05:31:44,091 ERROR File "/opt/conda/lib/python3.7/site-packages/matplotlib/axes/_axes.py", line 5679, in imshow
2020-09-04 05:31:44,092 ERROR im.set_data(X)
2020-09-04 05:31:44,092 ERROR File "/opt/conda/lib/python3.7/site-packages/matplotlib/image.py", line 685, in set_data
2020-09-04 05:31:44,093 ERROR "float".format(self._A.dtype))
2020-09-04 05:31:44,093 ERROR TypeError
2020-09-04 05:31:44,093 ERROR :
2020-09-04 05:31:44,093 ERROR Image data of dtype object cannot be converted to float
After 417b11d3e69e0c3eea13db1283f7d9dee18f07d8,
2020-09-04 05:39:02,784 INFO Reading: ./outputs/goes//raws/20202471648536_C13.nc
2020-09-04 05:39:02,807 INFO NetCDF: HDF error
2020-09-04 05:39:02,808 INFO 2.989587593880576
2020-09-04 05:39:02,808 INFO 3.5251001155878576
2020-09-04 05:39:02,817 INFO Existing grid information not found! Bad file?
2020-09-04 05:39:02,817 INFO Image data not found! Bad file?
2020-09-04 05:39:02,817 INFO Old projection information: None
2020-09-04 05:39:02,818 INFO NEW projection information: Area ID: lcc_otf
Description: On-the-fly lcc area
Projection: {'ellps': 'WGS84', 'lat_0': '34.7443', 'lat_1': '34.7443', 'lat_2': '34.7443', 'lon_0': '-111.4223', 'proj': 'lcc'}
Number of columns: 1552
Number of rows: 1655
Area extent: (-334576.2194, -331922.8127, 334557.235, 337079.5669)
2020-09-04 05:39:02,818 INFO Image data not found, skipping file.
2020-09-04 05:39:02,818 ERROR Traceback (most recent call last):
2020-09-04 05:39:02,818 ERROR File "looper_goes16aws.py", line 186, in <module>
2020-09-04 05:39:02,819 ERROR forceDown=forceDownloads, forceRegen=forceRegenPlot)
2020-09-04 05:39:02,819 ERROR File "looper_goes16aws.py", line 119, in main
2020-09-04 05:39:02,819 ERROR forceRegen=forceRegen)
2020-09-04 05:39:02,819 ERROR File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 380, in makePlots
2020-09-04 05:39:02,819 ERROR del crs, fig, ax, line1, line2, ngrid, ndat
2020-09-04 05:39:02,819 ERROR UnboundLocalError
2020-09-04 05:39:02,819 ERROR :
2020-09-04 05:39:02,819 ERROR local variable 'crs' referenced before assignment
So close! I realized this before the last commit but it's late and I forgot.
Commit 1680378e28a802d28f562e7b99a44065e6f2a9ed is truly stupid, and I hate it. But, it's the least impactful change that I can do right now to work around bad files before I properly check the contents of these things before I do anything else. That's a much bigger change in logic, so this is fine for now until I have time to circle back around and fix it properly.
The log (and CPU usage) is now blissfully silent. Yay.
This is going to stay open until the logic is changed for BOTH GOES and NEXRAD data to make sure everything in them is in there, before any actual processing on them starts. That's the only way to fix this properly!
For GOES, it seems like the best place to take care of this is in nightshift.goes.plot.crop_image()
since that's where the file is actually read in. The radar one is of course different, and the file is read in during the main loop inside nightshift.radar.plot.makePlots()
. Maybe I should make them more similar since there's some commented out bits on the GOES side from when I was hunting memory leaks? Or maybe I should make the radar one more like the GOES one.
Not sure what's up, seems like the NC files isn't complete and this exception is untrapped for some reason.