search

LowellObservatory / NightShift

Collection of sub-modules used in the NightWatch (and other Night*) project
Mozilla Public License 2.0
1 stars 1 forks source link

GOES NC crash #15

Open astrobokonon opened 3 years ago

astrobokonon commented 3 years ago 
2020-09-04 05:00:04,677 INFO     Reading: ./outputs/goes//raws/20202471648536_C13.nc
2020-09-04 05:00:04,700 ERROR    Traceback (most recent call last):
2020-09-04 05:00:04,700 ERROR      File "looper_goes16aws.py", line 186, in <module>
2020-09-04 05:00:04,700 ERROR    forceDown=forceDownloads, forceRegen=forceRegenPlot)
2020-09-04 05:00:04,700 ERROR      File "looper_goes16aws.py", line 119, in main
2020-09-04 05:00:04,701 ERROR    forceRegen=forceRegen)
2020-09-04 05:00:04,701 ERROR      File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 280, in makePlots
2020-09-04 05:00:04,701 ERROR    pCoeff=pCoeff)
2020-09-04 05:00:04,701 ERROR      File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 121, in crop_image
2020-09-04 05:00:04,701 ERROR    old_grid, imgdata = G16_ABI_L2_ProjDef(dat)
2020-09-04 05:00:04,701 ERROR      File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 53, in G16_ABI_L2_ProjDef
2020-09-04 05:00:04,702 ERROR    imgdata = nc['CMI'][:]
2020-09-04 05:00:04,702 ERROR      File "netCDF4/_netCDF4.pyx", line 4408, in netCDF4._netCDF4.Variable.__getitem__
2020-09-04 05:00:04,702 ERROR      File "netCDF4/_netCDF4.pyx", line 5352, in netCDF4._netCDF4.Variable._get
2020-09-04 05:00:04,702 ERROR      File "netCDF4/_netCDF4.pyx", line 1887, in netCDF4._netCDF4._ensure_nc_success
2020-09-04 05:00:04,702 ERROR    RuntimeError
2020-09-04 05:00:04,702 ERROR    :
2020-09-04 05:00:04,702 ERROR    NetCDF: HDF error

Not sure what's up, seems like the NC files isn't complete and this exception is untrapped for some reason.

astrobokonon commented 3 years ago

I can put bandaids on this, but I really should restructure to make sure all the bits are in the actual datafile before I do anything to it. Otherwise I'll be playing whack-a-mole forever.

astrobokonon commented 3 years ago

After 38ef3ba05d2d88d4e8f04970bb608666d46b3398, 

2020-09-04 05:17:54,984 ERROR    Traceback (most recent call last):
2020-09-04 05:17:54,984 ERROR      File "looper_goes16aws.py", line 186, in <module>
2020-09-04 05:17:54,984 ERROR    forceDown=forceDownloads, forceRegen=forceRegenPlot)
2020-09-04 05:17:54,984 ERROR      File "looper_goes16aws.py", line 119, in main
2020-09-04 05:17:54,985 ERROR    forceRegen=forceRegen)
2020-09-04 05:17:54,985 ERROR      File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 292, in makePlots
2020-09-04 05:17:54,985 ERROR    pCoeff=pCoeff)
2020-09-04 05:17:54,985 ERROR      File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 126, in crop_image
2020-09-04 05:17:54,985 ERROR    old_grid, imgdata = G16_ABI_L2_ProjDef(dat)
2020-09-04 05:17:54,985 ERROR      File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 62, in G16_ABI_L2_ProjDef
2020-09-04 05:17:54,985 ERROR    satH = proj_var.perspective_point_height
2020-09-04 05:17:54,985 ERROR    AttributeError
2020-09-04 05:17:54,985 ERROR    :
2020-09-04 05:17:54,986 ERROR    'NoneType' object has no attribute 'perspective_point_height'
astrobokonon commented 3 years ago

After a54f733776f9a1c0084f95abfc18ee23da463c58,

2020-09-04 05:26:35,159 INFO     Reading: ./outputs/goes//raws/20202471648536_C13.nc
2020-09-04 05:26:35,184 INFO     NetCDF: HDF error
2020-09-04 05:26:35,184 ERROR    Traceback (most recent call last):
2020-09-04 05:26:35,184 ERROR      File "looper_goes16aws.py", line 186, in <module>
2020-09-04 05:26:35,185 ERROR    forceDown=forceDownloads, forceRegen=forceRegenPlot)
2020-09-04 05:26:35,185 ERROR      File "looper_goes16aws.py", line 119, in main
2020-09-04 05:26:35,185 ERROR    forceRegen=forceRegen)
2020-09-04 05:26:35,185 ERROR      File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 293, in makePlots
2020-09-04 05:26:35,185 ERROR    pCoeff=pCoeff)
2020-09-04 05:26:35,185 ERROR      File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 127, in crop_image
2020-09-04 05:26:35,185 ERROR    old_grid, imgdata = G16_ABI_L2_ProjDef(dat)
2020-09-04 05:26:35,186 ERROR      File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 103, in G16_ABI_L2_ProjDef
2020-09-04 05:26:35,186 ERROR    return old_grid, imgdata
2020-09-04 05:26:35,186 ERROR    UnboundLocalError
2020-09-04 05:26:35,186 ERROR    :
2020-09-04 05:26:35,186 ERROR    local variable 'old_grid' referenced before assignment
astrobokonon commented 3 years ago

After 39c01f30bd37db0539d67a4277847076c3c06fea,

2020-09-04 05:31:43,904 INFO     Reading: ./outputs/goes//raws/20202471648536_C13.nc
2020-09-04 05:31:43,927 INFO     NetCDF: HDF error
2020-09-04 05:31:43,927 INFO     2.989587593880576
2020-09-04 05:31:43,928 INFO     3.5251001155878576
2020-09-04 05:31:43,937 INFO     Existing grid information not found! Bad file?
2020-09-04 05:31:43,938 INFO     Image data not found! Bad file?
2020-09-04 05:31:43,938 INFO     Old projection information: None
2020-09-04 05:31:43,938 INFO     NEW projection information: Area ID: lcc_otf
Description: On-the-fly lcc area
Projection: {'ellps': 'WGS84', 'lat_0': '34.7443', 'lat_1': '34.7443', 'lat_2': '34.7443', 'lon_0': '-111.4223', 'proj': 'lcc'}
Number of columns: 1552
Number of rows: 1655
Area extent: (-334576.2194, -331922.8127, 334557.235, 337079.5669)
2020-09-04 05:31:43,939 INFO     (-334576.2193889556, 334557.2349663073, -331922.812666423, 337079.56686652114)
2020-09-04 05:31:43,939 INFO     669133.4543552629
2020-09-04 05:31:43,939 INFO     669002.3795329442
2020-09-04 05:31:43,939 INFO     1.0001959257938817
2020-09-04 05:31:43,939 INFO     (7.0, 7.0)
2020-09-04 05:31:44,087 ERROR    Traceback (most recent call last):
2020-09-04 05:31:44,087 ERROR      File "looper_goes16aws.py", line 186, in <module>
2020-09-04 05:31:44,087 ERROR    forceDown=forceDownloads, forceRegen=forceRegenPlot)
2020-09-04 05:31:44,088 ERROR      File "looper_goes16aws.py", line 119, in main
2020-09-04 05:31:44,088 ERROR    forceRegen=forceRegen)
2020-09-04 05:31:44,088 ERROR      File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 338, in makePlots
2020-09-04 05:31:44,088 ERROR    cmap=cmap)
2020-09-04 05:31:44,088 ERROR      File "/opt/conda/lib/python3.7/site-packages/matplotlib/pyplot.py", line 2684, in imshow
2020-09-04 05:31:44,089 ERROR    None else {}), **kwargs)
2020-09-04 05:31:44,089 ERROR      File "/opt/conda/lib/python3.7/site-packages/cartopy/mpl/geoaxes.py", line 1123, in imshow
2020-09-04 05:31:44,089 ERROR    result = matplotlib.axes.Axes.imshow(self, img, *args, **kwargs)
2020-09-04 05:31:44,090 ERROR      File "/opt/conda/lib/python3.7/site-packages/matplotlib/__init__.py", line 1599, in inner
2020-09-04 05:31:44,090 ERROR    return func(ax, *map(sanitize_sequence, args), **kwargs)
2020-09-04 05:31:44,090 ERROR      File "/opt/conda/lib/python3.7/site-packages/matplotlib/cbook/deprecation.py", line 369, in wrapper
2020-09-04 05:31:44,090 ERROR    return func(*args, **kwargs)
2020-09-04 05:31:44,091 ERROR      File "/opt/conda/lib/python3.7/site-packages/matplotlib/cbook/deprecation.py", line 369, in wrapper
2020-09-04 05:31:44,091 ERROR    return func(*args, **kwargs)
2020-09-04 05:31:44,091 ERROR      File "/opt/conda/lib/python3.7/site-packages/matplotlib/axes/_axes.py", line 5679, in imshow
2020-09-04 05:31:44,092 ERROR    im.set_data(X)
2020-09-04 05:31:44,092 ERROR      File "/opt/conda/lib/python3.7/site-packages/matplotlib/image.py", line 685, in set_data
2020-09-04 05:31:44,093 ERROR    "float".format(self._A.dtype))
2020-09-04 05:31:44,093 ERROR    TypeError
2020-09-04 05:31:44,093 ERROR    :
2020-09-04 05:31:44,093 ERROR    Image data of dtype object cannot be converted to float
astrobokonon commented 3 years ago

After 417b11d3e69e0c3eea13db1283f7d9dee18f07d8, 

2020-09-04 05:39:02,784 INFO     Reading: ./outputs/goes//raws/20202471648536_C13.nc
2020-09-04 05:39:02,807 INFO     NetCDF: HDF error
2020-09-04 05:39:02,808 INFO     2.989587593880576
2020-09-04 05:39:02,808 INFO     3.5251001155878576
2020-09-04 05:39:02,817 INFO     Existing grid information not found! Bad file?
2020-09-04 05:39:02,817 INFO     Image data not found! Bad file?
2020-09-04 05:39:02,817 INFO     Old projection information: None
2020-09-04 05:39:02,818 INFO     NEW projection information: Area ID: lcc_otf
Description: On-the-fly lcc area
Projection: {'ellps': 'WGS84', 'lat_0': '34.7443', 'lat_1': '34.7443', 'lat_2': '34.7443', 'lon_0': '-111.4223', 'proj': 'lcc'}
Number of columns: 1552
Number of rows: 1655
Area extent: (-334576.2194, -331922.8127, 334557.235, 337079.5669)
2020-09-04 05:39:02,818 INFO     Image data not found, skipping file.
2020-09-04 05:39:02,818 ERROR    Traceback (most recent call last):
2020-09-04 05:39:02,818 ERROR      File "looper_goes16aws.py", line 186, in <module>
2020-09-04 05:39:02,819 ERROR    forceDown=forceDownloads, forceRegen=forceRegenPlot)
2020-09-04 05:39:02,819 ERROR      File "looper_goes16aws.py", line 119, in main
2020-09-04 05:39:02,819 ERROR    forceRegen=forceRegen)
2020-09-04 05:39:02,819 ERROR      File "/home/lig/Codes/NightShift/nightshift/goes/plot.py", line 380, in makePlots
2020-09-04 05:39:02,819 ERROR    del crs, fig, ax, line1, line2, ngrid, ndat
2020-09-04 05:39:02,819 ERROR    UnboundLocalError
2020-09-04 05:39:02,819 ERROR    :
2020-09-04 05:39:02,819 ERROR    local variable 'crs' referenced before assignment

So close! I realized this before the last commit but it's late and I forgot.

astrobokonon commented 3 years ago

Commit 1680378e28a802d28f562e7b99a44065e6f2a9ed is truly stupid, and I hate it. But, it's the least impactful change that I can do right now to work around bad files before I properly check the contents of these things before I do anything else. That's a much bigger change in logic, so this is fine for now until I have time to circle back around and fix it properly.

The log (and CPU usage) is now blissfully silent. Yay.

astrobokonon commented 3 years ago

This is going to stay open until the logic is changed for BOTH GOES and NEXRAD data to make sure everything in them is in there, before any actual processing on them starts. That's the only way to fix this properly!

astrobokonon commented 3 years ago

For GOES, it seems like the best place to take care of this is in nightshift.goes.plot.crop_image() since that's where the file is actually read in. The radar one is of course different, and the file is read in during the main loop inside nightshift.radar.plot.makePlots(). Maybe I should make them more similar since there's some commented out bits on the GOES side from when I was hunting memory leaks? Or maybe I should make the radar one more like the GOES one.