Unidata / ldm-alchemy

Python tools to handle LDM data feeds and convert to other formats.
MIT License
12 stars 6 forks source link

Issue with GOES Full Disk #8

Closed WxmanJ closed 1 year ago

WxmanJ commented 1 year ago

I am using the latest alchemy repo and am encountering an issue assembling the Full Disk tiles. I receive the following error in the goes restitch log.

[TIRS00 KNES] 2023-07-20 20:55:08,400 [<module>]: Exception raised: Traceback (most recent call last): File "/awips2/ldm/etc/goes-restitch.py", line 429, in <module> loop.run_until_complete(read_stream(loop, read_in, queues, timeout=args.timeout)) File "/home/awips/anaconda3/envs/alchemy/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete return future.result() File "/awips2/ldm/etc/ldm.py", line 118, in read_stream await sink.put(product) File "/awips2/ldm/etc/goes-restitch.py", line 234, in put await self[dataset_name(item, self.filename)].enqueue(item) File "/awips2/ldm/etc/goes-restitch.py", line 44, in dataset_name sat_id = dataset.satellite_id.replace('-', '') File "src/netCDF4/_netCDF4.pyx", line 2922, in netCDF4._netCDF4.Dataset.__getattr__ File "src/netCDF4/_netCDF4.pyx", line 2864, in netCDF4._netCDF4.Dataset.getncattr File "src/netCDF4/_netCDF4.pyx", line 1445, in netCDF4._netCDF4._get_att File "src/netCDF4/_netCDF4.pyx", line 1927, in netCDF4._netCDF4._ensure_nc_success AttributeError: NetCDF: Attribute not found The issue seems to only occur with the Full Disk domain tiles - not CONUS.

mzuranski commented 1 year ago

It looks like the product throwing those errors is TIRS00 KNES, which is not a Cloud and Moisture Imagery product (i.e. not an ABI band). It's the Full-Disk GLM data that was added to NOAAPort/SBN a little while back: https://www.weather.gov/media/notification/pdf_2023_24/scn22-112_goes-r_glm_gridded_data_products_aaa.pdf

Goes-restitch.py was not designed for these GLM tiles, they're likely missing some metadata that this script relies on. My guess is you're still stitching together all the band 1-16 tiles as expected, and it's only this product throwing errors.

If you're receiving your data via LDM, the fix should be as simple as adjusting the pattern, the goal being to filter out the TIRS00 products. Here's an example:

old: NOTHER ^TI(RS).. KNES ([0-9][0-9][0-9][0-9][0-9][0-9]) ...

new: NOTHER ^TI(RS)(0[1-9]|1[0-6]) KNES ([0-9][0-9][0-9][0-9][0-9][0-9]) ...

Your pattern may be different, the important bit is (0[1-9]|1[0-6]) to filter out 00. Also, since this is adding a set of parentheses, if you use positional flags those may need to be shifted. In the above example, \2 would insert the 6-digit datetime, now it needs to be \3.

WxmanJ commented 1 year ago

Thank you, Mike. The new, narrow Full Disk pattern fixed the issue.