search

pietern / goestools

Tools to work with signals and files from GOES satellites
https://pietern.github.io/goestools/
BSD 2-Clause "Simplified" License
368 stars 82 forks source link

assertion failure: Expected to extract mode and channel from file name #57

Closed eminence closed 5 years ago

eminence commented 5 years ago

I'll try to poke at this later, but just making a record of this here for now

terminate called after throwing an instance of 'std::runtime_error'
  what():  Assertion `rv == 2` failed at /nas/achin/devel/goestools/src/goesproc/handler_goesr.cc:29, Expected to extract mode and channel from file name (OR_ABI-L2-RRQPEF-M6_G16_s20191981650398_e20191981700106_c20191981700191.lrit)
Aborted
tmbates12 commented 5 years ago

Looks like this file name gave it issues too:

what():  Assertion `rv == 2` failed at /root/Builds/goestools/src/goesproc/handler_goesr.cc:29, Expected to
 extract mode and channel from file name (OR_ABI-L2-ACHTF-M6_G16_s20191981810396_e20191981820104_c20191981821149.lrit)
eminence commented 5 years ago

According to docs here, the channel part of the filename is used for "single band CMI and DMW products only". In these two cases, we have different products (RRQPEF --> Rainfall Rate/QPE FullDisc; ACHTF --> Cloud Top Temperature FullDisc).

It looks like the product type is only encoded in the filename. So maybe we have to start parsing the filename to handle products that don't have a channel?

tmbates12 commented 5 years ago

The two parts that would be of importance would probably be the following:

https://github.com/pietern/goestools/blob/b0700de79e9a5b4c2d566fdbdab5d40e5a1a788f/src/goesproc/handler_goesr.cc#L21-L31

https://github.com/pietern/goestools/blob/b0700de79e9a5b4c2d566fdbdab5d40e5a1a788f/src/goesproc/handler_goesr.cc#L69-L84

The filename builder takes in the channel structs in order to stick them in the right folder, I haven't looked at the contents of the products yet, so I'm not sure if they'd need their own color lookup tables as well.

tmbates12 commented 5 years ago

It looks like @pietern has already addressed this commit before, but after running lritdump, it appears that the non CMIP products have an ancillary text header now. 

/root/OR_ABI-L2-RRQPEF-M6_G16_s20191982250398_e20191982300106_c20191982300191.lrit:
Primary (0):
  File type: 0 (Image Data)
  Header header length (bits/bytes): 34080/4260
  Data length (bits/bytes): 235358208/29419776
  Total length (bits/bytes): 235392288/29424036
Image structure (1):
  Bits per pixel: 8
  Columns: 5424
  Lines: 5424
  Compression: 1
Image navigation (2):
  Projection name: geos(-75.0)                     z▒7z▒7▒

  Column scaling: 20425338
  Line scaling: 20425338
  Column offset: 2711
  Line offset: 2711
Image data function (3):
  Data:
    $HALFTONE:=8
    _NAME:=rainfall_rate
    _UNIT:=mm h-1
    0:=0.0000
<-- Rest of Lookup table -->
    255:=99.6186
Annotation (4):
  Text: OR_ABI-L2-RRQPEF-M6_G16_s20191982250398_e20191982300106_c20191982300191.lrit
Time stamp (5):
  Time stamp: 2019-07-17 23:01:04
Ancillary text (6):
  Text: Time of frame start = 2019-07-17T22:50:39.8Z;Satellite = G16;Instrument = FM1;Channel = N/A;Imaging Mode = ABI Mode 6;Region = Full Disk;Reso                                        lution = 2km at nadir;Segmented = no
NOAA LRIT (129):
  Agency signature: NOAA
  Product ID: 16
  Product SubID: 0
  Parameter: 0
  NOAA-specific compression: 1
Rice compression (131):
  Flags: 177
  Pixels per block: 32
  Scan lines per packet: 1
eminence commented 5 years ago

So maybe we should start supporting these products.

Here's an example of what a FD rainfall rate image looks like: https://0x0.st/z9Du.jpg (with associated data)

eminence commented 5 years ago

There is definitely some interesting data here. Here's TPW (Total Precipitable Water)

And SST (Sea Surface Skin Temperature)

tmbates12 commented 5 years ago

I suppose we could create some lookup tables to make the images a bit more pleasing to the eye as well, the problem is we'd need a way to implement a way to distinguish these images, as currently the LUTs are applied based on the channel iirc.

tmbates12 commented 5 years ago

I'm working on the lookups now, here's a pair of examplers for RRQPEF and LSTF: https://imgur.com/a/peEYjTE

k5123 commented 5 years ago

Did you guys check the json files? With most of these products there's a table with floats included that seem to be some min->max gradient for whatever's in the images. Most of those that i've seen had that. Perhaps that could be used for making things look prettier...

Also i have a quick fix for unknown products in my fork. I just don't want to do a PR at this point, i feel this can be implemented much nicer than i did.

tmbates12 commented 5 years ago

Mostly a work in progress, this is what I have so far based on the max/min values: https://0x0.st/z956.txt

tmbates12 commented 5 years ago

And here's some for SSTF, DSIF; it seems that clouds prevent the imager from seeing parts of the image, which I suppose makes sense, like in the sea surface temperature where the clouds make the ocean appear as if it's frigid.

Also, just as an aside, it must be a formatting issue of some sort, but goesproc refuses to apply gradients if I include the new handler in my existing goesproc.conf, so as it stands I have to have a separate instance handling these new products. https://0x0.st/zpsD.conf

eminence commented 5 years ago

I don't know if this is helpful for calibrating the LUTs, but here's a noaa-colorized image of sea-surface temps (I have no idea if its generated from this same GOES product, though. The descriptions are pretty far over my head)

https://www.nhc.noaa.gov/sst/

tmbates12 commented 5 years ago

Ah I just noticed I didn't actually upload the images in my last post, here they are: https://imgur.com/a/e8uN8r7 As for that image you posted, I could adjust the colors to look like that, it is quite a bit more pleasing to the eye.

tmbates12 commented 5 years ago

@eminence, I followed the NESDIS link on the page you just sent and found this image: https://www.ospo.noaa.gov/data/sst/contour/global.cf.gif This ended up being pretty useful and I was able to make a new gradient with it:

[handler.gradient.SSTF]
  points = [
    { units = 271, color = "#cc04ae" },
    { units = 273.25, color = "#b005c6" },
    { units = 275.5, color = "#8f06e4" },
    { units = 277.75, color = "#6c1ff7" },
    { units = 280, color = "#486afa" },
    { units = 282.25, color = "#24b4fc" },
    { units = 284.5, color = "#00ffff" },
    { units = 287.25, color = "#00e696" },
    { units = 289, color = "#00d54b" },
    { units = 291.3, color = "#0cc600" },
    { units = 293.6, color = "#59d800" },
    { units = 295.85, color = "#a5ea00" },
    { units = 298.1, color = "#f2fc00" },
    { units = 300.35, color = "#ffc900" },
    { units = 302.6, color = "#ff9400" },
    { units = 304.85, color = "#ff5500" },
    { units = 307.1, color = "#ff1500" },
    { units = 340, color = "#ffffff" }
  ]

Here's a resulting image: https://imgur.com/a/D0YNC00

eminence commented 5 years ago

That looks pretty awesome! The purple between South America and Africa (and in the Gulf of Mexico) doesn't seem quite right, though.

BTW, what code are you running locally? The code from #58, or did you come up with your own fix?

tmbates12 commented 5 years ago

I'm using your pull, #58, the extraneous purples are all from cloud cover it would seem, here's a side by side with channel 13: https://imgur.com/a/XBvnVyc It makes sense since the cloud tops are going to be much, much colder than the ocean below.

eminence commented 5 years ago

Ahh, very interesting. Good point. I suppose this would have been accounted for in global.cf.gif

tmbates12 commented 5 years ago

Here's a better table for LSTF, based off of this, the color scale for the Aqua and Terra sats https://neo.sci.gsfc.nasa.gov/view.php?datasetId=MOD_LSTD_CLIM_M

  [handler.gradient.LSTF]
  points = [
    { units = 248, color = "#beeaea" },
    { units = 256.75, color = "#3692e6" },
    { units = 265.5, color = "#3251cd" },
    { units = 274.5, color = "#631ebf" },
    { units = 283, color = "#c6019d" },
    { units = 291.75, color = "#e30d00" },
    { units = 300.5, color = "#ec6200" },
    { units = 309.25, color = "#f1c900" },
    { units = 318, color = "#f6ec79" },
    { units = 353.2, color = "#ffffff" },
    { units = 354, color = "#000000" }
  ]

Result: https://imgur.com/a/KQpK6n6

tmbates12 commented 5 years ago

New TPWF from NOAA, and a better DSIF from CIMSS Here's the color tables:

[handler.gradient.DSIF]
  points = [
    { units = 0, color = "#543e20" },
    { units = 500, color = "#a58154" },
    { units = 750, color = "#cfa370" },
    { units = 1000, color = "#c3afc2" },
    { units = 1500, color = "#8181c5" },
    { units = 1750, color = "#6363a3" },
    { units = 2000, color = "#969656" },
    { units = 2500, color = "#fffc02" },
    { units = 2750, color = "#ffd327" },
    { units = 3000, color = "#ff9063" },
    { units = 3500, color = "#ff231e" },
    { units = 4000, color = "#b10032" },
    { units = 4250, color = "#680063" },
    { units = 4500, color = "#8d0090" },
    { units = 4961.3914, color = "#ea00ef" },
    { units = 4980.9245, color = "#000000" }
  ]
  [handler.gradient.TPWF]
  points = [
    { units = 0, color = "#3a290b" },
    { units = 11, color = "#d1a777" },
    { units = 12, color = "#8999ee" },
    { units = 22, color = "#584c96" },
    { units = 23, color = "#9a6a5c" },
    { units = 32, color = "#72a429" },
    { units = 33, color = "#97c03c" },
    { units = 43, color = "#f4fe00" },
    { units = 44, color = "#fd775e" },
    { units = 53, color = "#982a19" },
    { units = 54, color = "#6f0069" },
    { units = 65, color = "#fb00ee" },
    { units = 66, color = "#ffc8ff" },
    { units = 69, color = "#f0bbef" },
    { units = 99, color = "#f069ef" },
    { units = 99.6187, color = "#ffffff" },
  ]

Examples: https://imgur.com/a/dThFOkY All that's left is RRQPEF, which I should finish Soon™

tmbates12 commented 5 years ago

Here's my latest goesproc.conf: https://0x0.st/zpbk.conf It's a modified goesproc-goesr.conf.in, so it should process all the products from GOES-16, just remember to replace ${CMAKE_INSTALL_PREFIX} with whatever you're using on your system. I don't have any ACHTF LRIT files, so I didn't have a chance to test that gradient.

ahobgood commented 5 years ago

I'm using your pull, #58, the extraneous purples are all from cloud cover it would seem, here's a side by side with channel 13: https://imgur.com/a/XBvnVyc It makes sense since the cloud tops are going to be much, much colder than the ocean below.

First off, awesome work @eminence with the quick patch and @tmbates12 with the gradients for all of these Level II products!

https://www.goes-r.gov/products/baseline-SST.html

As seen above, the SSTF product would use the ABI cloud and ice masks to blank out those regions. This data product does not currently seem to use those masks. That said, any temperature below 270.65K (-2.5C) is highly likely to be indicating ice of some form or another -- either sea ice or ice clouds. I'm currently using a slightly modified version of @tmbates12 's SSTF gradient, with a couple of extra points on the cold end to blank out ice and cloud tops.

  [handler.gradient.SSTF]
  points = [
    { units = 180, color = "#ffffff" },
    { units = 270.65, color = "#ffffff" },
    { units = 273.25, color = "#b005c6" },
    { units = 275.5, color = "#8f06e4" },
         <...>

The resulting image looks like this: https://imgur.com/a/Pqyd3mh

There's still some lower clouds that are cluttering the output, but we lack the masks to deal with them.

pietern commented 5 years ago

This is awesome guys!

I took the fix and threw a bunch of refactoring on top to hopefully arrive at the same functionality you're getting now, but with the addition of a products configurable in the configuration.

Some relevant commits:

I ran the following configuration against a few sample L2 LRITs and it looks like it works:

[[handler]]
type = "image"
origin = "goes16"
products = [ "^cmip" ]
directory = "./goes16/non-cmip/{product:short|lower}/{time:%Y-%m-%d}"
filename = "GOES16_{product:short}_{region:short}_{time:%Y%m%dT%H%M%SZ}"
format = "jpg"

  [[handler.map]]
  path = "/usr/local/share/goestools/ne/ne_50m_admin_0_countries_lakes.json"
  color = "#FFFF00"

  [[handler.map]]
  path = "/usr/local/share/goestools/ne/ne_50m_admin_1_states_provinces_lakes.json"
  color = "#FFFF00"
eminence commented 5 years ago

Looks great @pietern -- been running for about 24 hours and things look solid. Closing this issue as fixed!

And thanks again to @tmbates12 for working on these gradients. A small note to anyone trying to copy them with the latest fixes from @pietern -- The product names don't end with the letter F, so instead of [handler.gradient.SSTF], you should use [handler.gradient.SST]

pietern commented 5 years ago

@eminence Excellent! Thank you for the confirmation. You can also use the lowercase product names by the way. The handler ensures all comparisons are not case sensitive.

pietern commented 5 years ago

@tmbates12 Would you be interested in adding the non-CMIP gradients to the default configuration files?

tmbates12 commented 5 years ago

Of course! I can get the pull done today.

pietern commented 5 years ago

@tmbates12 That's awesome! Thank you very much!