search

ssec / sift

A visualization application for satellite imagery
http://sift.ssec.wisc.edu/
GNU General Public License v3.0
49 stars 14 forks source link

Cannot do a Band Difference in 1.1.2 #285

Open ScottLindstrom opened 4 years ago

ScottLindstrom commented 4 years ago

I've loaded up GOES-16 Band 13/Band 15 CONUS sectors. Something like this: OR_ABI-L1b-RadC-M6C13_G16_s20201341001116_e20201341003500_c20201341004030.nc

from iris / the cluster. They load and display in SIFT with no issue. I can make an RGB with them as well. But when I try to make an algebraic, SIFT does not cooperate. I get this trace-back:

 File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/site-packages/uwsift/view/layer_details.py", line 229, in update_display
    self._get_shared_color_limits(shared_info, layer_info, this_prez)
  File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/site-packages/uwsift/view/layer_details.py", line 148, in _get_shared_color_limits
    unit_info = self.document[this_prez.uuid][Info.UNIT_CONVERSION]
  File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/collections/__init__.py", line 916, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/collections/__init__.py", line 908, in __missing__
    raise KeyError(key)
KeyError: <Info.UNIT_CONVERSION: 'unit_conversion'>
DEBUG 17:02:45 layer_tree:setData:L690 setData 0
DEBUG 17:02:45 layer_tree:setData:L703 toggle layer visibility for row 0 to False
Traceback (most recent call last):
  File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/site-packages/uwsift/view/layer_details.py", line 229, in update_display
    self._get_shared_color_limits(shared_info, layer_info, this_prez)
  File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/site-packages/uwsift/view/layer_details.py", line 148, in _get_shared_color_limits
    unit_info = self.document[this_prez.uuid][Info.UNIT_CONVERSION]
  File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/collections/__init__.py", line 916, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "/home/scottl/SIFT/SIFT_1.1.2/lib/python3.7/collections/__init__.py", line 908, in __missing__
    raise KeyError(key)
KeyError: <Info.UNIT_CONVERSION: 'unit_conversion'>

Other unexpected things that happen: It does create the layer, and it put the SWD (the name I give the difference field) at the top of the layer list. It only lists the two layers -- SWD and B13. B15 is somehow vanished until I unclick B13 at which point all 3 layers show up in the layer list window. If I unclick B13 and B15, I see a black field (the SWD). I can't do a colormap change on it -- also get the unit_conversion error above thrown.

The files I used are in ftp:/ftp.ssec.wisc.edu/pub/ssec/scottl/

Or, on the cluster: /arcdata/goes_restricted/grb/goes16/2020/2020_05_13_134/abi/L1b/RadC/

Two important notes:
(1) I am running this by tunneling into crusher from outside. I don't know how that affects what is going on. (2) I'm not completely remembering exactly how SIFT was installed. I run 'source ./SIFT_1.1.2/bin/activate ' before running SIFT to get things going.

djhoese commented 4 years ago

Interestingly I get a different but similar error:

WARNING: Traceback (most recent call last):
  File "/home/davidh/repos/git/sift/uwsift/view/create_algebraic.py", line 147, in _create_algebraic
    self.doc.create_algebraic_composite(operations=operations, namespace=namespace, info=info)
  File "/home/davidh/repos/git/sift/uwsift/model/document.py", line 2101, in create_algebraic_composite
    dataset[Info.UNIT_CONVERSION] = units_conversion(dataset)
  File "/home/davidh/repos/git/sift/uwsift/model/document.py", line 156, in units_conversion
    punits = preferred_units(dsi)
  File "/home/davidh/repos/git/sift/uwsift/model/document.py", line 144, in preferred_units
    if dsi[Info.STANDARD_NAME] == 'toa_bidirectional_reflectance':
  File "/home/davidh/miniconda3/envs/sift_pyqt5_py37/lib/python3.7/collections/__init__.py", line 914, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "/home/davidh/miniconda3/envs/sift_pyqt5_py37/lib/python3.7/collections/__init__.py", line 906, in __missing__
    raise KeyError(key)
KeyError: <Info.STANDARD_NAME: 'standard_name'>

And my SIFT application completely crashes. I think this calls for a 1.1.3 release. I'm running Satpy's master branch so that could explain the difference, but I'm still surprised.

djhoese commented 4 years ago

I think I found two issues, at least for the version of everything I'm running. Two things:

  1. There is a bug in the metadata guidebook stuff that isn't setting standard_name correctly. Once fixed I was able to create a difference layer. However, the limits were terribly wrong.
  2. Satpy's ABI readers are including metadata for valid_range but it is set to (0, 4095) which is the original in-file range. I'm made a PR for Satpy to fix this (https://github.com/pytroll/satpy/pull/1213). Once merged this shouldn't be a problem. I'll have to double check AMI and AHI too.

Edit: Doesn't look like AMI or AHI have this valid_range in their metadata (it's a pretty CF standard kind of thing which neither of those instruments are).

djhoese commented 4 years ago

@ScottLindstrom Is there a chance that there is an error a little higher up in your terminal when you get this error?

ScottLindstrom commented 4 years ago

I didn't see one. All the debug in opening I cleared, and then tried the difference

katherinekolman commented 4 years ago

I tested creating an algebraic layer with the Satpy and SIFT master branches and it seems to be working (at least on my end). I used files from the ftp server and didn't get any error output.

Just to confirm, is this what it is supposed to look like?

s1

I can also do colormap changes:

s2
ScottLindstrom commented 4 years ago

If you change the range to something like -10 to +10, what does the image look like?

From: Katherine Kolman notifications@github.com Sent: Friday, May 22, 2020 5:50 PM To: ssec/sift sift@noreply.github.com Cc: Scott Lindstrom scottl@ssec.wisc.edu; Mention mention@noreply.github.com Subject: Re: [ssec/sift] Cannot do a Band Difference in 1.1.2 (#285)

I tested creating an algebraic layer with the Satpy and SIFT master branches and it seems to be working (at least on my end). I used files from the ftp server and didn't get any error output.

Just to confirm, is this what it is supposed to look like? [s1]https://user-images.githubusercontent.com/30475397/82714429-1911bc00-9c54-11ea-8930-af818a24f04e.PNG

I can also do colormap changes: [s2]https://user-images.githubusercontent.com/30475397/82714481-642bcf00-9c54-11ea-9647-14dfe259efab.PNG

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ssec/sift/issues/285#issuecomment-632936187, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD3IMP3DEXTK2QDQFHEVLMDRS36UFANCNFSM4NDPHCVA.

ScottLindstrom commented 4 years ago

You'll see grey scale yes when you make the difference. I think a range of something like +/- 164?

From: Katherine Kolman notifications@github.com Sent: Friday, May 22, 2020 5:50 PM To: ssec/sift sift@noreply.github.com Cc: Scott Lindstrom scottl@ssec.wisc.edu; Mention mention@noreply.github.com Subject: Re: [ssec/sift] Cannot do a Band Difference in 1.1.2 (#285)

I tested creating an algebraic layer with the Satpy and SIFT master branches and it seems to be working (at least on my end). I used files from the ftp server and didn't get any error output.

Just to confirm, is this what it is supposed to look like? [s1]https://user-images.githubusercontent.com/30475397/82714429-1911bc00-9c54-11ea-8930-af818a24f04e.PNG

I can also do colormap changes: [s2]https://user-images.githubusercontent.com/30475397/82714481-642bcf00-9c54-11ea-9647-14dfe259efab.PNG

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ssec/sift/issues/285#issuecomment-632936187, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD3IMP3DEXTK2QDQFHEVLMDRS36UFANCNFSM4NDPHCVA.

katherinekolman commented 4 years ago

This is what the image looks like when the range is changed to -10 to +10:

s3