KeyError when creating true_color for ABI

[WxmanJ]

Describe the bug Receiving a no dataset matching error.

To Reproduce

from satpy import Scene, find_files_and_readers
import satpy
import pyorbital
from glob import glob

files = glob("/home/awips/scripts/raw_satpy/Test/*")

scn = Scene(filenames=files, reader='abi_l1b')
scn.load(['true_color'])
scn.save_dataset('true_color', filename='/home/awips/scripts/satpy_truecolor.png')

Expected behavior Generate a true color image from the 3 ABI channels (1-3). The Test folder contains a file from each channel at the same time. They are all level1b radiance.

Actual results

(satpy) [awips@justin scripts]$ python satpy_channel2.py 
/home/awips/anaconda3/envs/satpy/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/home/awips/anaconda3/envs/satpy/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/home/awips/anaconda3/envs/satpy/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/home/awips/anaconda3/envs/satpy/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
Platform file /home/awips/anaconda3/envs/satpy/lib/python3.6/site-packages/satpy/etc/platforms.txt not found.
Traceback (most recent call last):
  File "/home/awips/anaconda3/envs/satpy/lib/python3.6/site-packages/satpy/readers/__init__.py", line 297, in __getitem__
    return super(DatasetDict, self).__getitem__(item)
KeyError: 'true_color'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "satpy_channel2.py", line 11, in <module>
    scn.save_dataset('true_color', filename='/home/awips/scripts/satpy_truecolor.png')
  File "/home/awips/anaconda3/envs/satpy/lib/python3.6/site-packages/satpy/scene.py", line 1051, in save_dataset
    return writer.save_dataset(self[dataset_id], filename=filename,
  File "/home/awips/anaconda3/envs/satpy/lib/python3.6/site-packages/satpy/scene.py", line 616, in __getitem__
    return self.datasets[key]
  File "/home/awips/anaconda3/envs/satpy/lib/python3.6/site-packages/satpy/readers/__init__.py", line 299, in __getitem__
    key = self.get_key(item)
  File "/home/awips/anaconda3/envs/satpy/lib/python3.6/site-packages/satpy/readers/__init__.py", line 288, in get_key
    best=best, **dfilter)
  File "/home/awips/anaconda3/envs/satpy/lib/python3.6/site-packages/satpy/readers/__init__.py", line 239, in get_key
    raise KeyError("No dataset matching '{}' found".format(str(key)))
KeyError: "No dataset matching 'DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)' found"

Screenshots If applicable, add screenshots to help explain your problem.

Environment Info:

• OS: Linux with Anaconda3

[djhoese]

You need to resample the bands to the same resolution. To keep the same projection you can use the native resample:

new_scn = scn.resample(resampler='native')

[WxmanJ]

Thank you, David. That fixed the issue.

Trying to create a full disk image seems almost impossible due to file size. Is there any way to reduce it without impacting resolution? I know switching the file format can lower it, but it is still pretty extreme.

[djhoese]

You have a couple, but not many, options. I know you said without impacting resolution but the easiest solution is to use the 1km resolution of the full disk instead of the 500m resolution. You can do this by doing:

new_scn = scn.resample(scn.min_area(), resampler='native')

This is picking the lowest resolution out of the currently loaded datasets. So if you are loading more than true_color you will get 2km resolution for normal abi_l1b data.

Your next option is to compress your geotiffs and optionally tile them for even better compression. It has been a while since I've done this but you should be able to accomplish by playing with the TILED/BLOCKXSIZE/BLOCKYSIZE options from gdal (which satpy and rasterio are using under the hood, https://www.gdal.org/frmt_gtiff.html). You should be able to pass these to save_dataset or save_datasets by doing:

new_scn.save_datasets(..., tiled='YES', blockxsize=2048, blockysize=2048, compress='LZW')

If I recall correctly we default to DEFLATE compression which should be giving the best/smallest results. @mraspaud can correct me if I'm wrong.

[djhoese]

@WxmanJ any update on this? Did this help?

[djhoese]

I'm going to consider this solved. If you are still having issues feel free to reopen.