pytroll / pyspectral

Pyspectral is a package to read and manipulate satellite sensor spectral responses and solar irradiance spectra
https://pyspectral.readthedocs.org/
GNU General Public License v3.0
65 stars 38 forks source link

[SatPy] Can't create ABI full disk true color #45

Closed adioses closed 6 years ago

adioses commented 6 years ago

This is the code I'm using

# 
from satpy import Scene, find_files_and_readers
from satpy.resample import get_area_def
from pyresample.utils import get_area_def

area_id = 'Sudamerica'
x_size = 3000
y_size = 1200
area_extent = (-5570248.4773392612, -4263473.5610361192, -384719.90821206354, 1339786.2707295895)
projection = '+proj=geos +a=6378137. +b=6356752.31414 +lon_0=-75 +f=.00335281068119356027 +h=35786023 +sweep=x'
description = "Sud geos"
proj_id = 'geos0'
areadef = get_area_def(area_id, description, proj_id, projection,x_size, y_size, area_extent)

scn = Scene(sensor='abi', filenames=['OR_ABI-L1b-RadF-M3C01_G16_s20182211830459_e20182211841226_c20182211841271.nc','OR_ABI-L1b-RadF-M3C02_G16_s20182211830459_e20182211841226_c20182211841260.nc','OR_ABI-L1b-RadF-M3C03_G16_s20182211830459_e20182211841226_c20182211841272.nc'])
rgbname = 'true_color'
local_scene = scn.resample(areadef)
local_scene.show(rgbname)

Problem description

I'm trying to generate a corrected true color images from GOES-16 netcdf files available en AWS. I tried the example gave for Himawari-8 (https://github.com/pytroll/pytroll-examples/blob/master/satpy/ahi_true_color_pyspectral.ipynb). First, I couldn't use find_files_and_readers, so I define the filenames by myself. No problem on that.

Second, I couldn't process full disk image because of memory lack, I'm running the code in a 16GB RAM and 12 core server. Then I tried to process just South America. I used area definition from this web https://github.com/pytroll/satpy/blob/master/satpy/etc/areas.def

I got the following error:

Traceback (most recent call last): File "true_pytroll.py", line 28, in local_scene.show(rgbname) File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 986, in show img = get_enhanced_image(self[dataset_id].squeeze(), overlay=overlay) File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 573, in getitem return self.datasets[key] File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/readers/init.py", line 299, in getitem key = self.get_key(item) File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/readers/init.py", line 288, in get_key best=best, **dfilter) File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/readers/init.py", line 239, in get_key raise KeyError("No dataset matching '{}' found".format(str(key))) KeyError: "No dataset matching 'DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)' found"

Thank you very much.

djhoese commented 6 years ago

Thanks for the bug report. This seems like it is a satpy error and in the future this is probably better on the satpy repository. Anyway...

Could you try running at the top of your code from satpy.utils import debug_on; debug_on() and running your code again and pasting the output here. The error message you got (which I thought I had cleaned up) is saying that the true_color composite wasn't created. The debug messages should show you why.

It's possible satpy is looking for more than just C01, C02, and C03 data, but I can't think of a reason why.

adioses commented 6 years ago
[DEBUG: 2018-08-10 11:43:08 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc'
[DEBUG: 2018-08-10 11:43:08 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_eps_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_eps_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_eps_l1b.yaml']
[WARNING: 2018-08-10 11:43:08 : satpy.readers.yaml_reader] No filenames found for reader: avhrr_eps_l1b
[DEBUG: 2018-08-10 11:43:08 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml']
[INFO: 2018-08-10 11:43:08 : satpy.readers] Cannot use ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml']
[DEBUG: 2018-08-10 11:43:08 : satpy.readers] while constructing a Python object
cannot find module 'satpy.readers.grib' (No module named pygrib)
  in "<string>", line 9, column 18:
        file_reader: !!python/name:satpy.readers.grib ... 
                     ^
[DEBUG: 2018-08-10 11:43:08 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdfeos_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdfeos_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdfeos_l1b.yaml']
[INFO: 2018-08-10 11:43:08 : satpy.readers] Cannot use ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdfeos_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdfeos_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdfeos_l1b.yaml']
[DEBUG: 2018-08-10 11:43:08 : satpy.readers] while constructing a Python object
cannot find module 'satpy.readers.hdfeos_l1b' (No module named pyhdf.error)
  in "<string>", line 438, column 18:
        file_reader: !!python/name:satpy.readers.hdfe ... 
                     ^
[DEBUG: 2018-08-10 11:43:08 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l1b.yaml']
[WARNING: 2018-08-10 11:43:09 : satpy.readers.yaml_reader] No filenames found for reader: nc_olci_l1b
[DEBUG: 2018-08-10 11:43:09 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_xrit.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_xrit.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_xrit.yaml']
[INFO: 2018-08-10 11:43:09 : satpy.readers] Cannot use ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_xrit.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_xrit.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_xrit.yaml']
[DEBUG: 2018-08-10 11:43:09 : satpy.readers] while constructing a Python object
cannot find module 'satpy.readers.mipp_xrit' (cannot import name area_defs_to_extent)
  in "<string>", line 6, column 11:
      reader: !!python/name:satpy.readers.mipp ... 
              ^
[DEBUG: 2018-08-10 11:43:09 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l2.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l2.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l2.yaml']
[WARNING: 2018-08-10 11:43:09 : satpy.readers.yaml_reader] No filenames found for reader: nc_olci_l2
[DEBUG: 2018-08-10 11:43:09 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_sdr.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_sdr.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_sdr.yaml']
[WARNING: 2018-08-10 11:43:10 : satpy.readers.yaml_reader] No filenames found for reader: viirs_sdr
[DEBUG: 2018-08-10 11:43:10 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ghrsst_osisaf.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ghrsst_osisaf.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ghrsst_osisaf.yaml']
[WARNING: 2018-08-10 11:43:11 : satpy.readers.yaml_reader] No filenames found for reader: ghrsst_osisaf
[DEBUG: 2018-08-10 11:43:11 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_electrol.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_electrol.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_electrol.yaml']
[WARNING: 2018-08-10 11:43:11 : satpy.readers.yaml_reader] No filenames found for reader: hrit_electrol
[DEBUG: 2018-08-10 11:43:11 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_h8.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_h8.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_h8.yaml']
[INFO: 2018-08-10 11:43:11 : satpy.readers] Cannot use ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_h8.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_h8.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_h8.yaml']
[DEBUG: 2018-08-10 11:43:11 : satpy.readers] while constructing a Python object
cannot find module 'satpy.readers.mipp_xrit' (cannot import name area_defs_to_extent)
  in "<string>", line 7, column 11:
      reader: !!python/name:satpy.readers.mipp ... 
              ^
[DEBUG: 2018-08-10 11:43:11 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/aapp_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/aapp_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/aapp_l1b.yaml']
[WARNING: 2018-08-10 11:43:11 : satpy.readers.yaml_reader] No filenames found for reader: avhrr_aapp_l1b
[DEBUG: 2018-08-10 11:43:11 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/olci.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/olci.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/olci.yaml']
[WARNING: 2018-08-10 11:43:11 : satpy.readers.yaml_reader] No filenames found for reader: nc_olci
[DEBUG: 2018-08-10 11:43:11 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_l1b.yaml']
[WARNING: 2018-08-10 11:43:12 : satpy.readers.yaml_reader] No filenames found for reader: viirs_l1b
[DEBUG: 2018-08-10 11:43:12 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml']
[DEBUG: 2018-08-10 11:43:12 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['OR_ABI-L1b-RadF-M3C01_G16_s20182211830459_e20182211841226_c20182211841271.nc', 'OR_ABI-L1b-RadF-M3C02_G16_s20182211830459_e20182211841226_c20182211841260.nc', 'OR_ABI-L1b-RadF-M3C03_G16_s20182211830459_e20182211841226_c20182211841272.nc']
[DEBUG: 2018-08-10 11:43:13 : satpy.composites] Looking for composites config file abi.yaml
[DEBUG: 2018-08-10 11:43:13 : satpy.composites] Looking for composites config file visir.yaml
[DEBUG: 2018-08-10 11:43:13 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc'
Traceback (most recent call last):
  File "true_pytroll.py", line 29, in <module>
    local_scene.show(rgbname)
  File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 986, in show
    img = get_enhanced_image(self[dataset_id].squeeze(), overlay=overlay)
  File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 573, in __getitem__
    return self.datasets[key]
  File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/readers/__init__.py", line 299, in __getitem__
    key = self.get_key(item)
  File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/readers/__init__.py", line 288, in get_key
    best=best, **dfilter)
  File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/readers/__init__.py", line 239, in get_key
    raise KeyError("No dataset matching '{}' found".format(str(key)))
KeyError: "No dataset matching 'DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)' found"
djhoese commented 6 years ago

Ah I should have realized this earlier. You need to actually load the data scn.load(['true_color']).

Edit: Note this should happen right after the Scene creation and before resampling.

adioses commented 6 years ago

I got the same error Memory. It's weird because I process full disk image in the same server using GDAL.

[DEBUG: 2018-08-10 12:06:16 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc' [DEBUG: 2018-08-10 12:06:16 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_compact.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_compact.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_compact.yaml'] [WARNING: 2018-08-10 12:06:19 : satpy.readers.yaml_reader] No filenames found for reader: viirs_compact [DEBUG: 2018-08-10 12:06:19 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nucaps.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nucaps.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nucaps.yaml'] [WARNING: 2018-08-10 12:06:19 : satpy.readers.yaml_reader] No filenames found for reader: nucaps [DEBUG: 2018-08-10 12:06:19 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/clavrx.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/clavrx.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/clavrx.yaml'] [INFO: 2018-08-10 12:06:19 : satpy.readers] Cannot use ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/clavrx.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/clavrx.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/clavrx.yaml'] [DEBUG: 2018-08-10 12:06:19 : satpy.readers] while constructing a Python object cannot find module 'satpy.readers.clavrx' (No module named pyhdf.SD) in "", line 9, column 18: file_reader: !!python/name:satpy.readers.clav ... ^ [DEBUG: 2018-08-10 12:06:19 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/fci_fdhsi.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/fci_fdhsi.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/fci_fdhsi.yaml'] [WARNING: 2018-08-10 12:06:19 : satpy.readers.yaml_reader] No filenames found for reader: fci_fdhsi [DEBUG: 2018-08-10 12:06:19 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml'] [INFO: 2018-08-10 12:06:19 : satpy.readers] Cannot use ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml'] [DEBUG: 2018-08-10 12:06:19 : satpy.readers] while constructing a Python object cannot find module 'satpy.readers.grib' (No module named pygrib) in "", line 9, column 18: file_reader: !!python/name:satpy.readers.grib ... ^ [DEBUG: 2018-08-10 12:06:19 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_hrpt_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_hrpt_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_hrpt_l1b.yaml'] [INFO: 2018-08-10 12:06:19 : satpy.readers] Cannot use ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_hrpt_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_hrpt_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_hrpt_l1b.yaml'] [DEBUG: 2018-08-10 12:06:19 : satpy.readers] while constructing a Python object cannot find module 'satpy.readers.hrpt' (No module named pygac.gac_calibration) in "", line 93, column 22: file_reader: !!python/name:satpy.readers.hrpt ... ^ [DEBUG: 2018-08-10 12:06:19 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/omps_edr.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/omps_edr.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/omps_edr.yaml'] [WARNING: 2018-08-10 12:06:20 : satpy.readers.yaml_reader] No filenames found for reader: omps_edr [DEBUG: 2018-08-10 12:06:20 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_slstr.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_slstr.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_slstr.yaml'] [WARNING: 2018-08-10 12:06:20 : satpy.readers.yaml_reader] No filenames found for reader: nc_slstr [DEBUG: 2018-08-10 12:06:20 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_sar_c.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_sar_c.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_sar_c.yaml'] [WARNING: 2018-08-10 12:06:20 : satpy.readers.yaml_reader] No filenames found for reader: safe_sar_c [DEBUG: 2018-08-10 12:06:20 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_jma.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_jma.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_jma.yaml'] [WARNING: 2018-08-10 12:06:21 : satpy.readers.yaml_reader] No filenames found for reader: hrit_jma [DEBUG: 2018-08-10 12:06:21 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ghrsst_osisaf.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ghrsst_osisaf.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ghrsst_osisaf.yaml'] [WARNING: 2018-08-10 12:06:21 : satpy.readers.yaml_reader] No filenames found for reader: ghrsst_osisaf [DEBUG: 2018-08-10 12:06:21 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/scatsat1_l2b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/scatsat1_l2b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/scatsat1_l2b.yaml'] [WARNING: 2018-08-10 12:06:21 : satpy.readers.yaml_reader] No filenames found for reader: scatsat1_l2b [DEBUG: 2018-08-10 12:06:21 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/acspo.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/acspo.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/acspo.yaml'] [WARNING: 2018-08-10 12:06:21 : satpy.readers.yaml_reader] No filenames found for reader: acspo [DEBUG: 2018-08-10 12:06:21 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_nwcsaf_msg.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_nwcsaf_msg.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_nwcsaf_msg.yaml'] [WARNING: 2018-08-10 12:06:21 : satpy.readers.yaml_reader] No filenames found for reader: nc_nwcsaf_msg [DEBUG: 2018-08-10 12:06:21 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ahi_hsd.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ahi_hsd.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ahi_hsd.yaml'] [WARNING: 2018-08-10 12:06:22 : satpy.readers.yaml_reader] No filenames found for reader: ahi_hsd [DEBUG: 2018-08-10 12:06:22 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_nwcsaf_pps.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_nwcsaf_pps.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_nwcsaf_pps.yaml'] [WARNING: 2018-08-10 12:06:22 : satpy.readers.yaml_reader] No filenames found for reader: nc_nwcsaf_pps [DEBUG: 2018-08-10 12:06:22 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/generic_image.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/generic_image.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/generic_image.yaml'] [WARNING: 2018-08-10 12:06:22 : satpy.readers.yaml_reader] No filenames found for reader: generic_image [DEBUG: 2018-08-10 12:06:22 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/geocat.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/geocat.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/geocat.yaml'] [WARNING: 2018-08-10 12:06:22 : satpy.readers.yaml_reader] No filenames found for reader: geocat [DEBUG: 2018-08-10 12:06:22 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l1b.yaml'] [WARNING: 2018-08-10 12:06:23 : satpy.readers.yaml_reader] No filenames found for reader: nc_olci_l1b [DEBUG: 2018-08-10 12:06:23 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_eps_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_eps_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_eps_l1b.yaml'] [WARNING: 2018-08-10 12:06:23 : satpy.readers.yaml_reader] No filenames found for reader: avhrr_eps_l1b [DEBUG: 2018-08-10 12:06:23 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdf4_caliopv3.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdf4_caliopv3.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdf4_caliopv3.yaml'] [INFO: 2018-08-10 12:06:23 : satpy.readers] Cannot use ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdf4_caliopv3.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdf4_caliopv3.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdf4_caliopv3.yaml'] [DEBUG: 2018-08-10 12:06:23 : satpy.readers] while constructing a Python object cannot find module 'satpy.readers.hdf4_caliopv3' (No module named pyhdf.SD) in "", line 40, column 18: file_reader: !!python/name:satpy.readers.hdf4 ... ^ [DEBUG: 2018-08-10 12:06:23 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/maia.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/maia.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/maia.yaml'] [WARNING: 2018-08-10 12:06:23 : satpy.readers.yaml_reader] No filenames found for reader: maia [DEBUG: 2018-08-10 12:06:23 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/iasi_l2.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/iasi_l2.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/iasi_l2.yaml'] [WARNING: 2018-08-10 12:06:23 : satpy.readers.yaml_reader] No filenames found for reader: iasi_l2 [DEBUG: 2018-08-10 12:06:23 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/gac_lac_l1.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/gac_lac_l1.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/gac_lac_l1.yaml'] [INFO: 2018-08-10 12:06:23 : satpy.readers] Cannot use ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/gac_lac_l1.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/gac_lac_l1.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/gac_lac_l1.yaml'] [DEBUG: 2018-08-10 12:06:23 : satpy.readers] while constructing a Python object cannot find module 'satpy.readers.gac_lac_l1' (No module named pygac.gac_calibration) in "", line 138, column 22: filereader: !!python/name:satpy.readers.gac ... ^ [DEBUG: 2018-08-10 12:06:23 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/amsr2_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/amsr2_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/amsr2_l1b.yaml'] [WARNING: 2018-08-10 12:06:24 : satpy.readers.yaml_reader] No filenames found for reader: amsr2_l1b [DEBUG: 2018-08-10 12:06:24 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_l1b.yaml'] [WARNING: 2018-08-10 12:06:24 : satpy.readers.yaml_reader] No filenames found for reader: viirs_l1b [DEBUG: 2018-08-10 12:06:24 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/native_msg.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/native_msg.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/native_msg.yaml'] [WARNING: 2018-08-10 12:06:25 : satpy.readers.yaml_reader] No filenames found for reader: native_msg [DEBUG: 2018-08-10 12:06:25 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_goes.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_goes.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_goes.yaml'] [WARNING: 2018-08-10 12:06:25 : satpy.readers.yaml_reader] No filenames found for reader: hrit_goes [DEBUG: 2018-08-10 12:06:25 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_aapp_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_aapp_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_aapp_l1b.yaml'] [WARNING: 2018-08-10 12:06:25 : satpy.readers.yaml_reader] No filenames found for reader: avhrr_aapp_l1b [DEBUG: 2018-08-10 12:06:25 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l2.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l2.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l2.yaml'] [WARNING: 2018-08-10 12:06:25 : satpy.readers.yaml_reader] No filenames found for reader: nc_olci_l2 [DEBUG: 2018-08-10 12:06:25 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_msi.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_msi.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_msi.yaml'] [INFO: 2018-08-10 12:06:25 : satpy.readers] Cannot use ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_msi.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_msi.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_msi.yaml'] [DEBUG: 2018-08-10 12:06:25 : satpy.readers] while constructing a Python object cannot find module 'satpy.readers.safe_msi' (No module named glymur) in "", line 10, column 22: file_reader: !!python/name:satpy.readers.safe ... ^ [DEBUG: 2018-08-10 12:06:25 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_msg.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_msg.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_msg.yaml'] [WARNING: 2018-08-10 12:06:26 : satpy.readers.yaml_reader] No filenames found for reader: hrit_msg [DEBUG: 2018-08-10 12:06:26 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2018-08-10 12:06:26 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['OR_ABI-L1b-RadF-M3C01_G16_s20182211830459_e20182211841226_c20182211841271.nc', 'OR_ABI-L1b-RadF-M3C02_G16_s20182211830459_e20182211841226_c20182211841260.nc', 'OR_ABI-L1b-RadF-M3C03_G16_s20182211830459_e20182211841226_c20182211841272.nc'] [DEBUG: 2018-08-10 12:06:27 : satpy.composites] Looking for composites config file abi.yaml [DEBUG: 2018-08-10 12:06:27 : satpy.composites] Looking for composites config file visir.yaml [DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional satellite_zenith_angle: Unknown dataset satellite_zenith_angle [DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional satellite_azimuth_angle: Unknown dataset satellite_azimuth_angle [DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional satellite_zenith_angle: Unknown dataset satellite_zenith_angle [DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional solar_azimuth_angle: Unknown dataset solar_azimuth_angle [DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional solar_zenith_angle: Unknown dataset solar_zenith_angle [DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional satellite_zenith_angle: Unknown dataset satellite_zenith_angle [DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional satellite_azimuth_angle: Unknown dataset satellite_azimuth_angle [DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional satellite_zenith_angle: Unknown dataset satellite_zenith_angle [DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional solar_azimuth_angle: Unknown dataset solar_azimuth_angle [DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional solar_zenith_angle: Unknown dataset solar_zenith_angle [DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional satellite_zenith_angle: Unknown dataset satellite_zenith_angle [DEBUG: 2018-08-10 12:06:27 : satpy.readers.abi_l1b] Reading in get_dataset C03. [DEBUG: 2018-08-10 12:06:27 : satpy.readers.abi_l1b] Calibrating to reflectances [DEBUG: 2018-08-10 12:06:27 : satpy.readers.abi_l1b] Reading in get_dataset C01. [DEBUG: 2018-08-10 12:06:27 : satpy.readers.abi_l1b] Calibrating to reflectances [DEBUG: 2018-08-10 12:06:27 : satpy.readers.abi_l1b] Reading in get_dataset C02. [DEBUG: 2018-08-10 12:06:27 : satpy.readers.abi_l1b] Calibrating to reflectances [WARNING: 2018-08-10 12:06:27 : satpy.readers.yaml_reader] Can't load ancillary dataset DQF [WARNING: 2018-08-10 12:06:27 : satpy.readers.yaml_reader] Can't load ancillary dataset DQF [WARNING: 2018-08-10 12:06:27 : satpy.readers.yaml_reader] Can't load ancillary dataset DQF [DEBUG: 2018-08-10 12:06:27 : satpy.composites] Applying sun zen correction [DEBUG: 2018-08-10 12:06:27 : satpy.composites] Computing sun zenith angles. Traceback (most recent call last): File "true_pytroll.py", line 27, in scn.load([rgbname]) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 850, in load keepables = self.generate_composites() File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 765, in generate_composites return self._read_composites(nodes) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 739, in _read_composites self._generate_composite(item, keepables) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 712, in _generate_composite *self.attrs) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/composites/init.py", line 341, in call lons, lats), File "/home/satelite/anaconda2/lib/python2.7/site-packages/pyorbital-1.3.1-py2.7.egg/pyorbital/astronomy.py", line 144, in cos_zen return (np.sin(lat) np.sin(dec) + np.cos(lat) np.cos(dec) np.cos(h__)) MemoryError

djhoese commented 6 years ago

A couple things that may be a problem:

  1. Instead of specifying sensor when you are creating your Scene, use reader='abi_l1b'. This will skip searching for a reader and will speed up the initial loading time of your script.
  2. What version of satpy are you using (satpy.__version__)? If you aren't on 0.9 then you will probably not be able to create a full disk true color image. Version 0.9 was satpy's first version using dask and xarray which allowed us major improvements in processing time and memory usage. We provide satpy 0.9 in the conda-forge channel for conda if you aren't already using it, but pip should be fine too.
  3. Do you need to be on python 2? This isn't really an issue for performance, but if this is a new project you are starting I highly recommend moving to python 3.

Lastly, satpy isn't simply putting the bands together. It is configured to perform multiple corrections to the data to produce the best true color image we know how to make. This includes a solar-zenith correction and a rayleigh scattering correction. The rayleigh correction requires a lot of time and memory compared to not using it. This is why gdal might not have memory issues (among other things) when creating full disk imagery. You could try loading true_color_raw which won't do the rayleigh correction.

I'm curious, what full disk imagery were you making with GDAL? Was it an ABI true color? How did you make the green band?

adioses commented 6 years ago

Thank, I'll make the changes you just suggest. I use Satpy 0.9, I'll change to python 3 and sensor to reader.

I completely understand that, this are not simply correction and take a lot of resources. I would read more about this correction. I'm trying to create corrected true color images operational, so I need to speed up the process.

I process ABI true color. In order to produce Green Band I use a equation I found in this web (http://edc.occ-data.org/goes16/python/) GreenB = 0.48358168 ch2 + 0.45706946 ch1 + 0.06038137 * ch3. Then I make some brightness adjustment, There is not physical sense on that is just to improve visualization.

adioses commented 6 years ago

I ran the script with CONUS images and the output is great. Thank you so much.

Also, I found an error: Platform file /home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/platforms.txt not found. how could I correct this?

djhoese commented 6 years ago

Don't worry about that message. It is more of a warning than an error and is actually expected in satpy right now. @adybbroe is there a solution for this platforms.txt being missing? Can't we stop the message from appearing in satpy?

djhoese commented 6 years ago

Also FYI @adioses I run satpy on my laptop (macbook with 16GB memory) and can produce an ABI true color full disk image in ~8 minutes (maybe less, can't remember). Another thing I just noticed is that you said you have 12 cores. This may be the reason you are running out of memory since by default dask (the lower level array library we're using) will create a worker thread for each logical core on your system. You may want to configure it to have less workers, something like 8, following the directions using ThreadPool here: http://dask.pydata.org/en/latest/scheduler-overview.html#configuring-the-schedulers

adioses commented 6 years ago

Thanks. I’ll try it. When I tried to run South America took like an hour and got out off memory.

Also I’ll try to reduce my processing area. I want to focus on South America and not in CONUS sector

djhoese commented 6 years ago

Hhhhmmm it should never take an hour with satpy 0.9+. Make sure you have semi-recent versions of xarray, dask, pyresample, and trollimage.

adioses commented 6 years ago

I updated all the libraries you mentioned. But I still have problems with FullDisk Images. I couldn't configure dask. It's not really clear for me I'll keep trying. Otherwise, I would like to know if there is a way to read just part of the file so it process less information or I should modify directly the NetCDF file?

djhoese commented 6 years ago

Configuring the number of workers is the easiest way to limit the amount of memory used since you will only have so many workers working at a single moment. As for configuring dask, right after the imports in your script do:

import dask
from multiprocessing.pool import ThreadPool
with dask.config.set(pool=ThreadPool(4)):
    # all of your other satpy-based code goes in this block of code including the .show call

SatPy is already attempting to minimize the amount of data being processed when you resampled the data. There is another method but it currently has a bug so I'm not going to suggest it right now.

Lastly, you can configure the "chunks" that dask creates from the data arrays. Each chunk is processed separately on an individual dask worker when possible. You can configure this right now by specifying an environment variable although we hope to make it easier in the future. You can set PYTROLL_CHUNK_SIZE to a value between 1024 and 4096 (default) on the command line before running your script. For bash that would be:

export PYTROLL_CHUNK_SIZE=2048
python your_script.py

Lastly...again, you could also provide cache_dir='/path/where/to/put/cache/files/' to your resample call. This won't fix your memory errors but it will make processing of the same geostationary area much faster on future executions.

adioses commented 6 years ago

My code now is this:

from satpy.utils import debug_on; debug_on()
from satpy import Scene, find_files_and_readers
from satpy.resample import get_area_def
from pyresample.utils import get_area_def
import dask
from multiprocessing.pool import ThreadPool
with dask.config.set(pool=ThreadPool(4)):

    scn = Scene(reader='abi_l1b', filenames=['OR_ABI-L1b-RadC-M3C01_G16_s20182211832268_e20182211835041_c20182211835090.nc','OR_ABI-L1b-RadC-M3C02_G16_s20182211832268_e20182211835041_c20182211835086.nc','OR_ABI-L1b-RadC-M3C03_G16_s20182211832268_e20182211835041_c20182211835089.nc'])
    rgbname = 'true_color'
    scn.load([rgbname])
    new_scn = scn.resample(scn.min_area(), resampler='native')
    new_scn.save_dataset(rgbname, filename='GOES_{datetime}.png'.format(datetime=scn.start_time.strftime('%Y%m%d%H%M')))

Now I have this error: [DEBUG: 2018-08-10 16:15:59 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc' [DEBUG: 2018-08-10 16:15:59 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml'] [DEBUG: 2018-08-10 16:15:59 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['OR_ABI-L1b-RadC-M3C01_G16_s20182211832268_e20182211835041_c20182211835090.nc', 'OR_ABI-L1b-RadC-M3C02_G16_s20182211832268_e20182211835041_c20182211835086.nc', 'OR_ABI-L1b-RadC-M3C03_G16_s20182211832268_e20182211835041_c20182211835089.nc'] Traceback (most recent call last): File "true_pytroll.py", line 9, in scn = Scene(reader='abi_l1b', filenames=['OR_ABI-L1b-RadC-M3C01_G16_s20182211832268_e20182211835041_c20182211835090.nc','OR_ABI-L1b-RadC-M3C02_G16_s20182211832268_e20182211835041_c20182211835086.nc','OR_ABI-L1b-RadC-M3C03_G16_s20182211832268_e20182211835041_c20182211835089.nc']) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 151, in init reader_kwargs=reader_kwargs) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 196, in create_reader_instances ppp_config_dir=self.ppp_config_dir) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/readers/init.py", line 599, in load_readers reader_instance.create_filehandlers(loadables) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/readers/yaml_reader.py", line 493, in create_filehandlers filename_set) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/readers/yaml_reader.py", line 480, in new_filehandlers_for_filetype return list(filtered_iter) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/readers/yaml_reader.py", line 453, in filter_fh_by_metadata for filehandler in filehandlers: File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/readers/yaml_reader.py", line 396, in new_filehandler_instances yield filetype_cls(filename, filename_info, filetype_info, *req_fh) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/readers/abi_l1b.py", line 57, in init chunks={'x': CHUNK_SIZE, 'y': CHUNK_SIZE}) File "/home/satelite/anaconda2/lib/python2.7/site-packages/xarray/backends/api.py", line 320, in open_dataset backendkwargs) File "/home/satelite/anaconda2/lib/python2.7/site-packages/xarray/backends/netCDF4.py", line 332, in open ds = opener() File "/home/satelite/anaconda2/lib/python2.7/site-packages/xarray/backends/netCDF4_.py", line 231, in _open_netcdf4_group ds = nc4.Dataset(filename, mode=mode, kwargs) File "netCDF4/_netCDF4.pyx", line 2015, in netCDF4._netCDF4.Dataset.init File "netCDF4/_netCDF4.pyx", line 1636, in netCDF4._netCDF4._ensure_nc_success IOError: [Errno 2] No such file or directory: '/home/satelite/Documents/GOES16/AB1_L1B/OR_ABI-L1b-RadC-M3C01_G16_s20182211832268_e20182211835041_c20182211835090.nc' Exception AttributeError: AttributeError("'NC_ABI_L1B' object has no attribute 'nc'",) in <bound method NC_ABI_L1B.del of <NC_ABI_L1B: 'OR_ABI-L1b-RadC-M3C01_G16_s20182211832268_e20182211835041_c20182211835090.nc'>> ignored

The file and the code are in the same folder and the files exist

djhoese commented 6 years ago

What does ls /home/satelite/Documents/GOES16/AB1_L1B/OR_ABI-L1b-RadC-M3C01_G16_s20182211832268_e20182211835041_c20182211835090.nc give you?

What about ls /home/satelite/Documents/GOES16/AB1_L1B/true_pytroll.py?

Edit: And are you running this from '/home/satelite/Documents/GOES16/AB1_L1B/?

adioses commented 6 years ago

Sorry. Omit the error, I changed the files that caused the error. One question, in this line

with dask.config.set(pool=ThreadPool(4)):

4 is the number of cores to use?

Also, If I check the process with top command the script don't use more than one core.

djhoese commented 6 years ago

Correct, my laptop creates 8 workers. Technically this is number of threads not cores where to me "cores" has the assumption that we're talking about multiple processes. I think you have to specify extra flags in top to get it to show you threads for a process, but that may depend on the operating system/version of top.

adioses commented 6 years ago

Same error

[DEBUG: 2018-08-10 17:03:48 : satpy.composites] Computing sun zenith angles. Traceback (most recent call last): File "true_pytroll.py", line 11, in scn.load([rgbname]) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 850, in load keepables = self.generate_composites() File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 765, in generate_composites return self._read_composites(nodes) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 739, in _read_composites self._generate_composite(item, keepables) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 712, in _generate_composite *self.attrs) File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/composites/init.py", line 341, in call lons, lats), File "/home/satelite/anaconda2/lib/python2.7/site-packages/pyorbital-1.3.1-py2.7.egg/pyorbital/astronomy.py", line 144, in cos_zen return (np.sin(lat) np.sin(dec) + np.cos(lat) np.cos(dec) np.cos(h__)) MemoryError

I tried using ThreadPool(4) and ThreadPool(8). Also I define export PYTROLL_CHUNK_SIZE=2048

adybbroe commented 6 years ago

@djhoese Can we conclude that this is rather a Satpy issue than a Pyspectral one?

djhoese commented 6 years ago

Sorry @adioses, i lost track of this issue. Adam yes it is a satpy or pyorbital issue but im not sure how much more I can offer as help. The newest versions of every library don't produce memory issues for most people.

adybbroe commented 6 years ago

Okay, I close it here then. Hope the memory error is no more an issue for you @adioses

djhoese commented 6 years ago

@adioses Feel free to comment here or create a new issue on the satpy repository if you still have issues.

adioses commented 6 years ago

Hi @djhoese, I just realice that Python version is really important. The performance of the process is incredibly better when I changed to python 3. Since I use python3 the performance allow me to process full disk in 4 minutes and do not take more than 8GB ram. Thanks for all your help