raybellwaves commented 4 years ago

Describe the bug

Running scn.load(['B04']) gives AttributeError: 'AHIHSDFileHandler' object has no attribute 'area'

To Reproduce

Tested on master

$ conda create -n test_env python=3.8
$ conda activate test_env
$ conda install -c conda-forge satpy
$ pip install git+https://github.com/pytroll/satpy
$ pip install s3fs
$ python

>>> from satpy import Scene
>>> import s3fs
>>> import glob
>>> import os
>>> fs = s3fs.S3FileSystem(anon=True)
>>> os.mkdir('tmp')
>>> fs.get('noaa-himawari8/AHI-L1b-Japan/2020/09/06/1900/*', 'tmp')
>>> filenames = glob.glob('tmp/*')
>>> scn = Scene(reader='ahi_hsd', filenames=filenames)
>>> scn.load(['B04'])

Expected behavior It's work in the same way as if the data was FLDK (see https://gist.github.com/raybellwaves/4dd2f1472468e9f67424b6a148e9ac18)

Actual results

/Users/Ray/miniconda3/envs/main/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
  proj_string = self.to_proj4()
[DEBUG: 2020-09-06 23:15:11 : ahi_hsd] Band number = 4
[DEBUG: 2020-09-06 23:15:11 : ahi_hsd] Time_interval: 2020-09-06 19:00:13.958434 - 2020-09-06 19:01:50.270175
[DEBUG: 2020-09-06 23:15:11 : ahi_hsd] Reading time 0:00:00.027995
[DEBUG: 2020-09-06 23:15:11 : ahi_hsd] Calibration time 0:00:00.009550
[DEBUG: 2020-09-06 23:15:11 : ahi_hsd] Band number = 4
[DEBUG: 2020-09-06 23:15:11 : ahi_hsd] Time_interval: 2020-09-06 19:02:48.229387 - 2020-09-06 19:04:15.042410
[DEBUG: 2020-09-06 23:15:11 : ahi_hsd] Reading time 0:00:00.012308
[DEBUG: 2020-09-06 23:15:11 : ahi_hsd] Calibration time 0:00:00.009026

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-093980669367> in <module>
----> 1 scn.load(['B04'])

~/miniconda3/envs/main/lib/python3.8/site-packages/satpy/scene.py in load(self, wishlist, calibration, resolution, polarization, level, generate, unload, **kwargs)
    971             raise KeyError("Unknown datasets: {}".format(unknown_str))
    972 
--> 973         self.read(**kwargs)
    974         if generate:
    975             keepables = self.generate_composites()

~/miniconda3/envs/main/lib/python3.8/site-packages/satpy/scene.py in read(self, nodes, **kwargs)
    878             required_nodes = self.wishlist - set(self.datasets.keys())
    879             nodes = self.dep_tree.leaves(nodes=required_nodes)
--> 880         return self._read_datasets(nodes, **kwargs)
    881 
    882     def generate_composites(self, nodes=None):

~/miniconda3/envs/main/lib/python3.8/site-packages/satpy/scene.py in _read_datasets(self, dataset_nodes, **kwargs)
    719         for reader_name, ds_ids in reader_datasets.items():
    720             reader_instance = self.readers[reader_name]
--> 721             new_datasets = reader_instance.load(ds_ids, **kwargs)
    722             loaded_datasets.update(new_datasets)
    723         self.datasets.update(loaded_datasets)

~/miniconda3/envs/main/lib/python3.8/site-packages/satpy/readers/yaml_reader.py in load(self, dataset_keys, previous_datasets, **kwargs)
    870             coords = [all_datasets.get(cid, None)
    871                       for cid in coordinates.get(dsid, [])]
--> 872             ds = self._load_dataset_with_area(dsid, coords, **kwargs)
    873             if ds is not None:
    874                 all_datasets[dsid] = ds

~/miniconda3/envs/main/lib/python3.8/site-packages/satpy/readers/yaml_reader.py in _load_dataset_with_area(self, dsid, coords, **kwargs)
    780 
    781         try:
--> 782             ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
    783         except (KeyError, ValueError) as err:
    784             logger.exception("Could not load dataset '%s': %s", dsid, str(err))

~/miniconda3/envs/main/lib/python3.8/site-packages/satpy/readers/yaml_reader.py in _load_dataset_data(self, file_handlers, dsid, **kwargs)
    665     def _load_dataset_data(self, file_handlers, dsid, **kwargs):
    666         ds_info = self.all_ids[dsid]
--> 667         proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
    668         # FIXME: areas could be concatenated here
    669         # Update the metadata

~/miniconda3/envs/main/lib/python3.8/site-packages/satpy/readers/yaml_reader.py in _load_dataset(dsid, ds_info, file_handlers, dim, pad_data)
    931 
    932         counter, expected_segments, slice_list, failure, projectable = \
--> 933             _find_missing_segments(file_handlers, ds_info, dsid)
    934 
    935         if projectable is None or failure:

~/miniconda3/envs/main/lib/python3.8/site-packages/satpy/readers/yaml_reader.py in _find_missing_segments(file_handlers, ds_info, dsid)
   1061             counter += 1
   1062         try:
-> 1063             projectable = fh.get_dataset(dsid, ds_info)
   1064             if projectable is not None:
   1065                 slice_list.append(projectable)

~/miniconda3/envs/main/lib/python3.8/site-packages/satpy/readers/ahi_hsd.py in get_dataset(self, key, info)
    333 
    334     def get_dataset(self, key, info):
--> 335         return self.read_band(key, info)
    336 
    337     def get_area_def(self, dsid):

~/miniconda3/envs/main/lib/python3.8/site-packages/satpy/readers/ahi_hsd.py in read_band(self, key, info)
    547         # Mask space pixels
    548         if self.mask_space:
--> 549             res = self._mask_space(res)
    550 
    551         return res

~/miniconda3/envs/main/lib/python3.8/site-packages/satpy/readers/ahi_hsd.py in _mask_space(self, data)
    493     def _mask_space(self, data):
    494         """Mask space pixels"""
--> 495         return data.where(get_geostationary_mask(self.area))
    496 
    497     def read_band(self, key, info):

AttributeError: 'AHIHSDFileHandler' object has no attribute 'area'

Environment Info:

OS: OSX
Satpy Version: master
PyResample Version: 1.16.0

Readers and writers dependencies (when relevant):


Readers
=======
abi_l1b:  ok
abi_l1b_scmi:  ok
abi_l2_nc:  ok
acspo:  ok
agri_l1:  ok
ahi_hrit:  ok
ahi_hsd:  ok
ami_l1b:  ok
amsr2_l1b:  ok
amsr2_l2:  ok
avhrr_l1b_aapp:  ok
avhrr_l1b_eps:  ok
avhrr_l1b_gaclac:  cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac')
avhrr_l1b_hrpt:  cannot find module 'satpy.readers.hrpt' (No module named 'pygac')
caliop_l2_cloud:  cannot find module 'satpy.readers.caliop_l2_cloud' (No module named 'pyhdf')
clavrx:  cannot find module 'satpy.readers.clavrx' (No module named 'pyhdf')
cmsaf-claas2_l2_nc:  ok
electrol_hrit:  ok
fci_l1c_fdhsi:  ok
fci_l2_nc:  ok
generic_image:  ok
geocat:  ok
ghrsst_l3c_sst:  ok
glm_l2:  ok
goes-imager_hrit:  ok
goes-imager_nc:  ok
gpm_imerg:  ok
grib:  cannot find module 'satpy.readers.grib' (No module named 'pygrib')
hsaf_grib:  cannot find module 'satpy.readers.hsaf_grib' (No module named 'pygrib')
hy2_scat_l2b_h5:  ok
iasi_l2:  ok
iasi_l2_so2_bufr:  cannot find module 'satpy.readers.iasi_l2_so2_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n           Error: ', ModuleNotFoundError("No module named 'eccodes'")))
jami_hrit:  ok
li_l2:  cannot find module 'satpy.readers.li_l2' (No module named 'h5netcdf')
maia:  ok
mersi2_l1b:  ok
mimicTPW2_comp:  ok
modis_l1b:  cannot find module 'satpy.readers.modis_l1b' (No module named 'pyhdf')
modis_l2:  cannot find module 'satpy.readers.modis_l2' (No module named 'pyhdf')
msi_safe:  cannot find module 'satpy.readers.msi_safe' (No module named 'glymur')
mtsat2-imager_hrit:  ok
nucaps:  ok
nwcsaf-geo:  ok
nwcsaf-msg2013-hdf5:  ok
nwcsaf-pps_nc:  ok
olci_l1b:  ok
olci_l2:  ok
omps_edr:  ok
safe_sar_l2_ocn:  ok
sar-c_safe:  ok
scatsat1_l2b:  ok
seviri_l1b_hrit:  ok
seviri_l1b_icare:  cannot find module 'satpy.readers.seviri_l1b_icare' (No module named 'pyhdf')
seviri_l1b_native:  ok
seviri_l1b_nc:  ok
seviri_l2_bufr:  cannot find module 'satpy.readers.seviri_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
seviri_l2_grib:  cannot find module 'satpy.readers.seviri_l2_grib' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
slstr_l1b:  ok
slstr_l2:  ok
smos_l2_wind:  ok
tropomi_l2:  ok
vaisala_gld360:  ok
vii_l1b_nc:  ok
vii_l2_nc:  ok
viirs_compact:  ok
viirs_edr_active_fires:  ok
viirs_edr_flood:  cannot find module 'satpy.readers.viirs_edr_flood' (No module named 'pyhdf')
viirs_l1b:  ok
viirs_sdr:  ok
virr_l1b:  ok

Writers

cf: ok geotiff: ok mitiff: ok ninjotiff: cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff') scmi: ok simple_image: ok

Extras

cartopy: No module named 'cartopy' geoviews: No module named 'geoviews'

mraspaud commented 4 years ago

@raybellwaves Thanks for submitting this issue. I tried your example and could reproduce the error. However, after further inspection of the downloaded files, it seems that you have 4 different areas downloaded, namely JP01, JP02, JP03 and JP04. Just choosing the files for one area makes it work:

...
    curfiles = glob.glob('tmp/*JP02*')
    scn = Scene(curfiles, reader='ahi_hsd')
    dq = 'B14'
    scn.load([dq])
    scn.show(dq)

Note that I chose B14 because I find that visible channels aren't too interesting during night :)

The real issue then is that the printed error is totally unhelpful in diagnosing the error though, so this should be addressed.

raybellwaves commented 4 years ago

it seems that you have 4 different areas downloaded

@mraspaud thanks for catching that!

The real issue then is that the printed error is totally unhelpful in diagnosing the error though, so this should be addressed.

Agreed. Welcome to close this and create a new issue.

Plantain commented 3 years ago

Hi, This error is still present, but now also using the unaltered FLDK example from https://gist.github.com/raybellwaves/4dd2f1472468e9f67424b6a148e9ac18

Traceback (most recent call last): File "test.py", line 19, in scn.load(['B04']) File "/usr/local/lib/python3.6/dist-packages/satpy/scene.py", line 1129, in load self._read_datasets_from_storage(kwargs) File "/usr/local/lib/python3.6/dist-packages/satpy/scene.py", line 1149, in _read_datasets_from_storage return self._read_dataset_nodes_from_storage(nodes, kwargs) File "/usr/local/lib/python3.6/dist-packages/satpy/scene.py", line 1155, in _read_dataset_nodes_from_storage loaded_datasets = self._load_datasets_by_readers(reader_datasets, kwargs) File "/usr/local/lib/python3.6/dist-packages/satpy/scene.py", line 1180, in _load_datasets_by_readers new_datasets = reader_instance.load(ds_ids, kwargs) File "/usr/local/lib/python3.6/dist-packages/satpy/readers/yaml_reader.py", line 919, in load ds = self._load_dataset_with_area(dsid, coords, kwargs) File "/usr/local/lib/python3.6/dist-packages/satpy/readers/yaml_reader.py", line 1089, in _load_dataset_with_area ds = super(GEOFlippableFileYAMLReader, self)._load_dataset_with_area(dsid, coords, kwargs) File "/usr/local/lib/python3.6/dist-packages/satpy/readers/yaml_reader.py", line 829, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, kwargs) File "/usr/local/lib/python3.6/dist-packages/satpy/readers/yaml_reader.py", line 711, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, kwargs) File "/usr/local/lib/python3.6/dist-packages/satpy/readers/yaml_reader.py", line 1141, in _load_dataset _find_missing_segments(file_handlers, ds_info, dsid) File "/usr/local/lib/python3.6/dist-packages/satpy/readers/yaml_reader.py", line 1293, in _find_missing_segments projectable = fh.get_dataset(dsid, ds_info) File "/usr/local/lib/python3.6/dist-packages/satpy/readers/ahi_hsd.py", line 378, in get_dataset return self.read_band(key, info) File "/usr/local/lib/python3.6/dist-packages/satpy/readers/ahi_hsd.py", line 592, in read_band res = self._mask_space(res) File "/usr/local/lib/python3.6/dist-packages/satpy/readers/ahi_hsd.py", line 539, in _mask_space return data.where(get_geostationary_mask(self.area)) AttributeError: 'AHIHSDFileHandler' object has no attribute 'area'

djhoese commented 3 years ago

@Plantain We know that the error is not helpful for the overall problem, but did you have the same issue @raybellwaves had? He was providing more than one time step (and region) to the Scene object at once.

djhoese commented 3 years ago

And that gist should work fine if your data meets the assumption of Ray's comment:

# Download a folder (which is one time stamp) and contains all 16 channels and 10 segments

djhoese commented 2 years ago

I'm going to close this issue since there haven't been any updates. The general issue is that you can't provide more than one time step (repeat cycle) to the Scene. The error message is not extremely helpful, but I'm not sure there is an easy way around that. A pull request is welcome if someone wants to improve the error handling of the AHI HSD reader's error when accessing self.area, but I don't think it is necessary.

pytroll / satpy

'AHIHSDFileHandler' object has no attribute 'area' #1348

Writers

Extras