search

pytroll / satpy

Python package for earth-observing satellite data processing
http://satpy.readthedocs.org/en/latest/
GNU General Public License v3.0
1.06k stars 292 forks source link

Can't load angle data from msi_safe in version 0.23 #1389

Closed yukaribbba closed 4 years ago

yukaribbba commented 4 years ago

Describe the bug When reading Sentinel-2 MSI L1C dataset, I got error messages saying it can't get any angle data, i.e. solar zenith angle. Atmospheric correction is unable to go through without them. Everything goes fine in 0.22.

To Reproduce Download dataset from SciHub

!/usr/bin/env python

from satpy import Scene, find_files_and_readers files = find_files_and_readers(base_dir="/mnt/hgfs/Downloads/Sentinel-2/", reader='msi_safe') scn = Scene(filenames=files) composite = 'true_color_marine_clean' scn.load([composite]) scn.save_dataset(composite, base_dir="/mnt/hgfs/Downloads/Sentinel-2/", writer='geotiff', compress=None)

Expected behavior Angle data in MSI L1C should be loaded normally.

Actual results /home/ll/anaconda3/lib/python3.8/site-packages/satpy/resample.py:150: UserWarning: Gradient search resampler not available, upgrade Pyresample. warnings.warn('Gradient search resampler not available, upgrade Pyresample.') /home/ll/anaconda3/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() Could not load dataset 'DataID(name='solar_azimuth_angle', resolution=10, modifiers=())': "Could not load DataID(name='solar_azimuth_angle', resolution=10, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 828, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, kwargs) File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 711, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, kwargs) File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 696, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='solar_azimuth_angle', resolution=10, modifiers=()) from any provided files" /home/ll/anaconda3/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() Could not load dataset 'DataID(name='satellite_zenith_angle', resolution=10, modifiers=())': "Could not load DataID(name='satellite_zenith_angle', resolution=10, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 828, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, kwargs) File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 711, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, kwargs) File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 696, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='satellite_zenith_angle', resolution=10, modifiers=()) from any provided files" /home/ll/anaconda3/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() /home/ll/anaconda3/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() Could not load dataset 'DataID(name='satellite_azimuth_angle', resolution=10, modifiers=())': "Could not load DataID(name='satellite_azimuth_angle', resolution=10, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 828, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, kwargs) File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 711, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, kwargs) File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 696, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='satellite_azimuth_angle', resolution=10, modifiers=()) from any provided files" /home/ll/anaconda3/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() Could not load dataset 'DataID(name='solar_zenith_angle', resolution=10, modifiers=())': "Could not load DataID(name='solar_zenith_angle', resolution=10, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 828, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, kwargs) File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 711, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, kwargs) File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 696, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='solar_zenith_angle', resolution=10, modifiers=()) from any provided files" /home/ll/anaconda3/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() /home/ll/anaconda3/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems proj_string = self.to_proj4() Traceback (most recent call last): File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/utils.py", line 295, in get_satpos orb_params = dataset.attrs['orbital_parameters'] KeyError: 'orbital_parameters'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/ll/Downloads/s2t.py", line 7, in scn.load([composite]) File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/scene.py", line 984, in load keepables = self._generate_composites() File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/scene.py", line 897, in _generate_composites return self._read_composites(nodes) File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/scene.py", line 873, in _read_composites self._generate_composite(item, keepables) File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/scene.py", line 847, in _generate_composite composite = compositor(prereq_datasets, File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/composites/init.py", line 570, in call sata, satz, suna, sunz = self.get_angles(vis) File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/composites/init.py", line 552, in get_angles sat_lon, sat_lat, sat_alt = get_satpos(vis) File "/home/ll/anaconda3/lib/python3.8/site-packages/satpy/utils.py", line 325, in get_satpos lon = dataset.attrs['satellite_longitude'] KeyError: 'satellite_longitude'

Environment Info:

abi_l1b: ok abi_l1b_scmi: ok abi_l2_nc: ok acspo: ok agri_l1: ok ahi_hrit: ok ahi_hsd: ok ami_l1b: ok amsr2_l1b: ok amsr2_l2: ok avhrr_l1b_aapp: ok avhrr_l1b_eps: ok avhrr_l1b_gaclac: cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac') avhrr_l1b_hrpt: cannot find module 'satpy.readers.hrpt' (No module named 'pygac') caliop_l2_cloud: cannot find module 'satpy.readers.caliop_l2_cloud' (No module named 'pyhdf') clavrx: cannot find module 'satpy.readers.clavrx' (No module named 'pyhdf') cmsaf-claas2_l2_nc: ok electrol_hrit: ok fci_l1c_fdhsi: ok fci_l2_nc: ok generic_image: ok geocat: ok ghrsst_l3c_sst: ok glm_l2: ok goes-imager_hrit: ok goes-imager_nc: ok gpm_imerg: ok grib: cannot find module 'satpy.readers.grib' (No module named 'pygrib') hsaf_grib: cannot find module 'satpy.readers.hsaf_grib' (No module named 'pygrib') hy2_scat_l2b_h5: ok iasi_l2: ok iasi_l2_so2_bufr: cannot find module 'satpy.readers.iasi_l2_so2_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n Error: ', ModuleNotFoundError("No module named 'eccodes'"))) jami_hrit: ok li_l2: cannot find module 'satpy.readers.li_l2' (No module named 'h5netcdf') maia: ok mersi2_l1b: ok mimicTPW2_comp: ok modis_l1b: cannot find module 'satpy.readers.modis_l1b' (No module named 'pyhdf') modis_l2: cannot find module 'satpy.readers.modis_l2' (No module named 'pyhdf') msi_safe: ok mtsat2-imager_hrit: ok nucaps: ok nwcsaf-geo: ok nwcsaf-msg2013-hdf5: ok nwcsaf-pps_nc: ok olci_l1b: ok olci_l2: ok omps_edr: ok safe_sar_l2_ocn: ok sar-c_safe: ok scatsat1_l2b: ok seviri_l1b_hrit: ok seviri_l1b_icare: cannot find module 'satpy.readers.seviri_l1b_icare' (No module named 'pyhdf') seviri_l1b_native: ok seviri_l1b_nc: ok seviri_l2_bufr: cannot find module 'satpy.readers.seviri_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes) seviri_l2_grib: cannot find module 'satpy.readers.seviri_l2_grib' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes) slstr_l1b: ok slstr_l2: ok smos_l2_wind: ok tropomi_l2: ok vaisala_gld360: ok vii_l1b_nc: ok vii_l2_nc: ok viirs_compact: ok viirs_edr_active_fires: ok viirs_edr_flood: cannot find module 'satpy.readers.viirs_edr_flood' (No module named 'pyhdf') viirs_l1b: ok viirs_sdr: ok virr_l1b: ok

Writers

cf: ok geotiff: ok mitiff: ok ninjotiff: cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff') scmi: ok simple_image: ok

Extras

cartopy: No module named 'cartopy' geoviews: No module named 'geoviews'

djhoese commented 4 years ago

I think @mraspaud would have the best chance of figuring this out since he was the last one to modify the reader. I think we've seen similar issues in other readers, but this may be the simplest case.

yukaribbba commented 4 years ago

Thanks @djhoese . I just tried to replace the msi_safe.py single file with 0.22 version, but it still doesn’t work. So maybe there are other bugs? Not pretty sure about it.

djhoese commented 4 years ago

Yeah that won't work. There were a lot of changes between versions outside of just the reader file. They can't be swapped like that (at least for these versions).

simonrp84 commented 4 years ago

Hello @yukaribbba, this was a bug in how we load the angles. I noticed the same problem and have just created a pull request that fixes this. If you can, please try that version of the code and check that it also works for you.

yukaribbba commented 4 years ago

Thank you everyone, @djhoese @mraspaud @simonrp84 . I've tried the fixed reader and it works.

simonrp84 commented 4 years ago

Great!