Sentinel-1 SAR reader fails to load some datasets

[mraspaud]

Describe the bug As reported by @oarcher in https://github.com/pytroll/pytroll-examples/issues/42 , loading some datasets in the Sentinel 1 reader fails.

To Reproduce

# Your code here
from satpy import available_readers, Scene, find_files_and_readers
import datetime

my_files = find_files_and_readers(base_dir='/windows_shared/tmp',
                                  reader='sar-c_safe',
                                  start_time=datetime.datetime(2018, 10, 13, 6, 0, 0),
                                  end_time=datetime.datetime(2018, 10, 13, 7, 30, 0))

scn = Scene(reader='sar-c_safe', filenames=my_files)

scn.load(scn.available_dataset_ids())

Expected behavior The datasets should be loaded

Actual results Text output of actual results or error messages including full tracebacks if applicable.

AttributeError: 'SAFEXML' object has no attribute 'read_xml_array'

```python Could not load dataset 'DataID(name='latitude', polarization='hh', resolution=80, modifiers=())': "Could not load DataID(name='latitude', polarization='hh', resolution=80, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 698, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='latitude', polarization='hh', resolution=80, modifiers=()) from any provided files" Could not load dataset 'DataID(name='latitude', polarization='hv', resolution=80, modifiers=())': "Could not load DataID(name='latitude', polarization='hv', resolution=80, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 698, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='latitude', polarization='hv', resolution=80, modifiers=()) from any provided files" Could not load dataset 'DataID(name='longitude', polarization='hh', resolution=80, modifiers=())': "Could not load DataID(name='longitude', polarization='hh', resolution=80, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 698, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='longitude', polarization='hh', resolution=80, modifiers=()) from any provided files" Could not load dataset 'DataID(name='longitude', polarization='hv', resolution=80, modifiers=())': "Could not load DataID(name='longitude', polarization='hv', resolution=80, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 698, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='longitude', polarization='hv', resolution=80, modifiers=()) from any provided files" Could not load dataset 'DataID(name='gamma_squared', polarization='hh', resolution=80, modifiers=())': "Could not load DataID(name='gamma_squared', polarization='hh', resolution=80, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 698, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='gamma_squared', polarization='hh', resolution=80, modifiers=()) from any provided files" Could not load dataset 'DataID(name='measurement', polarization='hv', resolution=80, calibration=, quantity=, modifiers=())': "Could not load DataID(name='measurement', polarization='hv', resolution=80, calibration=, quantity=, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 698, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='measurement', polarization='hv', resolution=80, calibration=, quantity=, modifiers=()) from any provided files" --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) in ----> 1 scn.load(scn.available_dataset_ids()) ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/scene.py in load(self, wishlist, calibration, resolution, polarization, level, generate, unload, **kwargs) 1152 self._wishlist |= needed_datasets 1153 -> 1154 self._read_datasets_from_storage(**kwargs) 1155 self.generate_possible_composites(generate, unload) 1156 ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/scene.py in _read_datasets_from_storage(self, **kwargs) 1172 """ 1173 nodes = self._dependency_tree.leaves(limit_nodes_to=self.missing_datasets) -> 1174 return self._read_dataset_nodes_from_storage(nodes, **kwargs) 1175 1176 def _read_dataset_nodes_from_storage(self, reader_nodes, **kwargs): ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/scene.py in _read_dataset_nodes_from_storage(self, reader_nodes, **kwargs) 1178 # Sort requested datasets by reader 1179 reader_datasets = self._sort_dataset_nodes_by_reader(reader_nodes) -> 1180 loaded_datasets = self._load_datasets_by_readers(reader_datasets, **kwargs) 1181 self._datasets.update(loaded_datasets) 1182 return loaded_datasets ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/scene.py in _load_datasets_by_readers(self, reader_datasets, **kwargs) 1203 for reader_name, ds_ids in reader_datasets.items(): 1204 reader_instance = self._readers[reader_name] -> 1205 new_datasets = reader_instance.load(ds_ids, **kwargs) 1206 loaded_datasets.update(new_datasets) 1207 return loaded_datasets ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py in load(self, dataset_keys, previous_datasets, **kwargs) 943 coords = [all_datasets.get(cid, None) 944 for cid in coordinates.get(dsid, [])] --> 945 ds = self._load_dataset_with_area(dsid, coords, **kwargs) 946 if ds is not None: 947 all_datasets[dsid] = ds ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py in _load_dataset_with_area(self, dsid, coords, **kwargs) 839 840 try: --> 841 ds = self._load_dataset_data(file_handlers, dsid, **kwargs) 842 except (KeyError, ValueError) as err: 843 logger.exception("Could not load dataset '%s': %s", dsid, str(err)) ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py in _load_dataset_data(self, file_handlers, dsid, **kwargs) 711 def _load_dataset_data(self, file_handlers, dsid, **kwargs): 712 ds_info = self.all_ids[dsid] --> 713 proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs) 714 # FIXME: areas could be concatenated here 715 # Update the metadata ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py in _load_dataset(dsid, ds_info, file_handlers, dim, **kwargs) 687 for fh in file_handlers: 688 try: --> 689 projectable = fh.get_dataset(dsid, ds_info) 690 if projectable is not None: 691 slice_list.append(projectable) ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/sar_c_safe.py in get_dataset(self, key, info) 131 if not data_items: 132 continue --> 133 data, low_res_coords = self.read_xml_array(data_items, xml_tag) 134 135 if key['name'].endswith('squared'): AttributeError: 'SAFEXML' object has no attribute 'read_xml_array' ```

Screenshots If applicable, add screenshots to help explain your problem.

Environment Info:

• OS: Linux
• Satpy Version: 0.28.1

[yukaribbba]

@mraspaud I encountered this problem a long time ago and #1700 works for me. But, can I request for more composites? Just like the 'preview' on distribution sites https://finder.creodias.eu/files/Sentinel-1/SAR/GRD/2021/07/01/S1B_IW_GRDH_1SDV_20210701T180539_20210701T180604_027604_034B7B_B02C.SAFE/preview/thumbnail.png or something like https://gisbits.wordpress.com/2017/10/28/green-romania-a-sentinel-1-story-of-mosaicing/

[mraspaud]

@yukaribbba thanks for the feedback! I don't know about the preview, as it is smaller and not georeferenced, but the other composite looks really cool! Do I understand right they use for R, G, B respectively VH linear, VV decibel, VV linear?

That would be the kind of composite that satpy should be able to produce indeed.

[yukaribbba]

@mraspaud Yes, for https://gisbits.wordpress.com/2017/10/28/green-romania-a-sentinel-1-story-of-mosaicing/, it seems they did it as what you said. But I have no idea for the preview.

[mraspaud]

Looks like it could work...

I don't have the possibility to play more with this at the moment, but if you @yukaribbba or someone else want to work on this, it just a matter of adding the green-sar composite to the sar.yaml composite file and adjust the corresponding enhancement stretching values.

  green-sar:
    compositor: !!python/name:satpy.composites.GenericCompositor
    prerequisites:
    - name: measurement
      polarization: vh
    - name: measurement
      polarization: vv
      quantity: dB
    - name: measurement
      polarization: vv
      quantity: natural
    standard_name: green-sar

[yukaribbba]

@mraspaud Great , I just saw #1761. I'm planning on test more datasets. Will give more feedbacks later.

[mraspaud]

Perfect! #1761 is a great place to give your feedback :)