search

pytroll / pytroll-examples

Collection of examples for pytroll satellite data processing
GNU General Public License v3.0
76 stars 34 forks source link

outdated sentinel-1 example #42

Closed oarcher closed 3 years ago

oarcher commented 3 years ago

I'm very new to satpy, and I'm trying to use sentinel-1 data.

sentinel1-false-color is outdated, and I am not not able to reproduce the example. I'm using satpy 0.28.1 from conda-forge.

I dont understand what arguments I have to provide to snc.load. Here is my attempt:

from satpy import available_readers, Scene, find_files_and_readers
import datetime

my_files = find_files_and_readers(base_dir='/windows_shared/tmp',
                                  reader='sar-c_safe',
                                  start_time=datetime.datetime(2018, 10, 13, 6, 0, 0),
                                  end_time=datetime.datetime(2018, 10, 13, 7, 30, 0))

scn = Scene(reader='sar-c_safe', filenames=my_files)

scn.load(scn.available_dataset_ids())
AttributeError: 'SAFEXML' object has no attribute 'read_xml_array'

```python Could not load dataset 'DataID(name='latitude', polarization='hh', resolution=80, modifiers=())': "Could not load DataID(name='latitude', polarization='hh', resolution=80, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 698, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='latitude', polarization='hh', resolution=80, modifiers=()) from any provided files" Could not load dataset 'DataID(name='latitude', polarization='hv', resolution=80, modifiers=())': "Could not load DataID(name='latitude', polarization='hv', resolution=80, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 698, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='latitude', polarization='hv', resolution=80, modifiers=()) from any provided files" Could not load dataset 'DataID(name='longitude', polarization='hh', resolution=80, modifiers=())': "Could not load DataID(name='longitude', polarization='hh', resolution=80, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 698, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='longitude', polarization='hh', resolution=80, modifiers=()) from any provided files" Could not load dataset 'DataID(name='longitude', polarization='hv', resolution=80, modifiers=())': "Could not load DataID(name='longitude', polarization='hv', resolution=80, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 698, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='longitude', polarization='hv', resolution=80, modifiers=()) from any provided files" Could not load dataset 'DataID(name='gamma_squared', polarization='hh', resolution=80, modifiers=())': "Could not load DataID(name='gamma_squared', polarization='hh', resolution=80, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 698, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='gamma_squared', polarization='hh', resolution=80, modifiers=()) from any provided files" Could not load dataset 'DataID(name='measurement', polarization='hv', resolution=80, calibration=, quantity=, modifiers=())': "Could not load DataID(name='measurement', polarization='hv', resolution=80, calibration=, quantity=, modifiers=()) from any provided files" Traceback (most recent call last): File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs) File "/home/oarcher/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py", line 698, in _load_dataset raise KeyError( KeyError: "Could not load DataID(name='measurement', polarization='hv', resolution=80, calibration=, quantity=, modifiers=()) from any provided files" --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) in ----> 1 scn.load(scn.available_dataset_ids()) ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/scene.py in load(self, wishlist, calibration, resolution, polarization, level, generate, unload, **kwargs) 1152 self._wishlist |= needed_datasets 1153 -> 1154 self._read_datasets_from_storage(**kwargs) 1155 self.generate_possible_composites(generate, unload) 1156 ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/scene.py in _read_datasets_from_storage(self, **kwargs) 1172 """ 1173 nodes = self._dependency_tree.leaves(limit_nodes_to=self.missing_datasets) -> 1174 return self._read_dataset_nodes_from_storage(nodes, **kwargs) 1175 1176 def _read_dataset_nodes_from_storage(self, reader_nodes, **kwargs): ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/scene.py in _read_dataset_nodes_from_storage(self, reader_nodes, **kwargs) 1178 # Sort requested datasets by reader 1179 reader_datasets = self._sort_dataset_nodes_by_reader(reader_nodes) -> 1180 loaded_datasets = self._load_datasets_by_readers(reader_datasets, **kwargs) 1181 self._datasets.update(loaded_datasets) 1182 return loaded_datasets ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/scene.py in _load_datasets_by_readers(self, reader_datasets, **kwargs) 1203 for reader_name, ds_ids in reader_datasets.items(): 1204 reader_instance = self._readers[reader_name] -> 1205 new_datasets = reader_instance.load(ds_ids, **kwargs) 1206 loaded_datasets.update(new_datasets) 1207 return loaded_datasets ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py in load(self, dataset_keys, previous_datasets, **kwargs) 943 coords = [all_datasets.get(cid, None) 944 for cid in coordinates.get(dsid, [])] --> 945 ds = self._load_dataset_with_area(dsid, coords, **kwargs) 946 if ds is not None: 947 all_datasets[dsid] = ds ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py in _load_dataset_with_area(self, dsid, coords, **kwargs) 839 840 try: --> 841 ds = self._load_dataset_data(file_handlers, dsid, **kwargs) 842 except (KeyError, ValueError) as err: 843 logger.exception("Could not load dataset '%s': %s", dsid, str(err)) ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py in _load_dataset_data(self, file_handlers, dsid, **kwargs) 711 def _load_dataset_data(self, file_handlers, dsid, **kwargs): 712 ds_info = self.all_ids[dsid] --> 713 proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs) 714 # FIXME: areas could be concatenated here 715 # Update the metadata ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/yaml_reader.py in _load_dataset(dsid, ds_info, file_handlers, dim, **kwargs) 687 for fh in file_handlers: 688 try: --> 689 projectable = fh.get_dataset(dsid, ds_info) 690 if projectable is not None: 691 slice_list.append(projectable) ~/anaconda3/envs/xsar/lib/python3.9/site-packages/satpy/readers/sar_c_safe.py in get_dataset(self, key, info) 131 if not data_items: 132 continue --> 133 data, low_res_coords = self.read_xml_array(data_items, xml_tag) 134 135 if key['name'].endswith('squared'): AttributeError: 'SAFEXML' object has no attribute 'read_xml_array' ```

mraspaud commented 3 years ago

Hi @oarcher, I have thanks for bringing this to our attention! I will try to reproduce the problem later today, but just to make sure, you are using unzipped GRD data, right?

oarcher commented 3 years ago

Hi @mraspaud , thanks for your attention !

Yes, I am using unzipped GRD data.

mraspaud commented 3 years ago

Merci pour les precisions!

I can reproduce the problem, so we definitely need to fix this: It seems like not all available datasets can be loaded, this is an issue I need to raise in satpy.

In the meanwhile, if you want a fresh example, you can try this (works best over sea-ice :), using EW data)

scn = Scene(reader='sar-c_safe', filenames=my_files)
scn.load(["sar-ice-log"])
scn.show("sar-ice-log")
mraspaud commented 3 years ago

@oarcher the example has now been updated. Thanks again for reporting this!

oarcher commented 3 years ago

Merci, ça marche !

Just to introduce myself, as I am a very new satpy user.

I work with S1 IW EW WV L1 data in GRD or SLC, over ocean, to retrieve wind speed and waves spectra.

I wrote xsar, that is a sentinel1 xarray/dask reader (see xsar.open_dataset example), and it's fits my needs for high resolution computations (like moving box gradients analysis).

But I am not happy for the georeferencing part, like landmask and coastlines. Satpy seems to be much better for that.

So my plan with satpy is to try the sar_c reader, to see if it can fit my computational needs. I will be also able to cross validate both xsar and satpy readers.

Ideally, I'd like drop xsar to use satpy. But if I've got problems, I will probably use something like pyresample in xsar.

mraspaud commented 3 years ago

Nice meeting you :) xsar looks great!

The satpy SAR_C reader is still quite rough around the edges, simply because I don't have much time to allocate to it, but I'm definitely interested in making it better (especially for reading different datasets, and adding SLC support). So I'd be happy helping you out with testing satpy, that will help me pinpoint what needs to be improved first for SAR processing.

Anyway, feel free to ping me in issues on github or even join us on the pytroll slack (see pytroll.github.io) if you want to discuss further.

Bon week-end! Martin