Patch to get correct line coordinates in the range noise dataset for SLC products

agrouaze commented 7 months ago

issue in the data IW SLC documented here: https://jira-projects.cls.fr/browse/MPCS-3581 and https://github.com/umr-lops/xsar_slc/issues/175 to replace https://github.com/umr-lops/xarray-safe-s1/pull/20 An example of product which needs correction: safe = '/home/datawork-cersat-public/cache/project/mpc-sentinel1/data/esa/sentinel-1a/L1/IW/S1A_IW_SLC__1S/2021/026/S1A_IW_SLC__1SDV_20210126T154330_20210126T154357_036311_0442A0_0003.SAFE' #subswath 1 -> shift 1503 lines

[x] check IW SLC
[x] check EW SLC -> found an issue, line_jump_meas=14 while line_jump_noise=15 -> we exclude EW from this patch for now.
[x] check IW GRD (not concerned by the correction)
[x] check IW SL2 (currently azimuth time used to correct is not OK -> cannot check it properly)
[x] check WV SLC (not concerned by the correction)
[x] check EW GRD (not concerned by the correction)
[x] compile sphinx doc
[x] test over 300 slice IW SLC (900 subswath)

agrouaze commented 7 months ago

for EW SLC the correction cannot be done for now:

ValueError                                Traceback (most recent call last)
Cell In[23], line 9
      7 reload(xsar)
      8 print(xsar.__file__)
----> 9 handlerS1 = xsar.Sentinel1Dataset(prod_name)
     10 # obxsar = xsar.open_dataset(prod_name)
     11 # obxsar
     12 handlerS1.datatree

File [/home1/datahome/agrouaze/sources/git/xsar/src/xsar/sentinel1_dataset.py:297](http://grougrou1:8888/home1/datahome/agrouaze/sources/git/xsar/src/xsar/sentinel1_dataset.py#line=296), in Sentinel1Dataset.__init__(***failed resolving arguments***)
    295 self.datatree['measurement'].attrs = self.datatree.attrs
    296 if self.sar_meta.product == 'SLC' and 'WV' not in self.sar_meta.swath:  # TOPS cases
--> 297     tmp = self.corrected_range_noise_lut(self.datatree)
    298     self.datatree['noise_range'].ds = tmp # the corrcted noise_range dataset shold now contain an attrs 'corrected_range_noise_lut'
    299 self.sliced = False

File [/home1/datahome/agrouaze/sources/git/xsar/src/xsar/sentinel1_dataset.py:323](http://grougrou1:8888/home1/datahome/agrouaze/sources/git/xsar/src/xsar/sentinel1_dataset.py#line=322), in Sentinel1Dataset.corrected_range_noise_lut(self, dt)
    321 line_jump_meas = dt['measurement']['line'][i_jump] # line number of jumps
    322 line_jump_noise = np.ravel(dt['noise_range']['line'][1:-1].data) # annotated line number of burst begining, this one is corrupted for some S1 TOPS product
--> 323 burst_first_lineshift = line_jump_meas-line_jump_noise
    324 if len(np.unique(burst_first_lineshift))==1:
    325     line_shift = int(np.unique(burst_first_lineshift)[0])

File [/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/_typed_ops.py:248](http://grougrou1:8888/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/_typed_ops.py#line=247), in DataArrayOpsMixin.__sub__(self, other)
    247 def __sub__(self, other: DaCompatible) -> Self:
--> 248     return self._binary_op(other, operator.sub)

File [/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/dataarray.py:4657](http://grougrou1:8888/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/dataarray.py#line=4656), in DataArray._binary_op(self, other, f, reflexive)
   4653 other_variable_or_arraylike: DaCompatible = getattr(other, "variable", other)
   4654 other_coords = getattr(other, "coords", None)
   4656 variable = (
-> 4657     f(self.variable, other_variable_or_arraylike)
   4658     if not reflexive
   4659     else f(other_variable_or_arraylike, self.variable)
   4660 )
   4661 coords, indexes = self.coords._merge_raw(other_coords, reflexive)
   4662 name = self._result_name(other)

File [/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/_typed_ops.py:476](http://grougrou1:8888/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/_typed_ops.py#line=475), in VariableOpsMixin.__sub__(self, other)
    475 def __sub__(self, other: VarCompatible) -> Self | T_DataArray:
--> 476     return self._binary_op(other, operator.sub)

File [/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/variable.py:2414](http://grougrou1:8888/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/variable.py#line=2413), in Variable._binary_op(self, other, f, reflexive)
   2411 attrs = self._attrs if keep_attrs else None
   2412 with np.errstate(all="ignore"):
   2413     new_data = (
-> 2414         f(self_data, other_data) if not reflexive else f(other_data, self_data)
   2415     )
   2416 result = Variable(dims, new_data, attrs=attrs)
   2417 return result

ValueError: operands could not be broadcast together with shapes (14,) (15,)

to be tested with : SENTINEL1_DS:/home/datawork-cersat-public/cache/project/mpc-sentinel1/data/esa/sentinel-1a/L1/EW/S1A_EW_SLC__1S/2018/272/S1A_EW_SLC__1SDH_20180929T054258_20180929T054346_023909_029C29_88FD.SAFE:EW1

agrouaze commented 7 months ago

for EW SLC the correction cannot be done for now:

ValueError                                Traceback (most recent call last)
Cell In[23], line 9
      7 reload(xsar)
      8 print(xsar.__file__)
----> 9 handlerS1 = xsar.Sentinel1Dataset(prod_name)
     10 # obxsar = xsar.open_dataset(prod_name)
     11 # obxsar
     12 handlerS1.datatree

File [/home1/datahome/agrouaze/sources/git/xsar/src/xsar/sentinel1_dataset.py:297](http://grougrou1:8888/home1/datahome/agrouaze/sources/git/xsar/src/xsar/sentinel1_dataset.py#line=296), in Sentinel1Dataset.__init__(***failed resolving arguments***)
    295 self.datatree['measurement'].attrs = self.datatree.attrs
    296 if self.sar_meta.product == 'SLC' and 'WV' not in self.sar_meta.swath:  # TOPS cases
--> 297     tmp = self.corrected_range_noise_lut(self.datatree)
    298     self.datatree['noise_range'].ds = tmp # the corrcted noise_range dataset shold now contain an attrs 'corrected_range_noise_lut'
    299 self.sliced = False

File [/home1/datahome/agrouaze/sources/git/xsar/src/xsar/sentinel1_dataset.py:323](http://grougrou1:8888/home1/datahome/agrouaze/sources/git/xsar/src/xsar/sentinel1_dataset.py#line=322), in Sentinel1Dataset.corrected_range_noise_lut(self, dt)
    321 line_jump_meas = dt['measurement']['line'][i_jump] # line number of jumps
    322 line_jump_noise = np.ravel(dt['noise_range']['line'][1:-1].data) # annotated line number of burst begining, this one is corrupted for some S1 TOPS product
--> 323 burst_first_lineshift = line_jump_meas-line_jump_noise
    324 if len(np.unique(burst_first_lineshift))==1:
    325     line_shift = int(np.unique(burst_first_lineshift)[0])

File [/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/_typed_ops.py:248](http://grougrou1:8888/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/_typed_ops.py#line=247), in DataArrayOpsMixin.__sub__(self, other)
    247 def __sub__(self, other: DaCompatible) -> Self:
--> 248     return self._binary_op(other, operator.sub)

File [/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/dataarray.py:4657](http://grougrou1:8888/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/dataarray.py#line=4656), in DataArray._binary_op(self, other, f, reflexive)
   4653 other_variable_or_arraylike: DaCompatible = getattr(other, "variable", other)
   4654 other_coords = getattr(other, "coords", None)
   4656 variable = (
-> 4657     f(self.variable, other_variable_or_arraylike)
   4658     if not reflexive
   4659     else f(other_variable_or_arraylike, self.variable)
   4660 )
   4661 coords, indexes = self.coords._merge_raw(other_coords, reflexive)
   4662 name = self._result_name(other)

File [/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/_typed_ops.py:476](http://grougrou1:8888/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/_typed_ops.py#line=475), in VariableOpsMixin.__sub__(self, other)
    475 def __sub__(self, other: VarCompatible) -> Self | T_DataArray:
--> 476     return self._binary_op(other, operator.sub)

File [/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/variable.py:2414](http://grougrou1:8888/opt/conda-envs/dev/lib/python3.10/site-packages/xarray/core/variable.py#line=2413), in Variable._binary_op(self, other, f, reflexive)
   2411 attrs = self._attrs if keep_attrs else None
   2412 with np.errstate(all="ignore"):
   2413     new_data = (
-> 2414         f(self_data, other_data) if not reflexive else f(other_data, self_data)
   2415     )
   2416 result = Variable(dims, new_data, attrs=attrs)
   2417 return result

ValueError: operands could not be broadcast together with shapes (14,) (15,)

to be tested with : SENTINEL1_DS:/home/datawork-cersat-public/cache/project/mpc-sentinel1/data/esa/sentinel-1a/L1/EW/S1A_EW_SLC__1S/2018/272/S1A_EW_SLC__1SDH_20180929T054258_20180929T054346_023909_029C29_88FD.SAFE:EW1

Solution proposed by @lanougue : line_jump_noise = np.ravel(dt['noise_range']['line'][1:1+len(line_jump_meas)].data) # annoted line number of burst begining

agrouaze commented 6 months ago

still an issue with S1A_IW_SLC__1SDV_20170907T102951_20170907T103021_018268_01EB76_AE7B.SAFE:

ValueError                                Traceback (most recent call last)
Cell In[2], line 5
      3 for subswath in multids.subdatasets.index:
      4     print(subswath)
----> 5     onesubswath = xsar.Sentinel1Dataset(subswath)
      6     print(onesubswath.get_bursts_polygons()['geometry'])

File ~/Documents/sources/xsar/src/xsar/sentinel1_dataset.py:297, in Sentinel1Dataset.__init__(***failed resolving arguments***)
    295 self.datatree['measurement'].attrs = self.datatree.attrs
    296 if "SLC" in str(self.datatree.attrs['product']) and 'WV' not in self.sar_meta.swath: # only TOPS IW or EW SLC
--> 297     tmp = self.corrected_range_noise_lut(self.datatree)
    298     self.datatree['noise_range'].ds = tmp # the corrected noise_range dataset should now contain an attrs 'corrected_range_noise_lut'
    299     # self.datatree = self.corrected_range_noise_lut(self.datatree)

File ~/Documents/sources/xsar/src/xsar/sentinel1_dataset.py:324, in Sentinel1Dataset.corrected_range_noise_lut(self, dt)
    322 line_jump_meas = dt['measurement']['line'][i_jump] # line number of jumps
    323 line_jump_noise = np.ravel(dt['noise_range']['line'][1:-1].data) # annotated line number of burst begining
--> 324 burst_first_lineshift = line_jump_meas-line_jump_noise
    325 if len(np.unique(burst_first_lineshift))==1:
    326     line_shift = int(np.unique(burst_first_lineshift)[0])

File ~/micromamba/envs/micromamba1/lib/python3.10/site-packages/xarray/core/_typed_ops.py:248, in DataArrayOpsMixin.__sub__(self, other)
    247 def __sub__(self, other: DaCompatible) -> Self:
--> 248     return self._binary_op(other, operator.sub)

File ~/micromamba/envs/micromamba1/lib/python3.10/site-packages/xarray/core/dataarray.py:4687, in DataArray._binary_op(self, other, f, reflexive)
   4683 other_variable_or_arraylike: DaCompatible = getattr(other, "variable", other)
   4684 other_coords = getattr(other, "coords", None)
   4686 variable = (
-> 4687     f(self.variable, other_variable_or_arraylike)
   4688     if not reflexive
   4689     else f(other_variable_or_arraylike, self.variable)
   4690 )
   4691 coords, indexes = self.coords._merge_raw(other_coords, reflexive)
   4692 name = self._result_name(other)

File ~/micromamba/envs/micromamba1/lib/python3.10/site-packages/xarray/core/_typed_ops.py:476, in VariableOpsMixin.__sub__(self, other)
    475 def __sub__(self, other: VarCompatible) -> Self | T_DataArray:
--> 476     return self._binary_op(other, operator.sub)

File ~/micromamba/envs/micromamba1/lib/python3.10/site-packages/xarray/core/variable.py:2411, in Variable._binary_op(self, other, f, reflexive)
   2408 attrs = self._attrs if keep_attrs else None
   2409 with np.errstate(all="ignore"):
   2410     new_data = (
-> 2411         f(self_data, other_data) if not reflexive else f(other_data, self_data)
   2412     )
   2413 result = Variable(dims, new_data, attrs=attrs)
   2414 return result

ValueError: operands could not be broadcast together with shapes (9,) (10,) 

You can ignore this error by setting the following in conf.py:

    nbsphinx_allow_errors = True

make: *** [Makefile:21: html] Error 2

agrouaze commented 6 months ago

still an issue with S1A_IW_SLC__1SDV_20170907T102951_20170907T103021_018268_01EB76_AE7B.SAFE:

ValueError                                Traceback (most recent call last)
Cell In[2], line 5
      3 for subswath in multids.subdatasets.index:
      4     print(subswath)
----> 5     onesubswath = xsar.Sentinel1Dataset(subswath)
      6     print(onesubswath.get_bursts_polygons()['geometry'])

File ~/Documents/sources/xsar/src/xsar/sentinel1_dataset.py:297, in Sentinel1Dataset.__init__(***failed resolving arguments***)
    295 self.datatree['measurement'].attrs = self.datatree.attrs
    296 if "SLC" in str(self.datatree.attrs['product']) and 'WV' not in self.sar_meta.swath: # only TOPS IW or EW SLC
--> 297     tmp = self.corrected_range_noise_lut(self.datatree)
    298     self.datatree['noise_range'].ds = tmp # the corrected noise_range dataset should now contain an attrs 'corrected_range_noise_lut'
    299     # self.datatree = self.corrected_range_noise_lut(self.datatree)

File ~/Documents/sources/xsar/src/xsar/sentinel1_dataset.py:324, in Sentinel1Dataset.corrected_range_noise_lut(self, dt)
    322 line_jump_meas = dt['measurement']['line'][i_jump] # line number of jumps
    323 line_jump_noise = np.ravel(dt['noise_range']['line'][1:-1].data) # annotated line number of burst begining
--> 324 burst_first_lineshift = line_jump_meas-line_jump_noise
    325 if len(np.unique(burst_first_lineshift))==1:
    326     line_shift = int(np.unique(burst_first_lineshift)[0])

File ~/micromamba/envs/micromamba1/lib/python3.10/site-packages/xarray/core/_typed_ops.py:248, in DataArrayOpsMixin.__sub__(self, other)
    247 def __sub__(self, other: DaCompatible) -> Self:
--> 248     return self._binary_op(other, operator.sub)

File ~/micromamba/envs/micromamba1/lib/python3.10/site-packages/xarray/core/dataarray.py:4687, in DataArray._binary_op(self, other, f, reflexive)
   4683 other_variable_or_arraylike: DaCompatible = getattr(other, "variable", other)
   4684 other_coords = getattr(other, "coords", None)
   4686 variable = (
-> 4687     f(self.variable, other_variable_or_arraylike)
   4688     if not reflexive
   4689     else f(other_variable_or_arraylike, self.variable)
   4690 )
   4691 coords, indexes = self.coords._merge_raw(other_coords, reflexive)
   4692 name = self._result_name(other)

File ~/micromamba/envs/micromamba1/lib/python3.10/site-packages/xarray/core/_typed_ops.py:476, in VariableOpsMixin.__sub__(self, other)
    475 def __sub__(self, other: VarCompatible) -> Self | T_DataArray:
--> 476     return self._binary_op(other, operator.sub)

File ~/micromamba/envs/micromamba1/lib/python3.10/site-packages/xarray/core/variable.py:2411, in Variable._binary_op(self, other, f, reflexive)
   2408 attrs = self._attrs if keep_attrs else None
   2409 with np.errstate(all="ignore"):
   2410     new_data = (
-> 2411         f(self_data, other_data) if not reflexive else f(other_data, self_data)
   2412     )
   2413 result = Variable(dims, new_data, attrs=attrs)
   2414 return result

ValueError: operands could not be broadcast together with shapes (9,) (10,) 

You can ignore this error by setting the following in conf.py:

    nbsphinx_allow_errors = True

make: *** [Makefile:21: html] Error 2

this issue is solved with latest patch.

umr-lops / xsar

Patch to get correct line coordinates in the range noise dataset for SLC products #204