Pixel Artifacts in areas of low backscatter

[desmondthemarsbear]

Hello,

this library started to be an interest of mine since I investigate areas of low backscatter on ocean surfaces. Since I started to use the library, I came across image artifacts that are related to pixel areas of very low backscatter. I have managed to draw up three individual preprocessing pipelines with SNAP-GPT, s1denoised and sentinel1_routines , such that the resultant images match pixel perfect. Below is a comparison of a SNAP-GPT and s1denoised image, where for the latter I used both the NERSC and NERSC_TG algorithm. It's mostly visible in the HV channel, but also in the HH channel if the backscatter is low enough (like below).

The following warning during processing hints at the reason for these image artifacts.

/home/mtontsch/micromamba/envs/SAR_preprocessing/lib/python3.10/site-packages/s1denoise/tools.py:55: RuntimeWarning: invalid value encountered in log10                                                                                                                                                                                                          
  d[pol] = (10 * np.log10(d[pol]) - scale[pol] * (inc - angular_offset)).astype(dtype)                                                                                                                                                                                                                                                                           
Correct HV band                                                                                                                                                                                                                                                                                                                                                  
06:16:17|40|nansat|_get_dataset_metadata|GDAL could not open /mnt/raid01/SAR/Sentinel-1/Arctic/WesternSvalbard/raw_root/2017/07/12/S1B_EW_GRDM_1SDH_20170712T063104_20170712T063204_006451_00B573_0B3B.zip, trying to read with Nansat mappers...

Is this a known issue? If so, is there a fairly straight forward solution how to mitigate these artifacts?

I use the library in a bash script with python /home/mtontsch/micromamba/envs/SAR_preprocessing/lib/python3.10/site-packages/s1denoise/scripts/s1_correction.py "$zipDir_path$zipFile_name" "$outputFileAnton_path" -a NERSC_TG -g

The particular S1 example scene below is S1A_EW_GRDM_1SDH_20170708T061439_20170708T061543_017376_01D050_04BE

[avalentino]

Is it not just a matter of very low backscattering areas that have been clipped to zero after denoising (I guess that the conversion to dB masks in some way zero values). I think that this kind of issue may happen if the de-noising process is not perfectly calibrated.

[desmondthemarsbear]

Do you mean the calibration via the denoising_parameters.json data? My guess I would need to use the experimental training and validation scripts to get a better calibration fit for my subset of S1 scenes. How straight forward is this if I have a list with 100+ S1 scenes that I am interested in?

[desmondthemarsbear]

How experimental are these scripts? Using run_experiment.py pb S1A HV /path /path results in a bunch of type errors:

 File "/home/mtontsch/micromamba/envs/SAR_preprocessing/lib/python3.10/site-packages/s1denoise/sentinel1image.py", line 744, in experiment_get_data
    pixel0 = self.noise_range['pixel']
TypeError: 'method' object is not subscriptable

File "/home/mtontsch/micromamba/envs/SAR_preprocessing/lib/python3.10/site-packages/s1denoise/sentinel1image.py", line 757, in experiment_get_data
    sigma0 = self.get_raw_sigma0_vectors_from_full_size(
TypeError: Sentinel1Image.get_raw_sigma0_vectors_from_full_size() got an unexpected keyword argument 'average_lines'

Looking at the 1.3.3. release, regarding the latter type error, the function parameters seem to match, contrary to 1.4.0.

[desmondthemarsbear]

So I managed to get the training scripts of the 1.4.0 version working, and trained on 300+ EW_GRDM_1SDH images, mostly for S1A, since they were from 2015 and 2016. Nonetheless, the resulting .json file from these new calibration parameters did not remove these artifacts.

Then I set up a new environment as shown in the following, and picked a scene near Svalbard that was listed in the training files for IPV 3.1(S1A_EW_GRDM_1SDH_20190707T055115_20190707T055215_028007_0329AA_21C6). The artifacts yet still remain.

    micromamba create -f path/environment.yml -c conda-forge python=3.8

    micromamba activate s1denoise

    pip install netCDF4 nansat 

    pip install https://github.com/nansencenter/sentinel1denoised/archive/v1.4.0.tar.gz

    (s1denoise) polarnc034 ~ % pip freeze
      beautifulsoup4==4.12.3
      certifi==2024.8.30
      cftime==1.6.4
      charset-normalizer==3.4.0
      GDAL @ file:///home/conda/feedstock_root/build_artifacts/libgdal-core_1723837536297/work/build/swig/python
      idna==3.10
      lxml @ file:///home/conda/feedstock_root/build_artifacts/lxml_1723458263021/work
      nansat==1.6.2
      netCDF4==1.7.1.post2
      numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1687808301083/work
      pillow==10.4.0
      pythesint==1.7.0
      python-dateutil==2.9.0.post0
      PyYAML==6.0.2
      rapidfuzz==3.9.7
      requests==2.32.3
      s1denoise @ https://github.com/nansencenter/sentinel1denoised/archive/v1.4.0.tar.gz#sha256=a72fcee52fceba1afc22597d6b7f65d42490d78f1a4ea88cd708288f127603ac
      scipy==1.10.1
      six==1.16.0
      soupsieve==2.6
      urllib3==2.2.3
      xdg==6.0.0

      (s1denoise) polarnc034 ~ % python --version
        Python 3.8.20

(s1denoise) polarnc034 ~ % s1_correction.py path/A1A_EW_GRDM_1SDH_20190707T055115_20190707T055215_028007_0329AA_21C6.zip path/output.tif -a NERSC -g
Correct HH band
WARNING: noise scaling for EW1 (IPF:3.1) is missing.
WARNING: power balancing for EW1 (IPF:3.1) is missing.
WARNING: noise scaling for EW2 (IPF:3.1) is missing.
WARNING: power balancing for EW2 (IPF:3.1) is missing.
WARNING: noise scaling for EW3 (IPF:3.1) is missing.
WARNING: power balancing for EW3 (IPF:3.1) is missing.
WARNING: noise scaling for EW4 (IPF:3.1) is missing.
WARNING: power balancing for EW4 (IPF:3.1) is missing.
WARNING: noise scaling for EW5 (IPF:3.1) is missing.
WARNING: power balancing for EW5 (IPF:3.1) is missing.
Correct HV band
Warning 6: driver GTiff does not support creation option FORMAT

Top row is processed with NERSC, bottom is raw .zip (log scaled). NERSC_TG changes the artifacts, but does not remove them.