No module named 's1etad_tools'

[abradley60]

Hi there, thanks for making this code publicly available. We are applying ETAD corrections directly to slc files as a part of our sentinel-1 workflow, and on a recent environment build I been encountering issues with installing s1etad that I have not previously faced. It appears that s1etad_tools is not longer installing.

Traceback (most recent call last):
  File "/home/ec2-user/sar-rtc-opera/rtc_otf.py", line 13, in <module>
    from etad import *
  File "/home/ec2-user/sar-rtc-opera/etad.py", line 14, in <module>
    from s1etad_tools.cli.slc_correct import s1etad_slc_correct_main
ModuleNotFoundError: No module named 's1etad_tools'
(rtc_opera) (rtc_otf_env) [ec2-user@ip-10-0-10-151 sar-rtc-opera]$ 

Here are the imports that are now breaking but were previously working (s1etad_tools is not found).

conda env create --file environment.yml

from s1etad import Sentinel1Etad, ECorrectionType
from s1etad_tools.cli.slc_correct import s1etad_slc_correct_main

My conda environment file if you wish to replicate:

name: rtc_opera
channels:
  - s1-etad
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1
  - _openmp_mutex=4.5
  - alsa-lib=1.2.11
  - argcomplete=3.2.2
  - attr=2.5.1
  - aws-c-auth=0.7.16
  - aws-c-cal=0.6.10
  - aws-c-common=0.9.13
  - aws-c-compression=0.2.18
  - aws-c-event-stream=0.4.2
  - aws-c-http=0.8.1
  - aws-c-io=0.14.5
  - aws-c-mqtt=0.10.2
  - aws-c-s3=0.5.2
  - aws-c-sdkutils=0.1.15
  - aws-checksums=0.1.18
  - aws-crt-cpp=0.26.2
  - aws-sdk-cpp=1.11.267
  - azure-core-cpp=1.11.1
  - azure-storage-blobs-cpp=12.10.0
  - azure-storage-common-cpp=12.5.0
  - blosc=1.21.5
  - brotli=1.1.0
  - brotli-bin=1.1.0
  - bzip2=1.0.8
  - c-ares=1.27.0
  - ca-certificates=2024.2.2
  - cairo=1.18.0
  - certifi=2024.2.2
  - cfitsio=4.3.1
  - cftime=1.6.3
  - contourpy=1.2.0
  - cycler=0.12.1
  - dbus=1.13.6
  - expat=2.6.1
  - font-ttf-dejavu-sans-mono=2.37
  - font-ttf-inconsolata=3.000
  - font-ttf-source-code-pro=2.038
  - font-ttf-ubuntu=0.83
  - fontconfig=2.14.2
  - fonts-conda-ecosystem=1
  - fonts-conda-forge=1
  - fonttools=4.49.0
  - freetype=2.12.1
  - freexl=2.0.0
  - gdal=3.8.4
  - geos=3.12.1
  - geotiff=1.7.1
  - gettext=0.21.1
  - giflib=5.2.1
  - glib=2.78.4
  - glib-tools=2.78.4
  - graphite2=1.3.13
  - gst-plugins-base=1.22.9
  - gstreamer=1.22.9
  - harfbuzz=8.3.0
  - hdf4=4.2.15
  - hdf5=1.14.3
  - icu=73.2
  - importlib-resources=6.1.2
  - importlib_resources=6.1.2
  - json-c=0.17
  - kealib=1.5.3
  - keyutils=1.6.1
  - kiwisolver=1.4.5
  - krb5=1.21.2
  - lame=3.100
  - lcms2=2.16
  - ld_impl_linux-64=2.40
  - lerc=4.0.0
  - libabseil=20240116.1
  - libaec=1.1.2
  - libarchive=3.7.2
  - libblas=3.9.0
  - libboost-headers=1.84.0
  - libbrotlicommon=1.1.0
  - libbrotlidec=1.1.0
  - libbrotlienc=1.1.0
  - libcap=2.69
  - libcblas=3.9.0
  - libclang=15.0.7
  - libclang13=15.0.7
  - libcrc32c=1.1.2
  - libcups=2.3.3
  - libcurl=8.5.0
  - libdeflate=1.19
  - libedit=3.1.20191231
  - libev=4.33
  - libevent=2.1.12
  - libexpat=2.6.1
  - libffi=3.4.2
  - libflac=1.4.3
  - libgcc-ng=13.2.0
  - libgcrypt=1.10.3
  - libgdal=3.8.4
  - libgfortran-ng=13.2.0
  - libgfortran5=13.2.0
  - libglib=2.78.4
  - libgomp=13.2.0
  - libgoogle-cloud=2.21.0
  - libgoogle-cloud-storage=2.21.0
  - libgpg-error=1.48
  - libgrpc=1.61.1
  - libiconv=1.17
  - libjpeg-turbo=3.0.0
  - libkml=1.3.0
  - liblapack=3.9.0
  - libllvm15=15.0.7
  - libnetcdf=4.9.2
  - libnghttp2=1.58.0
  - libnsl=2.0.1
  - libogg=1.3.4
  - libopenblas=0.3.26
  - libopus=1.3.1
  - libpng=1.6.43
  - libpq=16.2
  - libprotobuf=4.25.2
  - libre2-11=2023.09.01
  - librttopo=1.1.0
  - libsndfile=1.2.2
  - libspatialite=5.1.0
  - libsqlite=3.45.1
  - libssh2=1.11.0
  - libstdcxx-ng=13.2.0
  - libsystemd0=255
  - libtiff=4.6.0
  - libuuid=2.38.1
  - libvorbis=1.3.7
  - libwebp-base=1.3.2
  - libxcb=1.15
  - libxcrypt=4.4.36
  - libxkbcommon=1.6.0
  - libxml2=2.12.5
  - libxslt=1.1.39
  - libzip=1.10.1
  - libzlib=1.2.13
  - lxml=5.1.0
  - lz4-c=1.9.4
  - lzo=2.10
  - matplotlib=3.8.3
  - matplotlib-base=3.8.3
  - minizip=4.0.5
  - mpg123=1.32.4
  - munkres=1.1.4
  - mysql-common=8.0.33
  - mysql-libs=8.0.33
  - ncurses=6.4
  - netcdf4=1.6.5
  - nspr=4.35
  - nss=3.98
  - numpy=1.26.4
  - openjpeg=2.5.2
  - openssl=3.2.1
  - packaging=23.2
  - pandas=2.2.1
  - pcre2=10.42
  - pillow=10.2.0
  - pip=24.0
  - pixman=0.43.2
  - ply=3.11
  - poppler=24.02.0
  - poppler-data=0.4.12
  - postgresql=16.2
  - proj=9.3.1
  - pthread-stubs=0.4
  - pulseaudio-client=16.1
  - pyparsing=3.1.2
  - pyqt=5.15.9
  - pyqt5-sip=12.12.2
  - python=3.9.18
  - python-tzdata=2024.1
  - python_abi=3.9
  - pytz=2024.1
  - qt-main=5.15.8
  - re2=2023.09.01
  - readline=8.2
  - s1etad_tools=0.8.1
  - s2n=1.4.5
  - scipy=1.12.0
  - setuptools=69.1.1
  - shapely=2.0.3
  - simplekml=1.3.6
  - sip=6.7.12
  - six=1.16.0
  - snappy=1.1.10
  - sqlite=3.45.1
  - tiledb=2.20.1
  - tk=8.6.13
  - toml=0.10.2
  - tomli=2.0.1
  - tornado=6.4
  - tzcode=2024a
  - tzdata=2024a
  - unicodedata2=15.1.0
  - uriparser=0.9.7
  - wheel=0.42.0
  - xcb-util=0.4.0
  - xcb-util-image=0.4.0
  - xcb-util-keysyms=0.4.0
  - xcb-util-renderutil=0.3.9
  - xcb-util-wm=0.4.1
  - xerces-c=3.2.5
  - xkeyboard-config=2.41
  - xorg-kbproto=1.0.7
  - xorg-libice=1.1.1
  - xorg-libsm=1.2.4
  - xorg-libx11=1.8.7
  - xorg-libxau=1.0.11
  - xorg-libxdmcp=1.1.3
  - xorg-libxext=1.3.4
  - xorg-libxrender=0.9.11
  - xorg-renderproto=0.11.1
  - xorg-xextproto=7.3.0
  - xorg-xf86vidmodeproto=2.3.1
  - xorg-xproto=7.0.31
  - xz=5.2.6
  - zipp=3.17.0
  - zlib=1.2.13
  - zstd=1.5.5
  - pip:
      - affine==2.4.0
      - asf-search==7.0.6
      - attrs==23.2.0
      - boto3==1.34.57
      - botocore==1.34.57
      - charset-normalizer==3.3.2
      - click==8.1.7
      - click-plugins==1.1.1
      - cligj==0.7.2
      - dateparser==1.2.0
      - dem-stitcher==2.5.5
      - docker==7.0.0
      - fiona==1.9.5
      - geopandas==0.14.3
      - idna==3.6
      - importlib-metadata==7.0.1
      - jmespath==1.0.1
      - pymap3d==3.1.0
      - pyproj==3.6.1
      - python-dateutil==2.9.0.post0
      - pyyaml==6.0.1
      - rasterio==1.3.9
      - regex==2023.12.25
      - requests==2.31.0
      - s1etad==0.5.4
      - s3transfer==0.10.0
      - sentineleof==0.9.5
      - snuggs==1.4.7
      - tenacity==8.2.2
      - tqdm==4.66.2
      - tzlocal==5.2
      - urllib3==1.26.18

[avalentino]

Sorry @abradley60, but I need more context. If I remember correctly s1etad_tools was never part of the s1etad package. It is an experimental code in a sandbox.

[abradley60]

@avalentino yes apologies, they are two separate things but I had been working with them together. I have isolated the issue and it was a problem with multiple conflicting Conda environments on my end. This issue is resolved and can be closed.

You mention s1etad_tools is experimental, I really like the ease of the s1etad_slc_correct_main function to apply corrections directly to an slc. Do you plan on a firm release?

[avalentino]

Thanks @abradley60 for the feedback. The problem with s1etad_tools is that the 2D interpolator for SLC data is largely suboptimal (very time consuming and memory hangry). This is why the code repository was never published, I only made available conda packages that were needed in the context of the S1-NRB project.

Another issue is related to the interpolation of the tropospheric layer that , currently, does not take into account of the vertical gradient and could introduce artifacts in hi-resolution interferograms.

If you think that it is useful I can talk to my colleagues and publish the repository. In this period I do not have time to improve it and to introduce at least the correct troposphere interpolation but I do not exclude that I could do it in the (not so far) future.

May I ask for what kind of application do you use it?

[abradley60]

Thanks @avalentino . I was not aware of those existing issues so that is good information for me.

We are doing some software testing for creating radiometrically terrain corrected (RTC) backscatter products over Australia and Antarctica. We have an interest in creating products at a continental scale and wanted to understand the different software options we have. We have found that the ETAD (applied with the etad_tools package) increases the geometric accuracy of our scenes. We have tested using corner reflectors over both Australia and Antarctica. However, one thing we have noticed is that the ETAD correction appears so slightly worsen accuracy in the azimuth direction. Here is a preliminary figure showing the geolocation accuracy of products created at a 10m resolution with four softwares we are considering. The figure considers 40 corner reflectors across 12 scenes.

We are not yet considering ETAD for interferometric products but that may come down the line. Do you have any concerns with our current approach to applying the corrections for RTC products?

[avalentino]

Thanks for sharing. I'm very interested in your analysis, it could be relevant to one of the project I'm following. Maybe I can contact you via email if you agree.

Regarding your questions, I think that, for the assessment of geolocation, the current implementation should be fine, also considering that CR are on relatively flat areas.

I'm a little bit surprised by the results obtained with SNAP. According to [1] the out-of the box geolocation error for S1 SLC products is in the order of 2-4m, apparently SNAP is introducing a very large degradation.

Regarding results obtained with ISCE3+ETAD I see your point but I do not have an explanation at the moment. I should look more closely at the ISCE3 code.

As a general comment the out of the box accuracy of S1 SLC is better than 1m in azimuth and 0.5m in range [1], so the general comment is that the RTC processing and the reduced resolution seem to degrade the geolocation accuracy in a non negligible way.

This is even more evident when the ETAD product is applied, indeed, according to [1], the S1 SLC geolocation accuracy over the Surat calibration site should be in the order of 30cm in azimuth and 6cm in range.

[1] https://sentiwiki.copernicus.eu/__attachments/1681272/SAR-MPC-0634%20-%20Sentienl-1%20Annual%20Performance%20Report%202023%20-%201.3.pdf

[avalentino]

@abradley60 I have just discussed shortly with my colleagues and we find this study case very interesting. Please feel free to contact us at "s1-etad at esa dot int"

[johntruckenbrodt]

@abradley60, @avalentino I just discovered this. Would you mind adding me to the further discussions? I'd be very interested.

[avalentino]

Sure @johntruckenbrodt, it is always a pleasure to discuss with you :) Do you have any comment regarding the results obtained with SANP? Could it be another issue related to DEM management?

[abradley60]

Great thanks for that @avalentino. I agree the results are curious and we are keen to dive deeper to understand what is happening. The Copernicus 30m DEM referenced over the WGS84 Ellipsoid is being used for all products. S1ard handles the DEM sourcing itself which could explain the small differences between pyroSAR-SNAP and S1ard-SNAP. Where the larger SNAP offset and increased variance across corner reflectors comes from we are unsure and are keen to investigate further.

Excellent @johntruckenbrodt! It would be great to have you involved. We were planning to discuss our consolidated results with you so perhaps now is a great time to start the conversation. This is a part of a broader study to compare the software options so you may be interested in some of our other findings too. We are happy to share files and code.

I will send an email to "s1-etad at esa dot int" and cc you in John along with some other members of my team.

Thank you both for your interest.

[johntruckenbrodt]

Sounds great! Yes indeed, the DEM handling likely plays a major role. pyroSAR does offer functionality to prepare the DEM and use it in SNAP. This is also used inside s1ard. The major reasons are full control over the output grid and the possibility to seamlessly extrapolate the DEM tiles over ocean (where no DEM tiles exist). So it sure would be interesting to compare results between the SNAP auto-downloaded Copernicus DEM, the pyroSAR-prepared Copernicus DEM, and the SNAP auto-downloaded SRTM DEM w.r.t. ALE.
Furthermore, I think the output CRS plays a role as well. s1ard outputs data in UTM. The DEM itself however needs to be prepared in WGS84 because otherwise the RTC result is inferior. It would be worth checking differences between WGS84 and UTM as geocoding output.

Some further reading: https://forum.step.esa.int/t/pixel-misalignment-after-processing-series-of-sentinel-1-grd-images/32674 https://forum.step.esa.int/t/terrain-flattening-results-in-garbage-when-using-copernicus-30m-auto-download/33777 https://forum.step.esa.int/t/pixel-alignment-dem-resampling-during-terrain-correction/33854 https://forum.step.esa.int/t/terrain-flattening-and-sar-simulation-incorrect-for-sentinel-1-images/35927 https://forum.step.esa.int/t/inferior-terrain-flattening-results-when-using-external-dem/37221

[piyushrpt]

Thank you for putting together that comparison plot - very helpful. This showed up in my feed and I have a couple of related observations / questions:

1. Sign of azimuth correction from ETAD might be reversed potentially when applied to s1ard and pyrosar?
2. If this is the source of the discrepancy, looking at the approximate ellipsoid centroids - we might still be left with an offset that is possibly consistent with ISCE3-OPERA ellipsoid.

Thanks again.

[abradley60]

Thanks for your observations and feedback @piyushrpt.

1. The ETAD was applied in a consistent way to the input SLC's prior to being passed to each software, so the correction directions should be the same for all softwares. I'm making use of the s1etad_tools package so @avalentino is probably best positioned to confirm they are applied as intended. I will also check my code to make sure.
2. Yes it looks if this is the case the SNAP based workflows will be shifted closer to the center.

[avalentino]

I did quite limited tests on the tool long time ago. Of course the easiest way to check is to measure the location of any PT in the SLCs before and after the ETAD correction. I can try to recover the results in my records but probably it is easy for you to do this kind of test.

[johntruckenbrodt]

Great to have you jump in as well @piyushrpt. So this is what is done in s1ard:

from s1etad_tools.cli.slc_correct import s1etad_slc_correct_main
s1etad_slc_correct_main(s1_product='<SLC path>',
                        etad_product=<ETAD product>,
                        outdir='.',
                        nthreads=2,
                        order=0)

The default order=1 introduced a bias of about -0.5 dB. Antonio and I had some mail exchange on this some while ago. Here some demo plot for a test site over the Alps:

[abradley60]

I can confirm I am applying the corrections in the same way as @johntruckenbrodt. Here is the ETAD file and where it is used for our pyroSAR implementation.

  s1etad_slc_correct_main(s1_product=slc_path,
                        etad_product=etad,
                        outdir=slc_corrected_dir,
                        nthreads=nthreads,
                        order=0)  # using the default 1 introduces a bias of about -0.5 dB.

Here are the links to all of our workflows for the project if you are interested. I'll also release the comparison code where we do the analysis/comparison of the products and make the figures I have sent through (I'm just in the process of tidying it up). Our workflows are basically wrappers around the existing projects that handle acquiring files and uploading our results in AWS. Thanks to the amazing work of folks like @johntruckenbrodt we have all this at out fingertips! The hype-GAMMA products we just order from the ASF.

Software	Source Code	Project Code
Pyrosar	https://github.com/johntruckenbrodt/pyroSAR	https://github.com/GeoscienceAustralia/sar-pyrosar
S1_NRB	https://github.com/SAR-ARD/S1_NRB	https://github.com/GeoscienceAustralia/sar-s1-nrb
Opera-RTC	https://github.com/opera-adt/RTC	https://github.com/GeoscienceAustralia/sar-rtc-opera
Hyp3-Gamma	https://pypi.org/project/asf-hyp3/	https://github.com/GeoscienceAustralia/sar-hyp3-gamma-asf

[piyushrpt]

I'm not very familiar with the s1etad_tools code base but couple of quick observations after looking at it briefly:

1. scipy map_coordinates is used for interpolation and the prefilter kwarg might need to be set/unset along with order for desired performance - if radiometric calibration is of interest. I would still expect this call to blur the imagery - understandable since this was a sandbox tool.

2. order=0 probably just means nearest neighbor interpolation and probably should not be used for geolocation evaluation. For evaluating geolocation, might be worth using higher order at the expense of radiometric accuracy.

3. If the sign flip is the source of error - i, e. replace += by -= to test. The line to change is

lines_grid += delta_t / (dt / klines) in slc_correct.py

[hfattahi]

Thank you for sharing your interesting analysis. I understand that this issue is closed and the topic has diverged from the original question. But I thought I could provide a bit more clarification with regard to ISCE3 that may be of interest to the team here:

1- In the ISCE3 based workflow we developed for OPERA project to process Sentinel-1 SLCs to RTC we did not use ETAD products directly. Instead we implemented ETAD algorithm and for RTC we only applied hydrostatic tropospheric delay and bistatic delay which are the major contributors. Other ETAD components such as Doppler induced range shift or wet tropospheric delay or even ionospheric delay leading to cm level geolocation accuracy do not matter for 30 m or even 10 m RTC products and therefore we ignored them for RTC. However, I doubt if that is a source of the discrepancy between ISCE3 and other software. But I thought that clarification is needed. I wonder if you guys produce the ISCE3 RTC in the same way or you have changed the workflow to use the ETAD products.

2- The RTC products are not ideal products for evaluating the geolocation. The reason is that for geolocation error analysis we need to find the peak of the CR precisely and for that we need to oversample the complex SAR signal. While oversampling can be done easily for band limited SAR complex signal, it can not be done precisely for low resolution multilooked SAR backscatter products such as RTC. Basically I'm trying to say that your peak estimation has a larger error bar when you estimate it from RTC images compared to estimating it from complex SLCs. That is why most of our validation of ETAD corrections were done with Geocoded SLCs and not with RTC. Despite RTC, our geocoded SLC products are produced with additional corrections such as ionospheric delay and Doppler induced range shift. The following plot shows the results of geolocation analysis over our Rosmaond CRs from 2014 to 2023. Each point is the average geolocation error in east and north direction for all CRs in a given date and the bars show the standard deviation for that date. It is worth noting that most of the improvement comes from hydrostatic tropospheric delay in range and bistatic delay in azimuth directions. Ionosphere can contribute 10s of cm.

Our results seem to show consistent improvement in both range and azimuth directions after ETAD corrections.

NOTE: in this plot the X and Y axis are indeed east and north directions as the SLCs are geocoded.

[abradley60]

Hi @hfattahi thank you for joining the discussion, it's great to have your input here as well - your feedback is very valuable to us. Similar to John, we were planned on circulating our findings to yourself and Gustavo at JPL.

1. Yes I can confirm we had changed the workflow to enable the ETAD corrections. We are aware of the native OPERA-RTC atmospheric correction options for apply_bistatic_delay_correction and apply_static_tropospheric_delay_correction that you have mentioned. We had run three versions of the workflow in our tests; with these set to False and no ETAD correction, with these set to False and applying ETAD correction, and with these set to True and no ETAD correction. These represent the OPERA-ISCE3, OPERA-ISCE3 (with ETAD) and OPERA-ISCE3 (native atmospheric) scenarios in the figure below respectively. Our other testing has shown the importance of enabling these corrections on the RTC quality.

2. Yes this is true and we expect the estimation from oversampling pixels at 10m would introduce some of the additional variability and error when assessing geolocation with the RTC products. I have attached an example of this below for a CR in the Surat Basin. This is a very important call out that we will be sure to include with our analysis. Your teams analysis is far more robust for evaluating the corrections. For our purposes, we are not trying to assess the ‘true’ geolocation accuracy, but only as it applies to the RTC product, as that is what we will be delivering for people to use.

[avalentino]

@abradley60 do you mind if I convert this issue into a discussion and change the title?

[abradley60]

@avalentino yes please go ahead