SAOCOM L1A products are already calibrated

[impresionista]

I'm trying to read SAOCOM data without success.

Here is a minimal example to reproduce my error:

import os
from pathlib import Path

from eoreader.reader import Reader
from eoreader.bands import *

data_dir = os.path.abspath(os.path.join("..", "..", "data", "raw", "saocom"))
saocom_file = "S1A_OPER_SAR_EOSSP__CORE_L1A_OLVF_20220317T145008"

saocom_prod = Reader().open(data_dir, saocom_file)

print(saocom_prod)

eoreader.SaocomProduct 'S1A_OPER_SAR_EOSSP__CORE_L1A_OLVF_20220317T143007' Attributes: condensed_name: 20200801T103425_SAOCOM_VV_HH_VH_HV_SM_L1A path: /root/data/raw/saocom constellation: SAOCOM-1 sensor type: SAR product type: L1A default resolution: 10.0 acquisition datetime: 2020-08-01T10:34:25.550874 band mapping: VV: VV VV_DSPK: VV_DSPK HH: HH HH_DSPK: HH_DSPK VH: VH VH_DSPK: VH_DSPK HV: HV HV_DSPK: HV_DSPK needs extraction: True orbit direction: ASCENDING

It appears to correctly read the data/metadata, but it fails at getting some of the bands:

ok_bands = saocom_prod.get_existing_bands()
saocom_prod.load(bands=[VV])

Here is the traceback of the error: https://pastebin.com/SwL9D9Ju

I need help to load the data as an xarray.

Thanks!

[remi-braun]

Hello ! Thanks for the feedback I am currently in holidays so I will try to help you without access to my computer 😅

It seems this is a SNAP issue so I will propose a workaround in the meantime: could you modify the SAR Preprocess XML and remove the calibration step? You can add this custom GPT Graph following the instructions on the SAR page of the documentation

Hope this will work...

I will investiguate further when I can 😉

[remi-braun]

But Absolute radiometric calibration has already been applied to the product isn't strange for L1A data ? Do you know if SAOCOM has changed its product specifications ?

[impresionista]

But Absolute radiometric calibration has already been applied to the product isn't strange for L1A data ? Do you know if SAOCOM has changed its product specifications ?

I've confirmation that SAOCOM L1A products are calibrated.

[remi-braun]

Ok so that's the issue, you need to remove the calibration step from the SNAP GPT XML

I will make the workaround as soon as I'm back, around the end of august 😉

[remi-braun]

The other issue is Why the SAOCOM product I'm using passes the tests 🤔

[impresionista]

Thanks! That'll be great! Let me know if I can help you in any way.

[impresionista]

The other issue is Why the SAOCOM product I'm using passes the tests thinking

What products are you using? I'm working inside docker, so i'm trying to understand if SNAP is not installed correctly inside it. The error message wasn't enough for me to discern if there was an issue with my SNAP/GPT config or with the SAOCOM data.

Snap its installed using default configs:

RUN wget --no-verbose https://download.esa.int/step/snap/8.0/installers/esa-snap_all_unix_8_0.sh &&\
    chmod +x ./esa-snap_all_unix_8_0.sh &&\
    ./esa-snap_all_unix_8_0.sh -q &&\
    rm ./esa-snap_all_unix_8_0.sh

ENV PATH="/usr/local/snap/bin:$PATH"

Edit: Just in case, I've executed in another cell (i'm working in a jupyter notebook)

!echo $PATH

/usr/local/snap/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

It appears to be ok. And:

!gpt

Returns gpt execution:

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Incompatible GDAL 3.5.1 found on system. Internal GDAL 3.0.0 from distribution will be used. INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP. INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP. Usage: gpt | [options] [ ...]

Description: This tool is used to execute SNAP raster data operators in batch-mode. The operators can be used stand-alone or combined as a directed acyclic graph (DAG). Processing graphs are represented using XML. More info about processing graphs, the operator API, and the graph XML format can be found in the SNAP documentation.

...

[remi-braun]

You can now use SNAP 9.0 😉 And if you are using the 8th version it is best to install all the updates if you can

I also am using everything in a docker, I could help you if needed

I am using the 4 polarisation product you can download on their website (among others)

[impresionista]

Well, I have news.

1. I've tried using a basic gpt graph without any preprocessing filter, only read an write, with relative success. I got a new error regarding missing CRS.

<graph id="Graph">
  <version>1.0</version>

  <node id="Read">
    <operator>Read</operator>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>$file</file>
    </parameters>
  </node>

  <node id="Write">
    <operator>Write</operator>
    <sources>
      <sourceProduct refid="Read"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>$out</file>
      <formatName>BEAM-DIMAP</formatName>
    </parameters>
 </node>

</graph>

Python code:

saocom_vv = saocom_prod.load(bands=[VV])

Error message:

MissingCRS: CRS not found. Please set the CRS with 'rio.write_crs()'. Data variable: 20200614T103426_SAOCOM_VV_HH_VH_HV_SM_L1A_VV

---

2. I've tried adding a "Terrain-Correction" step to the preprocessing graph. The new error was about a division by cero.

<graph id="Graph">
  <version>1.0</version>

  <node id="Read">
    <operator>Read</operator>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>$file</file>
    </parameters>
  </node>

  <node id="Terrain-Correction">
    <operator>Terrain-Correction</operator>
    <sources>
      <sourceProduct refid="Read"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <sourceBands/>
      <demName>SRTM 3Sec</demName>
      <externalDEMFile/>
      <externalDEMNoDataValue>0.0</externalDEMNoDataValue>
      <externalDEMApplyEGM>true</externalDEMApplyEGM>
      <demResamplingMethod>BILINEAR_INTERPOLATION</demResamplingMethod>
      <imgResamplingMethod>BILINEAR_INTERPOLATION</imgResamplingMethod>
      <pixelSpacingInMeter>8.46</pixelSpacingInMeter>
      <pixelSpacingInDegree>7.599747303651153E-5</pixelSpacingInDegree>
      <mapProjection>GEOGCS[&quot;WGS84(DD)&quot;,
        DATUM[&quot;WGS84&quot;,
          SPHEROID[&quot;WGS84&quot;, 6378137.0, 298.257223563]],
        PRIMEM[&quot;Greenwich&quot;, 0.0],
        UNIT[&quot;degree&quot;, 0.017453292519943295],
        AXIS[&quot;Geodetic longitude&quot;, EAST],
        AXIS[&quot;Geodetic latitude&quot;, NORTH]]</mapProjection>
      <alignToStandardGrid>false</alignToStandardGrid>
      <standardGridOriginX>0.0</standardGridOriginX>
      <standardGridOriginY>0.0</standardGridOriginY>
      <nodataValueAtSea>true</nodataValueAtSea>
      <saveDEM>false</saveDEM>
      <saveLatLon>false</saveLatLon>
      <saveIncidenceAngleFromEllipsoid>false</saveIncidenceAngleFromEllipsoid>
      <saveLocalIncidenceAngle>false</saveLocalIncidenceAngle>
      <saveProjectedLocalIncidenceAngle>false</saveProjectedLocalIncidenceAngle>
      <saveSelectedSourceBand>true</saveSelectedSourceBand>
      <saveLayoverShadowMask>false</saveLayoverShadowMask>
      <outputComplex>false</outputComplex>
      <applyRadiometricNormalization>false</applyRadiometricNormalization>
      <saveSigmaNought>false</saveSigmaNought>
      <saveGammaNought>false</saveGammaNought>
      <saveBetaNought>false</saveBetaNought>
      <incidenceAngleForSigma0>Use projected local incidence angle from DEM</incidenceAngleForSigma0>
      <incidenceAngleForGamma0>Use projected local incidence angle from DEM</incidenceAngleForGamma0>
      <auxFile>Latest Auxiliary File</auxFile>
      <externalAuxFile/>
    </parameters>
  </node>

  <node id="Write">
    <operator>Write</operator>
    <sources>
      <sourceProduct refid="Terrain-Correction"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>$out</file>
      <formatName>BEAM-DIMAP</formatName>
    </parameters>
 </node>

</graph>

Python code:

saocom_vv = saocom_prod.load(bands=[VV])

Error message:

Division by zero https://pastebin.com/tQz3zeVy

---

3. I've tried using a multilook filter with and without "Terrain-correction" with no success, the errors were the same as 1. and 2.

[remi-braun]

For L1A you must set the Terrain Correction step as EOReader only reads orthorectified data in UTM

Could you use the graph without calibration that is already existing: eoreader/data/cplx_no_calib_preprocess_default.xml

Did you add the updates to your SNAP 8 ?

[impresionista]

Great! I'll try eoreader/data/cplx_no_calib_preprocess_default.xml graph.

Did you add the updates to your SNAP 8 ?

I updated to SNAP 9.

[impresionista]

With eoreader/data/cplx_no_calib_preprocess_default.xml seems to be working perfectly!

[remi-braun]

OK so happy to hear that! Could you try with the default graph just to see if it was SNAP version (without updates) that was the issue ?

[impresionista]

Yes, it seems that using SNAP 9 solved the issue.

I'll check with the SAR specialist in my team to find out if its working correctly, just in case there is something miss-configured.

[remi-braun]

I think that the updates of SNAP made the calibration step fail safe in some sort

I will add that in the documentation !

[impresionista]

I've checked with the team and it seams that the default filter is working correctly. I'll close this issue. Thanks for your help! :)

[remi-braun]

Thanks for the effort you put in helping me resolving this 😉 (I'm glad it was a SNAP issue 😅)

[JaumePrimer]

Hi, just jumping on to this issue, albeit closed.

I've had similar problems with SNAP 8.0 and SAOCOM. I'm using docker and all signs point to the problem being SNAP 8.0. I can't find any examples of a dockerfile for SNAP 9.0. Does an open source one exist?

[remi-braun]

Hello,

Here are the lines concerning the SNAP installation on my dockerfile.

# Install SNAP
# - download snap installer version 8.0 + TBX
# - change file execution rights for snap installer
# - install snap with gpt
# - link gpt so it can be used systemwide
ARG snap_version=9
ARG snap_update=snap_update.sh
ARG snap_script=esa-snap_sentinel_unix_"$snap_version"_0_0.sh
RUN wget http://step.esa.int/downloads/"$snap_version".0/installers/"$snap_script" \
&& chmod +x "$snap_script" \
&& ./"$snap_script" -q \
&& ln -s /usr/local/snap/bin/gpt /usr/bin/gpt \
# Update SNAP
&& chmod +x "$eeo_root""$snap_update"  \
&& "$eeo_root""$snap_update" \
# Add to line to GPT's VMoptions -> Bug (https://forum.step.esa.int/t/xmlfactory-error-using-snap-8/26566)
&& echo "-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl" >> /usr/local/snap/bin/gpt.vmoptions \
# Remove heavy script
&& rm "$snap_script"

The update script is:

#!/bin/bash

snap --nosplash --nogui --modules --update-all 2>&1 | while read -r line; do
    echo "$line"
    [ "$line" = "updates=0" ] && sleep 2 && pkill -TERM -f "snap/jre/bin/java"
done

Basically the difference between SNAP 8.0 and 9.0 is very thin, the 8.0 script is called esa-snap_sentinel_unix_8_0.sh when the one for 9.0 is esa-snap_sentinel_unix_9_0_0.sh

[JaumePrimer]

Thanks @remi-braun , that's a big help! I have most of the image building but running snap --nosplash --nogui --modules --update-all 2>&1 keeps on exiting with error code 33. Looking around, it sounds like it might be a problem with the server being busy. I've retried at different times of the day but the error is persisting. Is this a problem you've had? Am I just unlucky with my timing?

[remi-braun]

With SNAP 8.0 or 9.0 ? I've switched to 9.0 and I just reconstruted an image which worked just fine 😅

[JaumePrimer]

Working with SNAP 9.0.

I've just found the problem. Well, almost. I'm using a Mac M1 processor and couldn't install SNAP 9.0 in docker due to architectural differences. I instead used google cloud builds instead of building locally with docker as this way it will build on a Linux machine. This worked fine for the install, but failed on the update.

I ended up splitting the build in two:

1. install SNAP with google cloud build (on a Linux)
2. using the output from (1) as the base, update SNAP using docker build and a second Dockerfile locally on my Mac

I can't explain why the update didn't work on google, and the error codes gave no information. But in the end, it worked...

Thanks for the help, @remi-braun !

[remi-braun]

Happy to here that! For now, it seems Mac M1 are still hard to work with

FYI: EOReader is currently not tested on Macs, so if you encounter any error outside your Docker I may not be able to correct them 😅