Error reading Tandem-X - Githubissues

StephanLo commented 1 year ago

which operating system are you using?
Windows 10,
which environment is pyroSAR running in?
conda environment (with python=3.11.5)
which version of pyroSAR are you using?
one installed via conda (pyrosar=0.22.0)
which function of pyroSAR did you call with which parameters?
if applicable, which version of SNAP or GAMMA are you using in pyroSAR? SNAP v9.0

Maybe I am doing something wrong, but If I try to read tandem-x data it fails with error message "RuntimeError: data format not supported"

This happens if

I supply tandemx xml file (which I usually use for reading tandem-x into snap) to pyrosar identify
If I execute gpt graph that starts with reading tandem-x xml file, using for example:
```
from pyroSAR.snap.auxil import gpt
```

gpt(Path(data_dir, 'tandemx_to_snaphu.xml'), tmpdir= Path(data_dir, 'tmp'))

3. I also tried other ways of reading tandem-x scene, like supplying the parent directory name to 'identify', but all gave same error.
However, the same gpt graph that reads a tandem-x xml file in the first node executes correctly when called directly with gpt in the command line.

Thanks for the help! Do you need my graph's or other info?
[Update:] the Directory structure of the Tandem-x CoSSC data looks like this tree below. The file used to load is the 
*TDM1_SAR__COS_BIST_SM_S_SRA_20140311T212600_20140311T212608.xml* file in the parent folder.

E:. │ TDM1_SARCOS_BIST_SM_S_SRA_20140311T212600_20140311T212608.xml │ ├───COMMON_ANNOTATION │ image_layer_info.txt │ spectral_filter_frequencies.xml │ ├───COMMON_AUXRASTER │ coreg_type.ras │ dInSAROut_rgr.flt │ GradientsMask.ras │ qc_mm_shift_az.ras │ qc_mm_shift_rg.ras │ slave_shift_az.flt │ slave_shift_az_geo.flt │ slave_shift_rg.flt │ slave_shift_rg_geo.flt │ ├───COMMON_PREVIEW │ QL_GTC_amplitude.tif │ QL_GTC_amplitude.tif.kml │ QL_GTC_coherence.tif │ QL_GTC_coherence.tif.kml │ QL_GTC_DEM.tif │ QL_GTC_DEM.tif.kml │ QL_GTC_DEM_16bit.tif │ QL_GTC_DEM_16bit.tif.kml │ QL_SLT_ampl_composite.tif │ QL_SLT_coher.tif │ QL_SLT_dinsar_phase.tif │ QL_SLT_dinsar_radargr.tif │ QL_SLT_phase.tif │ QL_SLT_QM.tif │ ├───TDX1_SAR__SSC_BTX1_SM_S_SRA_20140311T212600_20140311T212608 │ │ TDX1_SARSSC_BTX1_SM_S_SRA_20140311T212600_20140311T212608.xml │ │ │ ├───ANNOTATION │ │ GEOREF.xml │ │ RFANTPAT_PHASE_HH_SRA_tanDEM_a2_075.xml │ │ │ ├───AUXRASTER │ │ CAL_QL_MRES_HH.tif │ │ CAL_QL_MRES_HH.tif.kml │ │ stdcalcomposite_MRES.tif │ │ stdcalcomposite_MRES.tif.kml │ │ STD_MRES_HH.tif │ │ STD_MRES_HH.tif.kml │ │ │ ├───IMAGEDATA │ │ IMAGE_HH_SRA_tanDEM_a2_075.cos │ │ │ ├───PREVIEW │ │ BROWSE.tif │ │ COMPOSITE_QL.tif │ │ MAP_PLOT.png │ │ QL_HH_SRA_tanDEM_a2_075.tif │ │ │ └───SUPPORT │ antennaPhasePattern.xsd │ GEARTH_POLY.kml │ generalHeader.xsd │ geoReference.xsd │ level1Product.xsd │ PGS_Types.xsd │ README.txt │ types_inc.xsd │ └───TSX1_SAR__SSC_BRX2_SM_S_SRA_20140311T212600_20140311T212608 │ TSX1_SAR__SSC_BRX2_SM_S_SRA_20140311T212600_20140311T212608.xml │ ├───ANNOTATION │ GEOREF.xml │ RFANTPAT_PHASE_HH_SRA_tanDEM_a2_075.xml │ ├───AUXRASTER │ CAL_QL_MRES_HH.tif │ CAL_QL_MRES_HH.tif.kml │ stdcalcomposite_MRES.tif │ stdcalcomposite_MRES.tif.kml │ STD_MRES_HH.tif │ STD_MRES_HH.tif.kml │ ├───IMAGEDATA │ IMAGE_HH_SRA_tanDEM_a2_075.cos │ ├───PREVIEW │ BROWSE.tif │ COMPOSITE_QL.tif │ MAP_PLOT.png │ QL_HH_SRA_tanDEM_a2_075.tif │ └───SUPPORT antennaPhasePattern.xsd GEARTH_POLY.kml generalHeader.xsd geoReference.xsd level1Product.xsd PGS_Types.xsd README.txt types_inc.xsd


- the full error message

RuntimeError Traceback (most recent call last) c:\path\tandem_X.qmd in line 1 ----> 27 scene = identify(name)

File c:\Users\stephan\mambaforge\envs\for_snap\Lib\site-packages\pyroSAR\drivers.py:125, in identify(scene) 123 except (RuntimeError, KeyError, AttributeError): 124 pass --> 125 raise RuntimeError('data format not supported')

RuntimeError: data format not supported

StephanLo commented 1 year ago

(I don't mean this is a bug necessarily, it is probably just me not understanding how to open the file correctly, it seems the bug label was added automatically and I can't see how to remove it. Sorry!)

johntruckenbrodt commented 1 year ago

Dear @StephanLo, thanks for reaching out. In principle Tandem-x CoSSC should be supported. The driver is TDX. This is a community contribution and I have never tried the driver iteself.

For reading a product you need to provide the top folder of the product instead of the XML file. Could you try importing this class directly and try to identify the top folder of your product?

from pyroSAR.drivers import TDX
scene = TDX('top product folder')

If this does not work, what is the error message?

StephanLo commented 1 year ago

Dear @johntruckenbrodt Thanks for your response

It worked thank you! Expect that I think it is maybe TDM instead of TDX, as in documentation

from pyroSAR.drivers import TDM
scene = TDM(Path(data_dir, name).as_posix())
print(scene)

Loads the product properly:

pyroSAR ID object of type TDM
acquisition_mode: SM
cycleNumber: 430
frameNumber: 2673131
lines: 28234
orbit: A
orbitNumber_abs: 71776
orbitNumber_rel: 133
polarizations: ['HH']
product: COSSC
projection: +proj=longlat +datum=WGS84 +no_defs
samples: 17438
sensor: TDM1
spacing: (0.00028334792733302643, 9.10033168464242e-09)
start: 20200524T212643
stop: 20200524T212651

However, I'm still struggling, because identify doesn't seem to work, despite the same parent product directory being supplied.

from pathlib import Path
from pyroSAR import identify
data_dir = Path("E:/stephan/files/Landfill/dem_timeseries/vanZyl_sites")

name = "TDM1_SAR__COS_BIST_SM_S_SRA_20200524T212643_20200524T212651"
identify(Path(data_dir, name).as_posix())

gives error

File [c:\Users\stephan\mambaforge\envs\for_snap\Lib\site-packages\pyroSAR\drivers.py:125](file:///C:/Users/stephan/mambaforge/envs/for_snap/Lib/site-packages/pyroSAR/drivers.py:125), in identify(scene)
    [123](file:///c%3A/Users/stephan/mambaforge/envs/for_snap/Lib/site-packages/pyroSAR/drivers.py?line=122)     except (RuntimeError, KeyError, AttributeError):
    [124](file:///c%3A/Users/stephan/mambaforge/envs/for_snap/Lib/site-packages/pyroSAR/drivers.py?line=123)         pass
--> [125](file:///c%3A/Users/stephan/mambaforge/envs/for_snap/Lib/site-packages/pyroSAR/drivers.py?line=124) raise RuntimeError('data format not supported')

RuntimeError: data format not supported

And, it seems the gpt command from pyroSAR.snap.auxil calls the identify function internally, so gpt then fails for the TDM CoSSC data, because identify returns the same 'data not supported' error

my_graph = "[...]/graphs/pyro_tandemx_to_snaphu.xml"

from pyroSAR.snap.auxil import gpt
gpt(my_graph, tmpdir= Path(data_dir, 'tmp'))

Error message:

from pyroSAR.snap.auxil import gpt
----> [158](file:///c%3A/Users/stephan/hokudai/PhD/Landfill/Code/SAR_idea/python/snap/tandem_X.qmd?line=157) gpt(my_graph, tmpdir= Path(data_dir, 'tmp'))

File [c:\Users\stephan\mambaforge\envs\for_snap\Lib\site-packages\pyroSAR\snap\auxil.py:363](file:///C:/Users/stephan/mambaforge/envs/for_snap/Lib/site-packages/pyroSAR/snap/auxil.py:363), in gpt(xmlfile, tmpdir, groups, cleanup, gpt_exceptions, gpt_args, removeS1BorderNoiseMethod)
    [361](file:///c%3A/Users/stephan/mambaforge/envs/for_snap/Lib/site-packages/pyroSAR/snap/auxil.py?line=360) else:
    [362](file:///c%3A/Users/stephan/mambaforge/envs/for_snap/Lib/site-packages/pyroSAR/snap/auxil.py?line=361)     read = workflow['Read']
--> [363](file:///c%3A/Users/stephan/mambaforge/envs/for_snap/Lib/site-packages/pyroSAR/snap/auxil.py?line=362)     scene = identify(read.parameters['file'])
    [365](file:///c%3A/Users/stephan/mambaforge/envs/for_snap/Lib/site-packages/pyroSAR/snap/auxil.py?line=364) tmpdir_bnr = os.path.join(tmpdir, 'bnr')
    [366](file:///c%3A/Users/stephan/mambaforge/envs/for_snap/Lib/site-packages/pyroSAR/snap/auxil.py?line=365) tmpdir_sub = os.path.join(tmpdir, 'sub')

File [c:\Users\stephan\mambaforge\envs\for_snap\Lib\site-packages\pyroSAR\drivers.py:125](file:///C:/Users/stephan/mambaforge/envs/for_snap/Lib/site-packages/pyroSAR/drivers.py:125), in identify(scene)
    [123](file:///c%3A/Users/stephan/mambaforge/envs/for_snap/Lib/site-packages/pyroSAR/drivers.py?line=122)     except (RuntimeError, KeyError, AttributeError):
    [124](file:///c%3A/Users/stephan/mambaforge/envs/for_snap/Lib/site-packages/pyroSAR/drivers.py?line=123)         pass
--> [125](file:///c%3A/Users/stephan/mambaforge/envs/for_snap/Lib/site-packages/pyroSAR/drivers.py?line=124) raise RuntimeError('data format not supported')

RuntimeError: data format not supported

Do you have any advice on this? Thank you for the assistance! kindly Stephan

StephanLo commented 1 year ago

Just another comment that I think might be of interest,

I think the problem is maybe specific to the TDM driver for CoSSC data. Because, identify works for TSX products

Basically, In the CoSSC product folder there are two sub-directories (for each of the two co-registered scenes) And, if I supply one of the sub-directories to identify, it succeeds, and sees the loaded product as a TSX product. (the same sub-directory can also be read directly with the TSX driver)

from pathlib import Path
from pyroSAR import identify
data_dir = Path("E:/stephan/files/Landfill/dem_timeseries/vanZyl_sites")

# rio 2020
name = "TDM1_SAR__COS_BIST_SM_S_SRA_20200524T212643_20200524T212651"
# The sub directory for one of the coregistered scenes
sub_name = "TDX1_SAR__SSC_BTX1_SM_S_SRA_20200524T212643_20200524T212651"

scene = identify(Path(data_dir, name, sub_name).as_posix())

print(scene)

Output:

pyroSAR ID object of type TSX
acquisition_mode: SM
cycleNumber: 430
frameNumber: 2673131
lines: 28234
orbit: A
orbitNumber_abs: 71776
orbitNumber_rel: 133
polarizations: ['HH']
product: SSC
projection: +proj=longlat +datum=WGS84 +no_defs
samples: 17438
sensor: TDX1
spacing: (0.00028334792733302643, 9.10033168464242e-09)
start: 20200524T212643
stop: 20200524T212651

johntruckenbrodt commented 1 year ago

Dear @StephanLo. I have found the error and will merge the fix in a minute. Thanks again.

StephanLo commented 1 year ago

Thank you very much!

johntruckenbrodt / pyroSAR

Error reading Tandem-X #263