TerraSAR-X COSSC images are not read correctly

[lsma604]

When importing XML files (linked to image data of .cos format) with the TSXReader, I am finding that the resulting complex data is not correct. The image size is correct, and there is some structure to the resulting intensity image, however it is very noisy. So to me it sounds like there is a data format issue (e.g. int16 being read as a float) or endianness (big endian int16 being read as little endian int16) at fault.

Below is a histogram of the I data that is read in using the sarpy library. Majority of the values are within two peaks, far away from 0. This again suggests to me that either incorrect data format casting or endianness is perhaps at fault.

And here is an example of the complex64 IQ data that sarpy has read from the TSXReader:

And here is what the complex data should look like, from the histogram in SNAP (the only viewer that I have that successfully reads the data). Majority of the IQ data is within -300 to +300, as it should be.

I also found that this issue was also encountered here: https://forum.step.esa.int/t/the-problem-in-reading-terrasar-x-slc-complex-data-with-snap-and-next/10605

[thomasmccullough]

Is it possible to provide a reference to the image used here? That would be very helpful to diagnose what's going on.

[lsma604]

Thanks for the reply. I have tried to find another example from available online datasets, unfortunately data such as this one and this one work fine and do not replicate my issue.

My only guess is that it is down to the data I am working with being from the bistatic interferometric product (COSSC), which contains two SAR images within it (TerraSAR-X and TanDEM-X captures, each with their own XML and .cos files). Both .cos files open fine in SNAP but do not read properly in sarpy. I have also tried opening the .cos files using the Orfeo Toolbox viewer (based on GDAL) which produces the same issue as detailed above. Note, I am NOT trying to read the interferometric data, just one component of it as if it were a single capture.

Unfortunately I am not aware of any easily available COSSC online datasets as a reference, I will try and look.

[thomasmccullough]

The intent in sarpy is not really to fully support any kind of terrasar product at all, but rather the product that is directly analogous to a SICD product. It's likely that your product is not what we're expecting for sarpy to work properly, but I'm not catching the difference. I think that SNAP evidently has much more complete support.

sarpy will only open a level 1 product, and possibly some subset of those. I did a quick check of the code, and I'm just checking that the root tag of the main xml file is level1Product, which is probably not quite enough of a check.

Would you open the main xml file (probably a browser like chrome is tidiest). The root node should be level1Product, or sarpy would have failed to open things. Can you see what the value of the itemName tag is (just a few spots down from the root tag). If it's not "Level 1B Product", then that's likely the problem.

Thanks.

[lsma604]

Here is the general header, I have just redacted the date/times. The Corresponding TDX1 XML file has the same general header, just with a slightly altered filename (TDX1_SAR_SSCBRX2...) Hopefully this is enough of a hint to let you know what could be going wrong, in particular the reference document.

<level1Product xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="SUPPORT/level1Product.xsd">
  <generalHeader fileName="TSX1_SAR__SSC_BTX1_SM_Q_DRA_2015XXXXTXXXXXX_2015XXXXTXXXXXX.xml" fileVersion="1.0" status="PRELIMINARY">
    <itemName>Level 1B Product</itemName>
    <mission>TSX-1</mission>
    <source>GS</source>
    <destination>User</destination>
    <generationSystem version="2.2">ITP</generationSystem>
    <generationTime>2017-XX-XXTXX:XX:XX.000000</generationTime>
    <referenceDocument>TX-GS-DD-3302 Basic Product Specification 1.5</referenceDocument>
    <revision/>
    <revisionComment/>
    <remark/>
  </generalHeader>

[thomasmccullough]

Unfortunately that looks generally the same as ones that appear to work fine. The only difference is that this generation system version is slightly behind what I have, even though this file is something like 8 years newer. I wouldn't think they would modify the COS file standard, but don't know - I'm at a bit of a loss.

I'll leave this for now, until the person with more specific knowledge can return from vacation and weigh in.

[utwade]

The difference between the regular COSAR and the bistatic COSSC is that the COSAR is in 16-bit integer and the COSSC is 16-bit "half precision" float. Details are here: https://tandemx-science.dlr.de/pdfs/TD-GS-PS-3028_TanDEM-X-Experimental-Product-Description_1.2.pdf

The key is the 9th word of the COSAR/COSSC denoting the version. COSAR will be set to 1 and COSSC will be set to 2.

The code current assumes version 1 for all cases, which is what results in the error.

Easiest solution is just to have the reader error out if it finds a COSAR/COSSC that is anything other than 1, since nothing else in the toolbox is setup to properly handle the metadata for that type of data, even if the binary data was read properly.

Someone could also try to do a full ingest, but that's a potentially much larger effort.

[lsma604]

Thank you @utwade, this would explain the artifacts I can see in the resulting image. I will leave it up to @thomasmccullough whether to close this issue or not as I do not have the expertise to make a pull request for this repo. For now I will just be bit-converting the binary output manually to half-precision float such that I can process the image.

[thomasmccullough]

This should be extremely easy to support. I should be able to get a fix in tomorrow, or early next week at the latest.

[piyushrpt]

Just a note that like @utwade pointed out - only the active sensor's imagery can currently be represented correctly as this is the same as any other TSX SLC. The data from the passive sensor is in a bistatic zero doppler system and already resampled on to the grid of the active sensor. I suspect that the SICD spec may not currently support this.

[thomasmccullough]

Thanks for pointing that out. I have not ever seen the full package for one of these. If I understand what you are saying correctly, then that issue is beyond the scope of sarpy. I suggest this is an issue of interpretation by the user, and the only thing that I am aiming to correct is interpreting the .cos file contents properly as a half precision float versus a 16-bit integer.

[lsma604]

Thanks all, reading in the active (BTX1) channel's corresponding .cos file with the correct datatype would solve my issue, I do not expect nor require SarPy to support bistatic interferometric data. Of note though, reading the passive (BRX2) channel by itself will likely work just fine with the same fix.

[lsma604]

Thank you. The COSARv2 SAR image I have reads perfectly fine with TSXReader in v1.3.7. Both the primary (BTX1) and secondary (BRX2) channels read well as individual .cos files independently.