`ESA burst map, burst database` != `Sentinel1BurstSlc.burst_id`

[seongsujeong]

Checked for duplicates

Yes - I've already checked

Describe the bug

It was discovered from the RTC result that some of the burst products has a cutoff. The issue took place especially around the coastal area of Brazil, Track number 126 in descending orbit. The QGIS screenshot below describes one of the examples.

The yellow box is the coverage for T126-269605-IW3 based on ESA burst map (and same for burst DB which is based on the burst map). The burst RTC products, which was for T126-269605-IW3, is right next to the yellow polygon. Based on the burst map, the RTC product above is covering the area for T126-269604-IW3. So I suspect that, for some reason, T126-269604-IW3 was recognized as T126-269605-IW3, and the processing has pulled the bounding box info for T126-269605-IW3.

What did you expect?

I think we need to solve the discrepancy in the burst map / database, and Sentinel1BurstSlc.burst_id

Reproducible steps

More information about the example data shown above:
SAFE name: S1A_IW_SLC__1SDV_20210416T083427_20210416T083454_037473_046B05_5F0C
Burst ID: `t126-269605-IW3`

Environment

- Version of this software [e.g. vX.Y.Z]
- Operating System: [e.g. MacOSX with Docker Desktop vX.Y]
...

[seongsujeong]

More information that I've figured out so far

• Processed the data in 2019 and 2023. The bursts look okay in those cases:
  • S1A_IW_SLC__1SDV_20230112T083427_20230112T083454_046748_059ABA_08CC.zip
  • S1A_IW_SLC__1SDV_20190708T083419_20190708T083446_028023_032A31_3850.zip,
  • S1A_IW_SLC__1SDV_20190708T083354_20190708T083421_028023_032A31_5F44.zip

Browse image of t126_269605_iw3 in 2019

Browse image of t126_269605_iw3 in 2021

Browse image of t126_269605_iw3 in 2023

[hfattahi]

We know that only few frames of T126 starting from equator on the descending track have issues and the rest of frames/bursts on the same track are fine. I wonder if the ascending node time is correct for those bursts with issues. The best way to check would be to download the zip file (or easier only the metadata) of few S1 frames on track 126 on the 2021 date of interest, over that region where the issue starts and covering parts of the track that does not show the issue. Then extract the ascending node time for all those frames to see if they are identical or not.

[scottstanie]

Hm i'm not reproducing the same error with COMPASS. Is it possible that the RTC code makes some assumption about which bursts will be present in a file? As in, do you assume that every zip file has "IW1, IW2, IW3" in even numbers?

Here I tried bursts t126_269603_iw3 t126_269604_iw3, and t126_269605_iw3 for 2019, 2021, and 2023. Everything looks fine from COMPASS outputs.

Full reproduction:

$ sardem --data COP --bbox -44.95 -6.3 -43.8 -5.0 -o dem.tif --output-format GTiff --output-type float32 &
$  wget "https://datapool.asf.alaska.edu/SLC/SA/S1A_IW_SLC__1SDV_20230112T083427_20230112T083454_046748_059ABA_08CC.zip" &
$ wget "https://datapool.asf.alaska.edu/SLC/SA/S1A_IW_SLC__1SDV_20210416T083427_20210416T083454_037473_046B05_5F0C.zip" &
$ wget "https://datapool.asf.alaska.edu/SLC/SA/S1A_IW_SLC__1SDV_20190708T083419_20190708T083446_028023_032A31_3850.zip"
$ s1_geocode_stack.py -nc -b t126_269603_iw3 t126_269604_iw3 t126_269605_iw3 -d dem.tif -s .
$ OMP_NUM_THREADS=8 find stack/ -name "*.sh"|xargs -P3 -n1 bash

then in jupyter

from PIL import Image
from pathlib import Path
import matplotlib.pyplot as plt

test_browse = sorted(Path("/home/staniewi/repos/s1-reader/stack").glob("t126_26960*/*/*png"))
print(f"{len(test_browse)} slc files")

fig, axes = plt.subplots(ncols=3, nrows=3)
axes = axes.ravel()
for f, ax in zip(test_browse, axes):
    ax.imshow(Image.open(test_browse[0]))
    ax.set_title(f.name)

[scottstanie]

Well actually it might be showing normal browse images, but the burst location is still off.... Was t126_269605_iw3 the only one that was incorrect in 2021? Or was the whole zip file shifted?

using s1_info --burst-bbox -i3 <filename>, it looks like the bursts are moving along correctly, but that 2021 has all the bursts shifted from the other years:

if we look at just t126_269605_iw3:

$ s1_info --burst-bbox -i3  S1A_IW_SLC__1SDV_20230112T083427_20230112T083454_046748_059ABA_08CC.SAFE/ | grep t126_269605_iw3
Found 1 Sentinel-1 SLC products.
Sentinel1BurstSlc: t126_269605_iw3 at 2023-01-12 08:34:50.454057 [-44.90403209173591, -6.128244693257249, -44.14033497451911, -5.779967417189208]

$ s1_info --burst-bbox -i3  S1A_IW_SLC__1SDV_20210416T083427_20210416T083454_037473_046B05_5F0C.SAFE/ | grep t126_269605_iw3
Found 1 Sentinel-1 SLC products.
Sentinel1BurstSlc: t126_269605_iw3 at 2021-04-16 08:34:36.437052 [-44.86408401066785, -5.941103955727937, -44.10456434344419, -5.612647386664313]

$ s1_info --burst-bbox -i3  S1A_IW_SLC__1SDV_20190708T083419_20190708T083446_028023_032A31_3850.SAFE | grep t126_269605_iw3
Found 1 Sentinel-1 SLC products.
Sentinel1BurstSlc: t126_269605_iw3 at 2019-07-08 08:34:31.136414 [-44.90045512046944, -6.107951510687787, -44.14094828299857, -5.779505047967723]

notice the bottom latitude (second number in the bboxes) goes from -6.13, to -5.94, back to -6.11

and, showing only the common bursts to all 3 years,

$ s1_info --burst-bbox -i3 S1A_IW_SLC__1SDV_20*zip
Found 3 Sentinel-1 SLC products.
Bursts in S1A_IW_SLC__1SDV_20190708T083419_20190708T083446_028023_032A31_3850.zip:
--------------------------------------------------------------------------------
Sentinel1BurstSlc: t126_269602_iw3 at 2019-07-08 08:34:22.862800 [-44.78966927323444, -5.60909841030692, -44.03095902592307, -5.281282777526784]
Sentinel1BurstSlc: t126_269603_iw3 at 2019-07-08 08:34:25.621357 [-44.82651994771249, -5.775306612824314, -44.06757843365229, -5.447405172995813]
Sentinel1BurstSlc: t126_269604_iw3 at 2019-07-08 08:34:28.377858 [-44.86345755092964, -5.941632290597277, -44.10422314703207, -5.613396772238901]
Sentinel1BurstSlc: t126_269605_iw3 at 2019-07-08 08:34:31.136414 [-44.90045512046944, -6.107951510687787, -44.14094828299857, -5.779505047967723]

Bursts in S1A_IW_SLC__1SDV_20210416T083427_20210416T083454_037473_046B05_5F0C.zip:
--------------------------------------------------------------------------------
Sentinel1BurstSlc: t126_269602_iw3 at 2021-04-16 08:34:28.163438 [-44.75302178180448, -5.441926200806879, -43.99492830110982, -5.114392667709432]
Sentinel1BurstSlc: t126_269603_iw3 at 2021-04-16 08:34:30.921994 [-44.79038522681391, -5.608201362575531, -44.03132819074274, -5.28053161740093]
Sentinel1BurstSlc: t126_269604_iw3 at 2021-04-16 08:34:33.678495 [-44.82716246079352, -5.774587554024421, -44.06789156405017, -5.44645495821352]
Sentinel1BurstSlc: t126_269605_iw3 at 2021-04-16 08:34:36.437052 [-44.86408401066785, -5.941103955727937, -44.10456434344419, -5.612647386664313]

Bursts in S1A_IW_SLC__1SDV_20230112T083427_20230112T083454_046748_059ABA_08CC.zip:
--------------------------------------------------------------------------------
Sentinel1BurstSlc: t126_269602_iw3 at 2023-01-12 08:34:42.180443 [-44.78888641152506, -5.609366314517695, -44.03097118639631, -5.281849899312893]
Sentinel1BurstSlc: t126_269603_iw3 at 2023-01-12 08:34:44.938999 [-44.82566531143449, -5.775697260992483, -44.06756427711856, -5.44784226358671]
Sentinel1BurstSlc: t126_269604_iw3 at 2023-01-12 08:34:47.697556 [-44.86256125829071, -5.942034457815453, -44.10423982895002, -5.613979902782154]
Sentinel1BurstSlc: t126_269605_iw3 at 2023-01-12 08:34:50.454057 [-44.90403209173591, -6.128244693257249, -44.14033497451911, -5.779967417189208]

[scottstanie]

Is it possible ESA has labelled the ascending node incorrectly?

I'm looking at two frames 12 days apart in 2021: S1A_IW_SLC__1SDV_20210416T083427_20210416T083454_037473_046B05_5F0C.SAFE S1A_IW_SLC__1SDV_20210428T083427_20210428T083454_037648_047116_2170.SAFE

$ asfsmd  --pol vv  S1A_IW_SLC__1SDV_20210428T083427_20210428T083454_037648_047116_2170
$ asfsmd  --pol vv  S1A_IW_SLC__1SDV_20210428T083427_20210428T083454_037648_047116_2170

Looking at the results from s1_info, the acquisition times and bounding boxes of each burst is very close for all 9... but they are labelled different burst IDs, off by 1.

$ s1_info --burst-bbox -i3  S1A_IW_SLC__1SDV_2021*
...
Bursts in S1A_IW_SLC__1SDV_20210416T083427_20210416T083454_037473_046B05_5F0C.SAFE:
Sentinel1BurstSlc: t126_269602_iw3 at 2021-04-16 08:34:28.163438 [-44.75302178180448, -5.441926200806879, -43.99492830110982, -5.114392667709432]
Sentinel1BurstSlc: t126_269603_iw3 at 2021-04-16 08:34:30.921994 [-44.79038522681391, -5.608201362575531, -44.03132819074274, -5.28053161740093]
...

Bursts in S1A_IW_SLC__1SDV_20210428T083427_20210428T083454_037648_047116_2170.SAFE:
Sentinel1BurstSlc: t126_269601_iw3 at 2021-04-28 08:34:28.513324 [-44.7520380950549, -5.443137114414478, -43.99390811974907, -5.115525344912369]
Sentinel1BurstSlc: t126_269602_iw3 at 2021-04-28 08:34:31.273936 [-44.78942149597747, -5.609472570431819, -44.03032787809721, -5.28172455956012]

notice that

• the acquisition of the first burst is 2021-04-16 08:34:28.163438 and 2021-04-28 08:34:28.513324, and these match in bounding boxes
• BUT when I add print(ascending_node_dt) inside the S1BurstId.from_burst_params, I get

  datetime(2021, 4, 16, 7, 43, 23, 716412)
  datetime(2021, 4, 28, 7, 43, 26, 943511)

  ...they are off by 3 seconds, which is the normal time for the IW1,2,3 acquisitions. So i don't see how our math could be wrong, it looks like we're coming up with a consistent ID based on the ANX time.

So i'm not sure how to solve this from here...

[hfattahi]

Is it possible ESA has labelled the ascending node incorrectly?

I'm looking at two frames 12 days apart in 2021: S1A_IW_SLC__1SDV_20210416T083427_20210416T083454_037473_046B05_5F0C.SAFE S1A_IW_SLC__1SDV_20210428T083427_20210428T083454_037648_047116_2170.SAFE

$ asfsmd  --pol vv  S1A_IW_SLC__1SDV_20210428T083427_20210428T083454_037648_047116_2170
$ asfsmd  --pol vv  S1A_IW_SLC__1SDV_20210428T083427_20210428T083454_037648_047116_2170

Looking at the results from s1_info, the acquisition times and bounding boxes of each burst is very close for all 9... but they are labelled different burst IDs, off by 1.

$ s1_info --burst-bbox -i3  S1A_IW_SLC__1SDV_2021*
...
Bursts in S1A_IW_SLC__1SDV_20210416T083427_20210416T083454_037473_046B05_5F0C.SAFE:
Sentinel1BurstSlc: t126_269602_iw3 at 2021-04-16 08:34:28.163438 [-44.75302178180448, -5.441926200806879, -43.99492830110982, -5.114392667709432]
Sentinel1BurstSlc: t126_269603_iw3 at 2021-04-16 08:34:30.921994 [-44.79038522681391, -5.608201362575531, -44.03132819074274, -5.28053161740093]
...

Bursts in S1A_IW_SLC__1SDV_20210428T083427_20210428T083454_037648_047116_2170.SAFE:
Sentinel1BurstSlc: t126_269601_iw3 at 2021-04-28 08:34:28.513324 [-44.7520380950549, -5.443137114414478, -43.99390811974907, -5.115525344912369]
Sentinel1BurstSlc: t126_269602_iw3 at 2021-04-28 08:34:31.273936 [-44.78942149597747, -5.609472570431819, -44.03032787809721, -5.28172455956012]

notice that

• the acquisition of the first burst is 2021-04-16 08:34:28.163438 and 2021-04-28 08:34:28.513324, and these match in bounding boxes
• BUT when I add print(ascending_node_dt) inside the S1BurstId.from_burst_params, I get

datetime(2021, 4, 16, 7, 43, 23, 716412)
datetime(2021, 4, 28, 7, 43, 26, 943511)

...they are off by 3 seconds, which is the normal time for the IW1,2,3 acquisitions. So i don't see how our math could be wrong, it looks like we're coming up with a consistent ID based on the ANX time.

So i'm not sure how to solve this from here...

Thanks @scottstanie the incorrect ascending node time should be the issue. @seongsujeong to make sure that is the issue it will be great to extract the ascending node time for all the frames in track 126 over the region of interest for the same date and see how the ascending node time has changed along the track.

[seongsujeong]

Shared the screenshot in the Slack thread, but sharing it here as well for the information. Looks like we have incorrect ANX time as they should be the same.

[hfattahi]

Thanks @seongsujeong , yes it looks like an incorrect ANX time for those northern frames. The ascending node time for the frames in the same track can not be different.

[seongsujeong]

Thanks @scottstanie and @hfattahi for the investigation and the insight. Removing [Bug] in the title and the label as it turned out there is an issue in the data, not our software.

Should we need to have some sort of detection algorithm for such anomaly down the road?

[ghajduch]

Dears, This is a known issue on some S1A and S1B products RAW data that are then reflected in cascade in the corresponding Level 1 and Level 2 products.

This is referenced in those two quality disclaimers. https://sar-mpc.eu/disclaimer/79/ https://sar-mpc.eu/disclaimer/80/

The invalid annotation of anx time does not have other impact on the product performance than shifting the computed ID of the bursts. Work is ongoing to identify the most appropriate way to fix this in Sentinel-1 data.

[hfattahi]

Thank you @ghajduch for confirming that the issue indeed is the invalid annotation anx time. Please let us know if you will come up with any possible workaround?

[ghajduch]

Dear @hfattahi , We are working collectively (CLS, ARESYS, ESA...) to find a solution to fix this for future products. This solution, once applied on nominal production of Level 1 products, will be described in the appropriate L1 Detailed Processing Model (https://sentinel.esa.int/fr/web/sentinel/user-guides/sentinel-1-sar/document-library/-/asset_publisher/1dO7RF5fJMbd/content/sentinel-1-level-1-detailed-algorithm-definition). It is likely that the same solution will then be applicable as a work around for old products, but this remains to be confirmed. In any case, this will documented in the aforementioned quality disclaimers and related ones (https://sar-mpc.eu/disclaimer/78/, https://sar-mpc.eu/disclaimer/77/, etc)

[piyushrpt]

This is a solved problem in our setup and we have also verified this by labeling the whole SLC archive (IW, V-transmit). @ghajduch and @hfattahi , if we can help in anyway do let me know. I can put you in touch with folks who might help set that up.

[hfattahi]

Thank you @piyushrpt and sorry for my late response. Thanks to ESA colleagues for their suggestions. We are moving towards determining the ANX time from orbit. We validated over the entire 2021 archive and it seems to be working. The Fix is here.

[seongsujeong]

Closing this issue as this issue was addressed in #121.