search

ASFHyP3 / hyp3-isce2

HyP3 plugin for ISCE2 processing
Apache License 2.0
11 stars 9 forks source link

Use of BurstIDs from ESA S1 Burst Database #66

Open scottyhq opened 1 year ago

scottyhq commented 1 year ago

Background

If generating many burst interferograms you run into the case where it's not clear what burst-number to use if the framing is different for different granules.

IPF>=3.4 has standard burstIDs in the metadata for every granule which would be nice to use https://sentinels.copernicus.eu/web/sentinel/-/publication-of-brust-id-maps-for-copernicus-sentinel-1

For the Iran Earthquake example in the readme, here is a quick plot of all the ASF S1 frames and the burst 64_IW2_136231 that is processed

Screenshot 2023-05-23 at 5 24 33 PM

Describe the solution you'd like

I'm thinking as alternative to setting --reference-burst-number and --secondary-burst-number you could have --burst-id IW2_136231 and the software would figure out which burst within the SLC to extract

Additional context

For IPF>=3.4 the burst_ids should be in the metadata, but for IPF<3.4 this would require an additional lookup maybe based on the burstid database polygons (available via the above link). short example below for reading the burst_id info

gf = gpd.read_file('S1_burstid_20220530/IW/sqlite/burst_map_IW_000001_375887.sqlite3',
                   bbox=(54.1, 27.4, 54.2, 27.5)
                  )
myburst = gf[(gf.relative_orbit_number == 64) & (gf.subswath_name == 'IW2') & (gf.burst_id == 136231)]                 
burst_id                                                          136231
subswath_name                                                        IW2
relative_orbit_number                                                 64
time_from_anx_sec                                             2514.65709
orbit_pass                                                    DESCENDING
geometry               MULTIPOLYGON Z (((54.146844 27.677253 0, 53.68...
asjohnston-asf commented 1 year ago

Agreed, providing SLC + swath + burst index is cumbersome. It's difficult to know where a particular burst is spatially located within the SLC, and the burst index for a particular location may not be consistent from SLC to SLC. Our hope is to abstract the SLC away and deal with bursts directly.

ASF has already built an API to download the data for a particular burst from a particular SLC. Our insar_tops_burst workflow uses that API to download just the data needed to create the interferogram. For more information, see https://sentinel1-burst-docs.asf.alaska.edu/ .

ASF also intends to advertise these individual SLC burst products in our DAAC archive and make them accessible via Vertex, CMR, and Earthdata Search. You can explore our initial non-production efforts in Earthdata Search UAT or via the CMR UAT Search API.

With the ability to directly search for individual bursts, we plan to update the interface for insar_tops_burst to simply take the names of the reference and secondary bursts. For example:

--reference-scene S1A_IW_SLC__1SDV_20230523T170718_20230523T170745_048664_05DA5B_8850
--secondary-scene S1A_IW_SLC__1SDV_20230511T170718_20230511T170745_048489_05D52B_07DE
--swath-number 1
--polarization VV
--reference-burst-number 5
--secondary-burst-number 5

becomes:

--reference-scene S1_249434_IW1_20230523T170733_VV_8850-BURST
--secondary-scene S1_249434_IW1_20230511T170732_VV_07DE-BURST

Screenshot from 2023-05-24 09-53-56

scottyhq commented 1 year ago

@asjohnston-asf thanks for this roadmap, it sounds nicely designed and it looks like things are already moving forward for v0.4.0. Can you comment on how bursts are currently being indexed and the timeline for adding them? For example, my search for a point in Seattle currently only turns up 6 bursts:

https://search.uat.earthdata.nasa.gov/search/granules?p=C1257024016-ASF&pg[0][v]=f&pg[0][gsk]=-start_date&q=bursts&sp[0]=-122.37451%2C47.61444&tl=1684954770!3!!&lat=47.67619905654632&long=-124.16528320312499&zoom=7

Screenshot 2023-06-01 at 3 28 31 PM
asjohnston-asf commented 1 year ago

@scottyhq I'm not in a position to make commitments, but my best guess is that we'll begin indexing bursts in production for new SLCs as they arrive sometime this summer, and that we'll index bursts for the back-catalog of SLCs sometime in the next year.

The bursts currently indexed in CMR UAT are just to support development work. The bulk are from SLCs acquired over a few weeks in May 2023, so I've only been seeing 1-4 acquisitions available for any given area.

The change we have in-progress for tops_insar_burst does limit use to only bursts available in CMR. That will be limiting for time series until the back catalog of bursts is available. I'm curious to hear how ASF's timeline fits with any work you might have in mind.

scottyhq commented 1 year ago

The change we have in-progress for tops_insar_burst does limit use to only bursts available in CMR. That will be limiting for time series until the back catalog of bursts is available. I'm curious to hear how ASF's timeline fits with any work you might have in mind.

Thanks for this information @asjohnston-asf , if the timeline is a year for the back-catalog then it would be nice to retain an option for the existing syntax.

So far we've just started experimenting with this locally, but have in mind time series analysis for different areas running via HYP3 (possibly setting up our own deployment). This is the workflow we're using to calculate the relative burst numbering in any SLC based on TANX, seems to work so far but not extensively tested!

https://nbviewer.org/github/relativeorbit/hyp3bursts/blob/main/workflow-ascending.ipynb

asjohnston-asf commented 1 year ago

@scottyhq do you have specific temporal/spatial ranges of data you'd like to explore in the short term? ASF could weigh the cost of indexing those bursts vs the cost of maintaining the second interface. I suspect we'll be motivated to have a complete temporal coverage for at least some small area of interest available sooner than later.

asjohnston-asf commented 1 year ago

FYI, separately, the OPERA project at JPL has an implementation of computing burst IDs for IPF <3.4 SLCs at https://github.com/opera-adt/s1-reader/blob/main/src/s1reader/s1_burst_id.py . There's discussion at https://github.com/opera-adt/s1-reader/issues/117 that the metadata from ESA is occasionally inaccurate causing the computed burst ID to be incorrect, however. ASF's seen similar occasional metadata issues with our burst extractor API.

scottyhq commented 1 year ago

@scottyhq do you have specific temporal/spatial ranges of data you'd like to explore in the short term?

Sure, here are two points of interest (-122.299, 47.657) and (-121.767, 44.133), and the full temporal range, but if it helps to restrict we could focus on 2019-2023.

the metadata from ESA is occasionally inaccurate causing the computed burst ID to be incorrect,

Thanks for the linked info, good to know!

asjohnston-asf commented 1 year ago

ASF made a provisional release of S1 burst products available in production last Friday! They're accessible through:

The collection currently includes:

Thanks to @benbart for helping add all of the historical data!

We likely won't officially integrate tops_insar_burst with the production archive for another week or two, but you can integrate locally by setting host=cmr.earthdata.nasa.gov at https://github.com/ASFHyP3/hyp3-isce2/blob/develop/src/hyp3_isce2/burst.py#L346 . Integration with HyP3 and submitting burst InSAR jobs through Vertex are further off; you'll still need to export your scene names/pair lists from your search results and use them to run tops_insar_burst locally.

Eager to hear any feedback you might have!

scottyhq commented 5 months ago

Just noting that >=v0.4 uses preprocessed ASF bursts (https://github.com/ASFHyP3/hyp3-isce2/releases/tag/v0.4.0) that almost match ESA naming scheme. Just the track number must be looked up in as 'Full Burst Id' in the metadata:

S1_136231_IW2_20200604T022312_VV_7C85-BURST (vs 064_136231_IW2)