search

nasa / opera-sds

Apache License 2.0
3 stars 2 forks source link

[Processing Request]: OPERA RTC-S1 for Dist-S1 R&D #39

Open cmarshak opened 8 months ago

cmarshak commented 8 months ago

Venue

PST

Product

RTC-S1

SAS Version

No response

SDS Version

No response

Input Data

This is a specific request for generating time-series of a given bursts that overlaps 300 sites around the globe.

The relevant table is here: https://github.com/OPERA-Cal-Val/dist-s1-research/blob/dev/marshak/3_dist_sites/slcs_for_processing.csv.zip

In terms of processing, the relevant information in the table is: (s1_slc_id, burst_id) --> (uniquely) JPL burst RTC-S1 product

This amounts to approximately 2 TBs of data assuming there is about ~30 MBs files per burst product generate (though is an overestimate I believe).

Share Results

Additional Notes

If you do not have the capability to limit to specified burst, we will need to talk as this generates orders of magnitude more data.

cmarshak commented 8 months ago

~Would like to process a few bursts in the list and see them delivered to UAT before proceeding to the entire list.~

This volume is not possible to deliver to UAT - so hoping this can be stored on our public PST clone's s3 bucket. There will be an excess of data generated, but we can work with you to delete the data that is not required for our development.

As noted above, we likely need 1-2 bursts per all the SLCs requested (which can have 20 - 40 bursts per product!). I am taking the highest end of all ranges for the approximate estimates above.

cmarshak commented 7 months ago

@niarenaw and @LucaCinquini - we have updated the request.

The SLCs span a subset of sites and only go from 2020-10-01 to 2022-12-01 (a little over 2 years).

There should be 8,703 SLCs in https://github.com/OPERA-Cal-Val/dist-s1-research/blob/dev/marshak/3_dist_sites/slcs_for_processing.csv.zip. The request will cost ~$3,050 based on Nick's estimate of $0.35 / SLC.

Let me know if there are any questions.

niarenaw commented 7 months ago

All SLCs processed except the following: S1A_IW_SLC__1SDV_20221117T060948_20221117T061015_045930_057EE2_89A5 S1A_IW_SLC__1SDV_20221117T153336_20221117T153403_045936_057F16_AD44 S1A_IW_SLC__1SDV_20221117T153516_20221117T153543_045936_057F16_AB2D S1A_IW_SLC__1SDV_20221117T171630_20221117T171657_045937_057F23_824A Further investigation needed on why these can't be downloaded from ASF.

cmarshak commented 7 months ago

Has to be a transient error.

I was able to download the first one through the search GUI in the browser:

https://search.asf.alaska.edu/#/?searchType=List%20Search&searchList=S1A_IW_SLC__1SDV_20221117T060948_20221117T061015_045930_057EE2_89A5&resultsLoaded=true&granule=S1A_IW_SLC__1SDV_20221117T060948_20221117T061015_045930_057EE2_89A5-SLC

niarenaw commented 6 months ago

Products should be complete now.

cmarshak commented 6 months ago

Thanks, Nick!