[Processing Request]: High Priority CalVal sites Request

gracebato commented 1 week ago

Venue

PST

Product

DISP-S1

SAS Version

No response

SDS Version

No response

Input Data

High Priority Frames:

[ ] F08622
[x] F08882
[ ] F09156
[x] F11116
[ ] F12640
[ ] F18903
[ ] F28486
[ ] F33039
[ ] F33065
[ ] F36542
[ ] F42779

Share Results

[ ] Google Earth Engine (GEE)
[X] NASA Distributed Active Archive Centers (DAAC) User Acceptance Testing (UAT)

Additional Notes

F11116 and F08882 were already processed in: https://github.com/nasa/opera-sds/issues/53

gracebato commented 1 week ago

Parameters should be similar to https://github.com/nasa/opera-sds/issues/53, e.g.

Date range: 20160701 - 20240905
k=15
m=5

philipjyoon commented 6 days ago

@gracebato just to confirm: You'd like these products delivered to ASF UAT correct? #53 did not request that.

Also does it make a difference if we process these on our INT venue instead of PST? The difference is that if we process on PST we will keep the products in PST S3 forever; wherever INT S3 will be deleted clean every deployment. So the question is: do these products need to be archived in PST S3 forever or just delivering to ASF UAT sufficient?

EDIT: After speaking with @LucaCinquini we decided that we will process on the PST venue and deliver to ASF UAT.

philipjyoon commented 5 days ago

We want to make sure that all frames process at least 4 years worth of data. So I'll process 2016-2021 covering 5 years. And if any of the frame still don't have 4 years worth of data, which is possible, I can extend the time range of those specific frames until that point.

This is a bit of manual work but not too bad. I can predetermine which frames would not have 4 years worth of data in the first 5 calendar years using the historical database.

gracebato commented 5 days ago

Hi @philipjyoon all DISP-S1 products goes to UAT for all produced going forward. So request https://github.com/nasa/opera-sds/issues/53 would also go to UAT. Thanks.

philipjyoon commented 2 days ago

This request will be executed in 2 variations with one dependency. @gracebato please correct me if the understanding is incorrect:

Variation 1: Using OPERA PCM version 3.1.0-rc.6.0 (latest version as of today) process these frames for at least 4 years. Product version is still v0.6
Dependency: After OPERA PCM version 3.1.0-rc.7.0 is released next week, first run the 3 frames in request #53 using product version v0.7 for 2016-2024
Once above dependency is satisfactory, run these frames using the same software and same product version for the entire historical period 2016-2024.

philipjyoon commented 2 days ago

Will use the following batch_proc for Variation 1

{
 "enabled": true,
 "label": "PST_Request_55",
 "processing_mode": "historical",
 "include_regions": "",
 "exclude_regions": "",
 "temporal": true,
 "data_start_date": "2016-07-01T00:00:00",
 "data_end_date": "2021-02-01T00:00:00",
 "k": 15,
 "m": 6,
 "frames": [8622, 9156, 12640, 18903, 28486, 33039, 33065, 36542, 42779],
 "wait_between_acq_cycles_mins": 10,
 "job_type": "cslc_query_hist",
 "provider_name": "ASF",
 "job_queue": "opera-job_worker-cslc_data_query_hist",
 "download_job_queue": "opera-job_worker-cslc_data_download_hist",
 "chunk_size": 1
 }

philipjyoon commented 4 hours ago

~65% complete as of now.

f28486 has finished. f33039 is taking by far the longest... it's currently only 35% complete processing around 2018 right now.

nasa / opera-sds