ASFHyP3 / hyp3-event-monitoring

A software stack to allow users to enable event monitoring and processing of data as it becomes available
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

find_new lambda reports `Validation Error` when attempting to submit jobs for new scenes #58

Closed asjohnston-asf closed 2 years ago

asjohnston-asf commented 2 years ago

find_new is the lambda function that looks for new Sentinel-1 scenes matching the search criteria for SARVIEWS events and submits new RTC and InSAR jobs to HyP3 for them: https://github.com/ASFHyP3/hyp3-event-monitoring/blob/develop/find_new/src/find_new.py

Since February 3, find_new has failed to submit any new jobs for any new scenes due to a Validation Error: granule_list may not be used in conjunction with other search parameters error when attempting to find appropriate secondary scenes for InSAR jobs. This is due to a change to the ASF Search API restricting the use of the granule_list search parameter. The error is thrown by this function call: https://github.com/ASFHyP3/hyp3-event-monitoring/blob/develop/find_new/src/find_new.py#L85 . The underlying bug for hyp3_sdk.asf_search.get_nearest_neighbors is at https://github.com/ASFHyP3/hyp3-sdk/issues/156

hyp3_sdk.asf_search.get_nearest_neighbors is deprecated. We should update find_new to leverage asf_search.baseline_search instead. An example using asf_search to find similar InSAR pairs is in HyP3 at https://github.com/ASFHyP3/hyp3/blob/develop/apps/process-new-granules/src/process_new_granules.py#L22

jtherrmann commented 2 years ago

Would something like this work?

def get_neighbors(granule, max_neighbors=2):
    stack = asf_search.baseline_search.stack_from_product(granule)
    stack = [item for item in stack if item.properties['temporalBaseline'] < 0]
    neighbors = [item.properties['sceneName'] for item in stack[-max_neighbors:]]
    return neighbors

Or do we want something more like what @jhkennedy wrote in https://github.com/ASFHyP3/hyp3-sdk/issues/156#issuecomment-1062301597 ?

Also, given that it seems like this is a commonly used feature, should we perhaps implement it somewhere else, such as hyp3_sdk? This would also help to avoid duplicating the tests at https://github.com/ASFHyP3/hyp3/blob/develop/tests/test_process_new_granules.py#L81 (we could move those tests to the new implementation location).