Allow subsetting capabilities to subscriber

mike-gangl commented 2 years ago

Currently the subscriber uses the bounding box information to search for granules of interest. With the addition of subset capabilities in harmony and our use of that, it would be great if the bounding box capabilities was enhanced to call harmony and do the actual subset on the data and only return data of interest WITHIN the bounding box.

from Jinbo: "data_subscriber" can be a potential one-stop tool for all 'download and analysis' groups and scenarios. To me as a user, subsetting capability like this will complete this 'swiss army knife'.

Update 4/11/2022 Needs:

[ ] Validation of BBOX required for Subsetter
[ ] Adding the subsetting capability to "subscriber" not just "downloader"
[ ] Testing against large requests (hundreds to thousands of jobs)
[ ] Better Error Handling - print out links to harmony workflow-ui or jobs requests, remove the job from the .harmony file when it fails
[ ] Need to work with harmony team on adding the "fail on error" -> false through harmony py

mike-gangl commented 2 years ago

We plan on implementing this feature in 22.3, so within the next few months this should be supported.

mike-gangl commented 1 year ago

if no subsetter exists for the collection- just download the data.

mike-gangl commented 1 year ago

Can we capture metrics on subset requests that don't have hooked up subsetter/harmony requests - get info on what datasets we should enable next.

mike-gangl commented 1 year ago

does the harmony library override the use-agent for harmony requests, what's the impact on that.

podaac / data-subscriber

Allow subsetting capabilities to subscriber #36