search

nsidc / earthaccess

Python Library for NASA Earthdata APIs
https://earthaccess.readthedocs.io/
MIT License
394 stars 79 forks source link

Download ASTER L1A data using earthaccess #314

Closed ShashankBice closed 10 months ago

ShashankBice commented 11 months ago

Hi folks, I want to see if there is a way I can programmatically download ASTER L1A data using earthaccess. I have been following the tutorials from the project Readme and ICESat-2 Hackweek tutorial, and I think I am able to search for granules for the particular dataset I need to download with the correct version specified. I am not able to download the data though.

Here is the command which I ran:

results = earthaccess.search_data(
    short_name = 'AST_L1A',
    version = '003',
    cloud_hosted = True,
    bounding_box = (-134.7,58.9,-133.9,59.2),
    temporal = ('2020-03-01','2020-04-30'),
    count = 100
)

I get two granules from this search. I then go ahead to download the 2 granules with the command: files = earthaccess.download(results, "./test_sample_data") This results in the below prompt, without download of any data:

Getting 2 granules, approx download size: 0.22 GB
QUEUEING TASKS | : 0it [00:00, ?it/s]
PROCESSING TASKS | : 0it [00:00, ?it/s]
COLLECTING RESULTS | : 0it [00:00, ?it/s]

A note, from the results description, it seems that this dataset is not yet cloud-hosted. If this info is helpful in debugging the issue.

Guidance here is very much appreciated :blush: I can provide more details as required tomorrow if needed to get to the bottom of this issue!

Thanks, Shashank

asteiker commented 11 months ago

Hi @ShashankBice Thanks for reporting this! It looks like this ASTER L1A dataset does not support direct download; I only see options to use an ordering service from Earthdata Search. I'm hoping folks from the LP.DAAC can chime in to confirm: @amfriesz @BriannaLind Any advice you could offer for programmatic ASTER downloads?

mfisher87 commented 11 months ago

I'd also like to think about / ask:

danielfromearth commented 11 months ago

Hi @ShashankBice Thanks for reporting this! It looks like this ASTER L1A dataset does not support direct download; I only see options to use an ordering service from Earthdata Search. I'm hoping folks from the LP.DAAC can chime in to confirm: @amfriesz @BriannaLind Any advice you could offer for programmatic ASTER downloads?

Hi @asteiker, I'm wondering: What you were looking at when you determined "I only see options to use an ordering service from Earthdata Search"?

(The reason I ask is to assess whether issues encountered when attempting to download DSCOVR_EPIC_L2_AER data using earthaccess are perhaps caused by configurations similar to the ASTER L1A data being investigated in this issue.)

BriannaLind commented 11 months ago

Yeah @asteiker is right, L1A needs to be 'staged' for delievery. @danielfromearth and @mfisher87 I'm not sure where you would see this in the record, but in Earthdata Search you see this: image ..... the data actaully needs to be staged for delivery and then you can choose whether you want an hdf or a geotiff.

@ShashankBice You should be able to earthaccess to get AST_L1T v003 ("collectionConceptId": "C1000000320-LPDAAC_ECS") its the ASTER Level 1 precision terrain corrected registered at-sensor radiance V003

ShashankBice commented 11 months ago

Hi everyone, Thanks for looking into this, I really appreciate it! Thanks @BriannaLind, unfortunately I won't be able to use L1T imagery for our applications, where we want the raw L1A sensor data to derive topography using stereo processing routines. We are creating interactive and executable photogrammetry tutorials, and will use ASTER data for the first iteration of tutorials because it is publicly available. It would have been good if the L1A data could be accessed programmatically, which would have allowed users to pull in data for their region of interest and try out the tutorials, but I understand the limitations of tape storage and the need of the data to be staged before it can be accessed.

Thanks for all of your work on earthacces, it does help us in a lot of our data retrieval workflows, Cheers, Shashank

BriannaLind commented 11 months ago

@betolink it would be interesting to explore how difficult it would be to add support for 'on-demand data' to 'earthaccess'

@asteiker would be right to say that if data is 'on-demand' it cannot be accessed programmatically without first 'ordering/downloading it'?

jhkennedy commented 11 months ago

@BriannaLind asynchronous requests take a little bit more work to handle well, and I don't know what tools/sdks/apis already exist for ASTER L1A, but at minimum, you'll need:

All that seems like something that should be handled by an Earthaccess "plugin" as it's likely very specific to each on demand dataset.

I'd be interested in helping to scope this out as it would be cool to get ASF's on-demand products accessible via Earthaccess too

mfisher87 commented 11 months ago

I agree with @jhkennedy 's take here; if I understand correctly, we'd need to manage orders with EGI. It is documented but 5 (?) years back I built an interface on top of EGI and my recollection is that there were a lot of EGI-specific concerns the app needed to be responsible for. Some of its parameters are shared with CMR, but it can do a lot of other stuff like call ESI services, so there's a lot more going on. I recall EGI making an unexpected change that broke our interface at some point as well. I think there's a granule limit of 1 million granules on EGI orders, and some other limitations our app would need to know.

If we can expose an extendable "downloader" interface/protocol, we could enable this type of functionality and avoid a hard dependency on EGI.

BriannaLind commented 11 months ago

Wow! @jhkennedy thanks for breaking it down in a way that is so easy to understand! @mfisher87 I wonder if the AppEEARS API would have a good interface/protocol model to build off of?

conversation adjacent question: what is the benefit of having a collection 'on demand' rather than distinct granules having distinct endpoints that are always accessible? Is it cheaper to store?

betolink commented 11 months ago

I was looking at https://cmr.earthdata.nasa.gov/search/granules.umm_json?short_name=AST_L1A and going back to what @danielfromearth mentioned, there should be a way in CMR to tell how to access the data even if they have to go through a service.

I like @jhkennedy's idea of having a plugin system to access ordering APIs. Not just for this dataset, I imagine having Harmony, AppEEars, OPeNDAP etc. How can we make it happen!?

asteiker commented 11 months ago

I really like where this is going too, great thoughts @jhkennedy ! @betolink Yes, this has always been a sticking point when it comes to access URLs curated directly in a UMM-C record vs associated UMM-S or UMM-T records (i.e. it's not straightforward to surface all access methods across these schemas, and we developed metadata mappings as part of our tools project for the NSIDC landing pages to address this). Regardless, I love this idea of plugins for given tools/services like ASF's ordering service, Harmony, AppEEARs, etc . I thought we had another Issue to surface associated services but maybe misremembering. We would want this to be driven by UMM-S/UMM-T <-> UMM-C associations as well.

andypbarrett commented 10 months ago

Should we open a new issue to explore/sketch out how to create plugins and close this one?

MattF-NSIDC commented 10 months ago

:100:

Opening now! EDIT: https://github.com/nsidc/earthaccess/issues/328

MattF-NSIDC commented 10 months ago

Opened #329 to track the work of improving this user experience and #328 to support alternate download mechanisms like the async one required for this dataset.

I think we can consider this specific ticket resolved: ASTER L1A download is currently not supported. If anyone wants to continue discussion specific to this dataset, please re-open!