Open jrbourbeau opened 9 months ago
Okay, so this is maybe user error. If I use provider="POCLOUD"
instead of provider="PODAAC"
, then the S3 download case works. I was able to figure this out my looking at the list of DAACs defined here
Naively I don't really care whether the provider is "PODAAC"
or "POCLOUD"
. My natural thought is "This data is coming from the PODAAC data center, earthaccess
can you please download it for me?". I'm also new to this space, so maybe most people find the "PODAAC"
vs. "POCLOUD"
distinction clear and obvious. I'm curious to get thoughts from others who are more familiar with these things, and the earthaccess
target user base, than I am.
@jrbourbeau I work at a DAAC and also find this confusing!
CMR itself doesn't really have a concept of "organization", what we naturally think of when we see a name like "PODAAC". CMR, instead, just has a concept of "providers", where each organization may have one or more providers:
I think, that before the cloud transition, "provider" essentially was synonymous with "organization," and so all the older, and therefore on-prem data is typically provided by a provider with the same name as the DAAC. Some^ (most?) DAACs decided to add a second provider to represent their cloud collections, and we're stuck with confusion between organization and provider, where the provider is really overloaded to mean "on prem" vs "cloud" at this point.
All that's background, but it would be nice to more elegantly handle org -> provider for users and better infer which provider is appropriate for a dataset/usecase.
^ At ASF we only have one provider, and we use a "datapool" app to abstract data location in the CMR records, so you have to actually inspect the records for S3 info to determine if it's in the cloud, or just "know". This has a litany of it's own problems but does mean we avoid the above confusion.
Thanks for the extra context @jhkennedy, that's useful.
Do DAACs have multiple cloud or on-prem-providers (e.g. "cloud-providers": ["POCLOUD", "POCLOUD2", "POCLOUD_for_real"]
)? I'm wondering if users can just specify the org (e.g. "PODAAC"
) and then earthaccess
can dispatch to the on-prem or cloud provider accordingly.
Do DAACs have multiple cloud or on-prem-providers (e.g. "cloud-providers": ["POCLOUD", "POCLOUD2", "POCLOUD_for_real"])?
Right now, not that I am aware of but I haven't inventoried them. There is definitely nothing preventing them from doing so.
I'm wondering if users can just specify the org (e.g. "PODAAC") and then earthaccess can dispatch to the on-prem or cloud provider accordingly.
Yeah, this would be nice, esp. since DAACs are generally organized by subject-area. @betolink thoughts here?
When we deal with the full record, earthaccess has some hacky code to derive the providers based on the DAAC. Perhaps we need to build a method to derive the provider from URL signatures... e.g.
s3://nsidc-cumulus-prod-public/ATLAS/ATL03/006/2018/10/14/ATL03_20181014000347_02350101_006_02_BRW.h5"
We can see that this is nsidc, just by pattern matching.
s3://podaac-ops-cumulus-protected/SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205/ssh_grids_v2205_1992101512.nc
same here.
Perhaps just a dictionary of S3 bucket
-> provider endpoint
would work here? @jhkennedy @jrbourbeau @MattF-NSIDC
I'm concerned that could be fragile. If that's an implementation detail and not a well-documented interface, it could change from under us.
Even if it is well-documented and considered "part of the public API" for the purpose of versioning and deprecation announcements, it feels like we should be able to answer this question by a more explicit CMR query so we don't need special logic that knows about URL structure.
:wave: "Hey CMR, what provider provides 'direct' access to {short_name}
?" :wave:
I don't have the time today to dive in to the CMR docs and research this, I just feel that's something it should be able to answer in a perfect world ;) Then earthaccess wouldn't have to store info about DAACs and which providers map to which DAAC.
Downloading HTTPS files with
works as expected with output
When I then switch to downloading S3 URLs directly (making sure to run in
us-west-2
too)I get the following output (and no files downloaded):
it looks like we're having trouble inferring the data provider, so if I then manually specify
provider="PODAAC"
I get an error