search

sat-utils / sat-api

One API to search public satellites metadata on AWS
https://sat-api.developmentseed.org/search/stac
MIT License
177 stars 20 forks source link

Support ingest of S3 requester pays buckets #187

Closed mhiley closed 5 years ago

mhiley commented 5 years ago

Currently, it doesn't appear that STAC items residing on an S3 requester pays bucket can be ingested.

If I specify url as an s3:// url, I get the error: Error: Invalid protocol: s3:.

If I specify url as https://, as expected, the ingest fails with AccessDenied because the request must be signed due to the requester pays policy.

What are your thoughts on making the sat-api ingest support s3:// as a protocol? Seems like it should be reasonably straightforward. I can likely take a stab at implementing if there hasn't already been work done to support this.

matthewhanson commented 5 years ago

Hi @mhiley , sorry for the delay, but thanks for bringing this up. It's true, the data must be public at this time, but being able to ingest private s3 data would be quite useful. It could be done either by recognizing s3 urls, or be using signed urls with https.

There hasn't been any work done on implementing this yet, if you want to make a PR that would be great.

matthewhanson commented 5 years ago

this is now on develop and will be included in the 0.3.0 release coming this week.