search

landsat-pds / landsat_ingestor

Scripts and other artifacts for landsat data ingestion into Amazon public hosting.
Apache License 2.0
102 stars 18 forks source link

Fewer requests to the USGS download-url endpoint #11

Closed kapadia closed 9 years ago

kapadia commented 9 years ago

Currently each scene is processed individually. Part of the processing is requesting an authenticated download url from USGS. It's currently done:

usgs.api.download('LANDSAT_8', 'EE', [scene_root], 'STANDARD')

Rather than submitting 1 request per scene, we can group multiple scenes together in a single request.

usgs.api.download('LANDSAT_8', 'EE', [scene_root_1, scene_root_2, ..., scene_root_n], 'STANDARD')

/cc @warmerdam

warmerdam commented 9 years ago

In practice we process them one scene at a time, and the cost of generating a download url is tiny compared to the resources consumed during downloading the scene. So I don't see much benefit, and potentially quite a bit of complication in this.