search

landsat-pds / landsat_ingestor

Scripts and other artifacts for landsat data ingestion into Amazon public hosting.
Apache License 2.0
102 stars 18 forks source link

Fewer requests to the USGS auth endpoint #7

Closed kapadia closed 9 years ago

kapadia commented 9 years ago

Right now a new API token is generated each time a download url is requested. The ingestor doesn't need to hit the auth endpoint each time:

https://github.com/landsat-pds/landsat_ingestor/blob/1d3876f7c7021e1b4100667beb275824bbd2786d/ingestor/puller_usgs.py#L45-L56

API tokens are valid for 1 hour after the initial request and reset each time a request is made. Let's be kind to USGS servers, and only request a single token.

/cc @warmerdam

warmerdam commented 9 years ago

Currently the way this work operationally there is a new subprocess launched (of puller.py) for each scene to be requested. How would you suggest keeping track if we have a valid login still available? I gather the usgs api can cache them in /tmp or something? Turning over to you to fix.

kapadia commented 9 years ago

@warmerdam Storing the API key in /tmp is the approach taken in the USGS module. See: https://github.com/mapbox/usgs/blob/master/usgs/api.py#L156-L157

Based on that, I assume we can remove a few lines of code:

https://github.com/landsat-pds/landsat_ingestor/commit/22acbbfa1e4557f3cf0df53708d36c860aa46102

and api.download_url will grab the API key /tmp if it exists.

Do you see any problems here?