search

landsat-pds / landsat_ingestor

Scripts and other artifacts for landsat data ingestion into Amazon public hosting.
Apache License 2.0
102 stars 18 forks source link

usgs request failure - perhaps we need retry? #4

Open warmerdam opened 9 years ago

warmerdam commented 9 years ago

I get stuff like the following at least occationally. Perhaps there is something that needs retry logic? 

+ l8_process_run.py -v -s auto --start-date=2015-01-15 --end-date=2015-02-06 --queue
logging in...
Traceback (most recent call last):
  File "/opt/planet/programs/landsat_ingestor/ingestor/l8_process_run.py", line 178, in <module>
    status = main(sys.argv[1:])
  File "/opt/planet/programs/landsat_ingestor/ingestor/l8_process_run.py", line 155, in main
    limit=args.limit)
  File "/opt/planet/programs/landsat_ingestor/ingestor/l8_process_run.py", line 21, in query_for_scenes
    os.environ['USGS_PASSWORD'])
  File "/opt/planet/programs/landsat_ingestor/ingestor/usgs/api.py", line 154, in login
    api_key = element.text
AttributeError: 'NoneType' object has no attribute 'text'
kapadia commented 9 years ago

@warmerdam do you happen to know what the response code was when this occurred? This error is raised from the usgs client library. Though the library does check for errors in a response, it doesn't catch empty responses (e.g. 500s)

warmerdam commented 9 years ago

I'm not sure - I'm not even sure I've seen this recently. Perhaps you could monitor the process and see if this is still happening sometimes and can be resolved.

kapadia commented 9 years ago

@warmerdam Changes to https://github.com/mapbox/usgs/commit/3b6cdc933086118277fb9471a4faf54a8fd7fe67 will now surface the error we receive from USGS so that we can figure out an appropriate solution on the landsat-ingestor side.

kapadia commented 9 years ago

The auth error finally surfaced, and is documented at: https://github.com/mapbox/usgs/issues/20. Now that we know how the error looks, we can handle it appropriately.

/cc @warmerdam