download_data() retry's forever when s3 returns 404

[bitsofinfo]

This will repeat forever if a file really does not exist and cannot be downloaded

2014-09-24 15:27:33,586 DEBUG check_cache_size get_memory_usage
2014-09-24 15:27:33,690 DEBUG get_key from cache '/0/0/31/63/46/2223d.jpg/2223d_1.0.jpg'
2014-09-24 15:27:33,691 DEBUG download_data range '/0/0/31/63/46/2223d.jpg/2223d_1.0.jpg' '{}' [thread 'Thread-34']
2014-09-24 15:27:33,752 ERROR S3ResponseError: 404 Not Found
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>0/0/31/63/46/2223d.jpg/2223d_1.0.jpg</Key><RequestId>222222</RequestId><HostId>232+bnQo3Yb/222</HostId></Error>
Traceback (most recent call last):
  File "/my/p/python2.6/site-packages/yas3fs/__init__.py", line 1732, in download_data
    bytes = key.get_contents_as_string()
  File "/my/p/python2.6/site-packages/boto/s3/key.py", line 1730, in get_contents_as_string
    response_headers=response_headers)
  File "/my/p/python2.6/site-packages/boto/s3/key.py", line 1603, in get_contents_to_file
    response_headers=response_headers)
  File "/my/p/python2.6/site-packages/boto/s3/key.py", line 1435, in get_file
    query_args=None)
  File "/my/p/python2.6/site-packages/boto/s3/key.py", line 1467, in _get_file_internal
    override_num_retries=override_num_retries)
  File "/my/p/python2.6/site-packages/boto/s3/key.py", line 325, in open
    override_num_retries=override_num_retries)
  File "/my/p/python2.6/site-packages/boto/s3/key.py", line 273, in open_read
    self.resp.reason, body)
S3ResponseError: S3ResponseError: 404 Not Found
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>0/0/31/63/46/2223d.jpg/2223d_1.0.jpg</Key><RequestId>2222</RequestId><HostId>2323222+bnQo3Yb/22</HostId></Error>
2014-09-24 15:27:33,753 INFO download_data error '/0/0/31/63/46/2223d.jpg/2223d_1.0.jpg' 0-10485759 [thread 'Thread-34'] -> retrying

[bitsofinfo]

This infinite loop prevents yas3fs from stopping cleanly as well. Here is another one

2014-09-25 12:55:56,582 INFO entries, mem_size, disk_size, download_queue, prefetch_queue, s3_queue: 1002, 0, 70472058, 0, 0, 0
2014-09-25 12:55:56,583 DEBUG new_locks, unused_locks: 0, 0
2014-09-25 12:55:56,583 DEBUG gc count0/threshold0, count1/threshold1, count2/threshold2: 362/700, 2/10, 2/10
2014-09-25 12:55:56,583 DEBUG check_threads 'False'
2014-09-25 12:55:58,566 DEBUG check_cache_size get_memory_usage
2014-09-25 12:55:58,655 DEBUG read awake '/1/2/3172476_1.0.jpg' '127280' '0' '0'
2014-09-25 12:55:58,655 DEBUG get_metadata -> '/1/2/3172476_1.0.jpg' 'attr' 'None'
2014-09-25 12:55:58,656 DEBUG get_metadata <- '/1/2/3172476_1.0.jpg' 'attr' 'None' '{u'st_ctime': 1411636835.0, u'st_mtime': 1411636835.0, u'st_nlink': 1, u'st_mode': 33188, u'st_blksize': 131072, u'st_size': 3728, u'st_gid': 495, u'st_uid': 497, u'st_atime': 1411636835.0}'
2014-09-25 12:55:58,656 DEBUG read '/1/2/3172476_1.0.jpg' '127280' '0' '0' out of range
2014-09-25 12:55:58,656 DEBUG enqueue_download_data '/1/2/3172476_1.0.jpg' 0 127280
2014-09-25 12:55:58,657 DEBUG read wait '/1/2/3172476_1.0.jpg' '127280' '0' '0'
2014-09-25 12:55:58,665 DEBUG download_data '/1/2/3172476_1.0.jpg' 0-10485759 [thread 'Thread-46']
2014-09-25 12:55:58,665 DEBUG get_key from cache '/1/2/3172476_1.0.jpg'
2014-09-25 12:55:58,666 DEBUG type k = '<class 'boto.s3.key.Key'>'
2014-09-25 12:55:58,666 DEBUG  dir k = '['BufferSize', 'DefaultContentType', 'RestoreBody', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__iter__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slotnames__', '__str__', '__subclasshook__', '__weakref__', '_get_base64md5', '_get_file_internal', '_get_key', '_get_md5', '_get_remote_metadata', '_normalize_metadata', '_send_file_internal', '_set_base64md5', '_set_key', '_set_md5', '_underscore_base_user_settable_fields', 'add_email_grant', 'add_user_grant', 'base64md5', 'base_user_settable_fields', 'bucket', 'cache_control', 'change_storage_class', 'close', 'closed', 'compute_md5', 'content_disposition', 'content_encoding', 'content_language', 'content_type', 'copy', 'delete', 'delete_marker', 'encrypted', 'endElement', 'etag', 'exists', 'expiry_date', 'f', 'filename', 'generate_url', 'get_acl', 'get_contents_as_string', 'get_contents_to_file', 'get_contents_to_filename', 'get_file', 'get_md5_from_hexdigest', 'get_metadata', 'get_redirect', 'get_torrent_file', 'get_xml_acl', 'handle_addl_headers', 'handle_encryption_headers', 'handle_restore_headers', 'handle_version_headers', 'is_latest', 'key', 'last_modified', 'local_hashes', 'make_public', 'md5', 'metadata', 'mode', 'name', 'next', 'ongoing_restore', 'open', 'open_read', 'open_write', 'owner', 'path', 'provider', 'read', 'resp', 'restore', 'send_file', 'set_acl', 'set_canned_acl', 'set_contents_from_file', 'set_contents_from_filename', 'set_contents_from_stream', 'set_contents_from_string', 'set_metadata', 'set_redirect', 'set_remote_metadata', 'set_xml_acl', 'should_retry', 'size', 'source_version_id', 'startElement', 'storage_class', 'update_metadata', 'version_id']'
2014-09-25 12:55:58,667 DEBUG      k = '<Key: party,0/0/31/72/47/3172476.jpg/3172476_1.0.jpg>'
2014-09-25 12:55:58,667 DEBUG download_data '/1/2/3172476_1.0.jpg' 0-10485759 [thread 'Thread-46'] already downloading

[bitsofinfo]

@danilop can you please comment w/ any thoughts as to what could be causing the condition in the latter comment? (i.e. the range read issue)

[bitsofinfo]

Fixed by #47 merge retry download/read ops configurability to avoid infinite loops