smore-inc / tinys3

A simple python S3 upload library. Inspired by requests
MIT License
285 stars 54 forks source link

URL used by S3Request.bucket_url() has changed and no longer works #54

Open leonardr opened 7 years ago

leonardr commented 7 years ago

Thanks for your work on a useful library.

I have upload code that works with tinys3 0.1.11 but fails with 0.1.12. The difference appears to be the destination URL tinys3 uses to upload a document. It's possible this is the same problem as issue #53, but I get a different error message.

Revision 308e3b9df41100dd676ff3abe452d2022c5b4e69 changed the type of URL generated by S3Request.bucket_url. In 0.1.11, a document I upload is sent to http://s3.amazonaws.com/{bucket}/{key}. This works fine. In 0.1.12, the URL used is http://{bucket}.s3.amazonaws.com/{key}, and I get a 403 error from S3:

  File "/home/leonardr/.local/lib/python2.7/site-packages/tinys3/pool.py", line 84, in as_completed
    yield r.result()
  File "/home/leonardr/.local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/home/leonardr/.local/lib/python2.7/site-packages/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/leonardr/.local/lib/python2.7/site-packages/tinys3/request_factory.py", line 348, in run
    r.raise_for_status()
  File "/home/leonardr/.local/lib/python2.7/site-packages/requests/models.py", line 893, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 403 Client Error: Forbidden for url: {url}

In revision 308e3b9df41100dd676ff3abe452d2022c5b4e69 the reason given for changing the URL format is "Come back of the virtual-hosting URL style as the old one actually isn't working with a recent bucket". I would guess each URL style works on some buckets and fails on others.

Please let me know if you need any more information about my S3 configuration.