Closed jzoldak closed 9 years ago
Hi Jesse!
Modify the S3 cache backend code so that if the user does not have credentials, it will connect as an anonymous user rather than erroring with "No handler was ready to authenticate... Check your credentials" and disabling the S3CacheBackend.
Solutions I can think of:
anon=True
when that happens. But then pip-accel will always need an HTTP response to find out that no credentials are available, and only then can it respond by issuing anonymous requests. That feels somewhat sub optimal :-).Hey Peter!
Yeah, I agree:
Perhaps another option might be to:
>>> from boto import Config
>>> Config().dump()
This shows my credentials if they are configured and shows no credentials if not.
Hi Jesse,
I believe pip-accel version 0.22 resolves your problem.
I went with option two in the end because I realized this past weekend that if I want to allow people to use Boto's "automagic" support for the EC2 metadata service there is no way to avoid the HTTP request so I should just accept it and move on. Anyway, the EC2 metadata service uses the IP address 169.254.169.254
which is a link-local IPv4 address range so I don't think this will cause problems in practice (e.g. hanging on a timeout).
Can you confirm whether pip-accel 0.22 resolves your problem? Thanks for the feature request!
Peter - I think it seems to work, but with a few caveats.
s3-readonly = yes
then when it fails uploading a file, the S3 cache backend is turned off. See this travis buildWe would probably catch these and any other corner case issues or bugs by writing unit tests for the S3CacheBackend. I could totally help do that if you'd like.
boto "Caught exception reading instance data" error messages get output when trying to read the metadata. See this travis build where I turned on verbose mode so you can see the flow (but the errors get printed out regardless of verbose mode).
This is now fixed: I silenced the Boto logger. There is an escape hatch in case someone ever does need to debug the usage of Boto inside pip-accel (for details refer to ec426eaa9ef3369bc0392d560c7f8b0c60212328).
If you do not set s3-readonly = yes then when it fails uploading a file, the S3 cache backend is turned off. See this travis build
I'll see if I can improve on this as well.
We would probably catch these and any other corner case issues or bugs by writing unit tests for the S3CacheBackend. I could totally help do that if you'd like.
There are unit tests for the S3 backend, or rather the test suite runs twice for each supported version of Python: Once with the S3 cache backend disabled and once with the S3 cache backend enabled. This uses FakeS3 so I don't need to create and pay for an actual Amazon S3 bucket.
If you do not set s3-readonly = yes then when it fails uploading a file, the S3 cache backend is turned off.
I believe this is now fixed as well. As far as I can tell FakeS3 doesn't support simulating read only bucket access so I had to figure out how to create a read only IAM policy in Amazon S3 in order to test this manually. For previous manual testing I had used the following IAM policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "*"
}
]
}
For this testing I switched to the following read only policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:Get*", "s3:List*"],
"Resource": "*"
}
]
}
In my tests the latest version correctly falls back to read only Amazon S3 access when a PUT
fails. Note that this last "feature" is pretty specific to your scenario @jzoldak, so while I didn't mind implementing it, do note that it's rather suboptimal... In my testing Boto would retry the PUT
quite a few times before giving up, so by not setting the bucket to read only explicitly some time is wasted by having Boto and pip-accel figure this out for themselves.
Can you confirm whether these changes resolve the caveats you mentioned?
I believe the issue you reported is fixed and the caveats you mentioned are also resolved, so I'm going to go ahead and close this issue now - I'm trying to keep an overview of active/relevant issues on GitHub and the long list isn't helping :-). If you believe there are still issues with the current implementation, feel free to reopen this issue or report a new one. Thanks for suggesting the feature!
Yes, I'm all set with this now. Thanks for all your work on this really great project.
S3 has the ability to set ACLs for anonymous access to buckets. This is wired into boto's s3.connection.S3Connection method via the anon parameter.
Modify the S3 cache backend code so that if the user does not have credentials, it will connect as an anonymous user rather than erroring with "No handler was ready to authenticate... Check your credentials" and disabling the S3CacheBackend.