S3 expose module

[GoogleCodeExporter]

I have altered an MIT licensed implementation of S3 called mock-s3 to function 
as a pyfs expose module.  I copied http.py's format to create the __init__.py.  
You can find my work here:

https://github.com/ClashTheBunny/mock-s3/tree/pyfs_expose_mocks3

(mock-s3 branch pyfs_expose_mocks3).

do:
 cd pyfs/fs/expose
 git clone https://github.com/ClashTheBunny/mock-s3 -b pyfs_expose_mocks3 mocks3

and now you can run something like __init__.py.

It requires:
 redis (and a redis-server) or mockredispy
and
 jinja2.

It needs some cleaning up and a much better readme, but as it is, it works 
fairly well on my localhost with the default parameters and the test scripts if 
I add mocking.localhost to my /etc/hosts file and fake AWS credentials.

TODO would include things like real authentication and such, but I'm not sure 
where this will go, so we'll see about what would be priority.

What would need to be done to get this in the next release?

Original issue reported on code.google.com by rand...@mason.ch on 21 Jul 2014 at 6:19

[GoogleCodeExporter]

Here is the output from svn diff for the mocks3 changes for people who don't 
use much git.

Original comment by rand...@mason.ch on 21 Jul 2014 at 8:44

Attachments:

• mocks3.diff

[GoogleCodeExporter]

Also, I updated it so that it has a __main__.py so that if you install pyfs in 
your path (or add it via environment variables) it is good enough just to run:
python -mfs.expose.mocks3
and set up dns and authentication.

Original comment by rand...@mason.ch on 21 Jul 2014 at 8:52

[GoogleCodeExporter]

So this serves an fs with the S3 protocol? I guess it's for unit testing 
purposes?

Original comment by willmcgugan on 21 Jul 2014 at 8:54

[GoogleCodeExporter]

It was originally designed for that, yes.  And I think it would still be very 
useful for that, specifically if you use the in-memory file system provided by 
pyfs with mockredis so that no other services were required and everything was 
stateless.

One other possibility is to use TahoeLAFS as a back end for s3 services.  Not 
that it would be fast, but it could be resilient and cheap in a way that other 
s3 services wouldn't be.

Specifically if a camlistore user wanted to distribute his or her data across a 
friend net, this may be eventually useful once authentication is included.

Original comment by rand...@mason.ch on 21 Jul 2014 at 9:06

[GoogleCodeExporter]

It's a cool project, but I don't think it's appropriate to get it in at PyFS 
release. We want to keep it as general purpose as possible, and this is quite a 
specialised use case. Best you host it in your own repos. I would like to tweet 
about it though.

Original comment by willmcgugan on 21 Jul 2014 at 9:33

[GoogleCodeExporter]

I'll try to get into the upstream project and then I'll let you know.

Original comment by rand...@mason.ch on 21 Jul 2014 at 2:06