Using StringIO/BytesIO instead of tempfiles with S3 filesystem?

PyFilesystem / pyfilesystem

Python filesystem abstraction layer

http://pyfilesystem.org/

BSD 3-Clause "New" or "Revised" License

288 stars 63 forks source link

Using StringIO/BytesIO instead of tempfiles with S3 filesystem? #208

Open shoyer opened 9 years ago

shoyer commented 9 years ago

I tried using pyfilesystem's s3fs to process a large quantity of data from Amazon's S3 (more than fits on disk) and I found that operation failed, because I ran out of space for the temp files s3fs creates.

Would it make sense to have an option or change the default to load files directly into memory instead of writing them to disk?