search

abashev / vfs-s3

Amazon S3 driver for Apache commons-vfs (Virtual File System) project
Apache License 2.0
93 stars 50 forks source link

S3 files are copied to local tmp dir then streamed through VFS #34

Closed themartynroberts closed 8 years ago

themartynroberts commented 8 years ago

Although the VFS abstraction works with streams this library appears to use an intermediary file stored locally in the temp directory. For example when obtaining an input stream, the S3 file is first downloaded into the temp directory then an input stream on that temp file is opened. See S3FileObject.doGetInputStream() which in turn calls S3FileObject.downloadOnce().

Is there any plan to change this such that files are streamed to and from S3 directly without the temp file step?

Direct streaming is what we have in the VFS http/https providers, see HttpFileObject.doGetInputStream(). Without this there are issues with handling large files.

abashev commented 8 years ago

HttpFileObject is uploading whole response into memory and provides just ByteArrayInputStream. Because we have to handle huge concurrent load with this library then we can't fill memory with all resolved files.