search

jszwec / s3fs

S3 FileSystem (fs.FS) implementation
MIT License
177 stars 20 forks source link

Add Seeker interface to file #5

Closed HolzeHan closed 2 years ago

HolzeHan commented 2 years ago

This PR implements the optional seeker interface for file

jszwec commented 2 years ago

Before I start reviewing, my first question is - why? What's the use case?

Each Seek call essentially makes another call to s3 which costs you money. There are a lot of functions that can take advantage of seeker behind the scenes, potentially making your aws bill higher.

It also introduces some issues, such as you not closing the body while calling seek (if it's opened) thus leaking go routines.

HolzeHan commented 2 years ago

Currently I am streaming 10GB files from s3 into an application. The file has a streamable format, so it can be read from most positions. It would therefore speed up my processing time, if I could seek in the file. In addition I want to add a wrapper around a fs which allows to stream data using multipart download. For that I need the fs to support the seek operation (every part needs to seek to its starting position and read from there). This would speed up the streaming from around 30MB/s to somewhere around 200MB/s

Leaking go routines is a good point. I'll check again with that in mind

jszwec commented 2 years ago

LGTM thanks!