filerjs / filer

Node-like file system for browsers
BSD 2-Clause "Simplified" License
619 stars 154 forks source link

Add support for fs.createReadStream(), fs.createWriteStream() #264

Open humphd opened 10 years ago

humphd commented 10 years ago

In order to scale MakeDrive, we're going to need to convert our server code to use streams instead of reading whole files into memory (our current readFile/writeFile approach is memory bound). Ideally we'd like to be able to pipe an fs.ReadStream to the web socket (or via a through stream and process for checksum, diff, etc), and not have to load large files into memory in order to process. To do this, we need to modify the rsync code to work with streams, createReadStream and createWriteStream. This means Filer needs to learn how to do it, which means we have to add streaming to our provider API, too.

Obviously in browser providers like IndexedDB we can't really stream (i.e., we always read the whole file, at least currently until #10). That's OK, we can fake it via browserify. However, with node.js server side providers (e.g., S3) we can stream the data out, which is perfect.

Probably the best way to do this is to begin by porting the node.js tests for createReadStream and createWriteStream over to Filer, then tackle the implementation.

For reference:

modeswitch commented 10 years ago

Streaming needs to be done in a way that doesn't hold an open transaction. I'd like to revisit the provider API as well. This might be a good time to refactor get and put.