In order to scale MakeDrive, we're going to need to convert our server code to use streams instead of reading whole files into memory (our current readFile/writeFile approach is memory bound). Ideally we'd like to be able to pipe an fs.ReadStream to the web socket (or via a through stream and process for checksum, diff, etc), and not have to load large files into memory in order to process. To do this, we need to modify the rsync code to work with streams, createReadStream and createWriteStream. This means Filer needs to learn how to do it, which means we have to add streaming to our provider API, too.
Obviously in browser providers like IndexedDB we can't really stream (i.e., we always read the whole file, at least currently until #10). That's OK, we can fake it via browserify. However, with node.js server side providers (e.g., S3) we can stream the data out, which is perfect.
Probably the best way to do this is to begin by porting the node.js tests for createReadStream and createWriteStream over to Filer, then tackle the implementation.
Streaming needs to be done in a way that doesn't hold an open transaction. I'd like to revisit the provider API as well. This might be a good time to refactor get and put.
In order to scale MakeDrive, we're going to need to convert our server code to use streams instead of reading whole files into memory (our current
readFile
/writeFile
approach is memory bound). Ideally we'd like to be able to pipe anfs.ReadStream
to the web socket (or via a through stream and process for checksum, diff, etc), and not have to load large files into memory in order to process. To do this, we need to modify the rsync code to work with streams,createReadStream
andcreateWriteStream
. This means Filer needs to learn how to do it, which means we have to add streaming to our provider API, too.Obviously in browser providers like IndexedDB we can't really stream (i.e., we always read the whole file, at least currently until #10). That's OK, we can fake it via browserify. However, with node.js server side providers (e.g., S3) we can stream the data out, which is perfect.
Probably the best way to do this is to begin by porting the node.js tests for
createReadStream
andcreateWriteStream
over to Filer, then tackle the implementation.For reference: