search

DamonOehlman / filestream

W3C File Reader API streaming interfaces
30 stars 11 forks source link

Streaming: Don't load the entire file into memory #17

Closed feross closed 9 years ago

feross commented 9 years ago

Before this change, FileReadStream would read the full file into memory with a single readAsArrayBuffer() call.

This commit calls readAsArrayBuffer() with a slice of the file blob for each call to _read(). This means the blob is never fully loaded into memory.

The same FileReader object is reused, just with different slices of the file blob passed into readAsArrayBuffer().

Also, this eliminates the event named 'readable' which was causing bugs when I piped this stream into another stream library.

I also added browser tests for FileReadStream.

Run the tests in a local browser with npm run test-local. Run the tests remotely with a free sauce labs account with npm test. Also, if we set up travis, it can run the tests automatically for us. :-)

DamonOehlman commented 9 years ago

Thanks heaps mate - I've been meaning to do this for some time, but keep getting distracted with other things. I'll merge this change and I've made you a collaborator (and also give you npm publish access) for future releases.

DamonOehlman commented 9 years ago

@feross I made the decision to bump this to version 4.0.0 as because I hadn't had tests for this prior (thanks for introducing them - I'll get Travis integrated soon) if there were any changes that affected people with the new slice behaviour it would be best if they intentionally upgraded to filestream@4 rather than it just happening.

@nathanoehlman This is behaviour I was talking to you about looking into for the stuff you have been using filestream for. I would recommend upgrading to this new version and seeing if you notice any improvements :)

feross commented 9 years ago

Nice! Thanks for publishing this so quickly :-)