search

zen-fs / core

A filesystem, anywhere
https://zen-fs.github.io/core/
MIT License
103 stars 14 forks source link

`Fetch`: Support HTTP Range Requests #53

Open james-pre opened 11 months ago

james-pre commented 11 months ago

@jvilk:

For large files (100MB+), downloading and storing the entire file in-memory is prohibitively expensive, and may not > be needed if only small parts of the file are needed.

It would be great to support HTTP range requests in both synchronous and asynchronous modes.

Some important questions:

  • How large should the chunks be? Should that be configurable?
  • Should we prefetch chunks? What prefetch policies are useful?
  • What configuration knobs should we expose to the user?
  • What eviction policy should we use? FIFO? How much data should we keep in-memory?
  • Do browsers cache data retrieved via HTTP range requests? This should make re-acquiring previously evicted blocks much cheaper / avoid draining the server over a long session.

I don't anticipate being able to work on this anytime soon, but I'm recording my thoughts here for now.

https://github.com/jvilk/BrowserFS/issues/219

anthonyhoegberg commented 1 week ago

What's needed for this to work? i might try it but im not quite familiar with the library yet so it might take time

james-pre commented 1 week ago

@anthonyhoegberg,

This requires some changes to the Fetch backend to support the underlying partial reading and writing of files and configuration for the option. Other than that, it is just a matter of sending the right HTTP headers. If you want to take a look at implementing the feature, I can help with any questions or comments you may have.