Send files to downloads without client.createServer or file.blob

[ddelange]

What version of this package are you using?

Want to migrate to v2

What problem do you want to solve?

Send multiple files from a torrent to Downloads (or other local folder) without having the complete files in memory.

This came up in https://github.com/blenderskool/blaze/pull/174 and I couldn't find an a suitable pattern in your docs.

Currently, there is file.streamURL (needs client.createServer, seems like too much overhead) and file.blob (needs whole file in memory, prohibitive).

How would a client best "copy a downloaded torrent" from webtorrent local storage to the user's local file system? Ideally including folder structure, or alternatively as a bunch of flat files in the Downloads folder?

What do you think is the correct solution to this problem?

Are you willing to submit a pull request to implement this change?

[SilentBot1]

Hi @ddelange

As documented in our API documentation page here, there are three ways to stream a file, which avoids having it all in memory:

• StreamX Readable Stream with StreamX documentation here.
• W3C ReadableStream
• async iterator

In regards to fixing https://github.com/blenderskool/blaze/issues/163 in https://github.com/blenderskool/blaze/pull/174, if you're fine prompting the user where they would like to save the files, I believe you can use fsa-chunk-store, and pass through the output of await window.showDirectoryPicker() as the rootDir argument, but keep in mind this is heavily polyfilled on Safari due to fileHandle.createWritable() not being implemented in Safari yet, though @ThaUnknown may be able to better comment on this approach.

If you do not wish to prompt for folder access, but still want to replicate the file structure, the only way I can think of is via a zip or tarball.

[ddelange]

hi @SilentBot1 :wave: thanks for the quick and detailed response! that's enough to go on for now, so I'll close with that. cheers!

[ThaUnknown]

wah I've been pinged,

at it's core webtorrent files expose async iterators, which are a low level, low overhead data streams which is likely what you're looking for, those are then transformed into other data formats using utility methods like blob, arrayBuffer, stream and createReadStream. Those all don't store any excess data in memory, as they are streams.

The real problem for you is like silent mentioned is saving said streams, by far the best method is to simply tell webtorrent to use a directory to store data via window.showDirectoryPicker and client.add rootDit, and webtorrent will re-create the file's structure there, however this is only supported by chrome.

The simplest method would be to use await file.blob() and then download that, but if the file is too big the browser will crash and run oom, if you're working with 10 GB+ files, the only cross browser compatible way i can think of is using client.createServer() which requires a custom service worker, and then downloading the file via HTMLAnchorElement.download and setting the href to new URL(file.streamURL) and running .click()

createServer overhead is actually very negligible, it's incredibly efficient