search

xuset / planktos

Serving websites over bittorrent
https://xuset.github.io/planktos/
MIT License
538 stars 19 forks source link

Copy identical pieces from older snapshots to the newly updated snapshot #46

Open xuset opened 7 years ago

xuset commented 7 years ago

When an update is found a new snapshot is created, the snapshot is downloaded from scratch despite the high likely that older snapshots already contain some of the needed data. This should be fixed by iterating over old snapshots and seeing if the old and new snapshot have any pieces in common by comparing the piece's hash. If the pieces are identical then copy the piece from the old snapshot's chunk store to the new.

georgeaf99 commented 7 years ago

The piece hashes for the old and new torrent should be diff-ed and then copied over to the new store.

georgeaf99 commented 7 years ago

We should copy pieces instead of packages

georgeaf99 commented 7 years ago

Post-Mortem:

I added the code to copy identical files or zeroes from old snapshots into the IndexedDB store used by WebTorrent. However, the stream that we use to read files from the store depends on uninitiated pieces having no value in the store, thus requiring us to rethink how to handled uninitialized pieces. Additionally, files in the store are not piece aligned, which would likely require us to adapt the createTorrent package.

These issues raised the complexity and difficulty of this task past the opportunity cost of pursuing this issue now. We will have to revisit this task later when it becomes a blocker for adoption.