RomVault / RVWorld

RomVault World
http://www.romvault.com
Apache License 2.0
83 stars 20 forks source link

Use compressed size in dat file for maximum zip file match #5

Open dmxraj opened 4 years ago

dmxraj commented 4 years ago

In a large zip file there are multiple files (roms, pngs etc). All these files are stored at some offset inside the zip file.

Example: Large.zip size 10GB File Compressed size F1.png 1000 F2.png 1500 F3.png 4000 F4.png 2000 F5.png 1000 . . . . F10000.png 8332

If some one has this zip file. And the first file F1.png is missing. Then torrent client marks the whole file to be downloaded again, since the torrent hash changes completely for all the chunks in zip file. This is how torrent works. Download the whole file wastes lots of bandwidth. This problem is very prominent in "MAME 0.xxx EXTRAs"

My solution is to fill the missing files (using file Compressed size) inside the zip file with dummy data (all zeroes). This way torrent will only download those chunks inside the zip file which has dummy data. Torrent will declare that it already has the remaining chunks of the zip file.

These Compressed Size can be a part of dat file: Example:

005
dmxraj commented 4 years ago

I have tested Compressed size method and here are the results:

I had EXTRAs version .196 from pleasuredom I applied RomVault on it using DAT of version .214. When I joined the torrent, only 6GB content matched out of 50GB. I fixed all zip files using Compressed size method. Now 26 GB matched out of 50GB.

gjefferyes commented 4 years ago

This is something I have also given some thought to. It would need the DAT to store the expected trrntzip compressed size, and then would have to make some way to make a ZIP with this padding. I would really like this to be done in a way that it did not make a corrupt zip.