Optimal settings for mkv to m2ts patch generation?

sisong / HDiffPatch

a C\C++ library and command-line tools for Diff & Patch between binary files or directories(folder); cross-platform; runs fast; create small delta/differential; support large files and limit memory requires when diff & patch.

Other

1.5k stars 279 forks source link

Optimal settings for mkv to m2ts patch generation? #388

Closed sgmihai closed 6 days ago

sgmihai commented 1 month ago

Hello. Trying to keep a backup of the original bluray .m2ts file after remuxing to .mkv. I was wondering, what is the theoretical best setting to get the smallest diff file for this scenario ?

For -m, readme says "requires (newFileSize+ oldFileSize5(or 9 when oldFileSize>=2GB))+O(1)" In this case, for a 26GB m2ts file, not even 256GB RAM would be enough?

If so, what is the next best thing ? I tried with "-s-64 -SD -c-zstd-21-24" and got a 531MB patch, which I believe is a bit too much. With xdelta3 I could have gotten something like ~360MB (estimate) but it stops finding matching data after a while, and can't change window size above 2GB. Any other tools to check for this purpose ?

sisong commented 1 month ago

in order to diffFile smaller, You can try:

-s change the min match length 64 to smaller, such as -s-16, etc., and the diff speed will be slower; Which value is best depends on data compressibility.
-c-zstd can change the compress dictionary 16MB increase to 256MB, such as -c-zstd-21-28, or change the compress level 21 to 22 (diff speed will be slower); Alternatively, it can be changed to -c-lzma2-9-256m (patch speed will be slower)

I don't know how you got the diffFile size of xdelta3; As far as I know, xdelta3 often generates huge diffFile when input huge files.

sgmihai commented 1 month ago

I retesed with -s-16 and -c-lzma2-9-256m, new diff size is 381MB, but took almost 10 hours to finish, on usb ssd. Took less than 10 minutes with xdelta3, too bad I can't increase window size.. Now I need to test -m mode. I have a machine with 256GB RAM, maybe with a bit of swap it will fit.

sisong commented 1 month ago

"took almost 10 hours to finish"
This is indeed slow, try it with -s-24 or -s-32, it will be a little faster.
（You can use smaller similar datas to try to found the better combination of parameters. ）
"I have a machine with 256GB RAM, maybe with a bit of swap it will fit."
That's max memory requires of diff algorithm, maybe not needed used 256GB memory in general practice.
(recommended to run with parallel threads with parameter like "-p-8")
-m default is -m-6, for binary data, you can try -m-4 etc. , maybe can got smaller diffFile.
(this parameter hardly affects the diff speed)