search

wchang22 / LuminS

A fast and reliable alternative to rsync for synchronizing local files written in Rust
MIT License
359 stars 17 forks source link

Diffing Algorithm #13

Open wchang22 opened 5 years ago

wchang22 commented 5 years ago

Implement file diffing such that:

naturallymitchell commented 5 years ago

only the differences are copied

you could use patches generated by diff-rs and applied using patch-rs

hashes files with the same modification time and size

you might be able to get something out of checksumdir

uazu commented 5 years ago

I use rsync locally for updating a whole-filesystem copy on an external ZFS drive (with certain excluded paths), before snapshotting it in ZFS. So for that kind of application, the aim would be to minimize the number of blocks written.

So a simple streaming byte-by-byte or block-by-block comparison between old and new versions would be fine for that. No need for diff or running-CRC. But if one day you want this to work remotely over the network, then you're going to need rsync-style running-CRC.

matu3ba commented 3 years ago

I use rsync locally for updating a whole-filesystem copy on an external ZFS drive (with certain excluded paths), before snapshotting it in ZFS. So for that kind of application, the aim would be to minimize the number of blocks written.

So a simple streaming byte-by-byte or block-by-block comparison between old and new versions would be fine for that. No need for diff or running-CRC. But if one day you want this to work remotely over the network, then you're going to need rsync-style running-CRC.

You need to keep both versions in cache for that, which slows you down significantly.