search

MaxDaten / grunt-rsync

A gruntplugin which copies files to a (remote) machine with rsync.
MIT License
12 stars 8 forks source link

Just transfer deltas #5

Closed tarciozemel closed 11 years ago

tarciozemel commented 11 years ago

It would be more efficient if the transfer were limited to the deltas of the files (& new files, of course) instead the entire files/folders structure every time the task runs...

This default behaviour can be overwritten with some configuration or just with a core update is possible?

MaxDaten commented 11 years ago

Hi @tarciozemel ,

thanks for your comment.

this plugin only wraps and utilize the system rsync. i'm not an expert in all rsync options but rsync already works with a delta-algorithm.

Correct me, if i'm wrong, but it checks with the remote rsync the time stamp and size of the files and only transfers the mismatched ones. With --checksum you can alternate this behavior, to compare checksums.

Furthermore it uses file-deltas for the transfer, but this makes only sense, if the disk-speed outperform the network bandwidth. Because of this fact it is disabled for local transfers by default (maybe implementation specific). You can force file-deltas with --no-whole-file in the additionalOptions field.

To determine the deltas rsync traverse the directory structure of course, and it is necessary with the stateless behavior of rsync. The only way prevent the traversing is to save some states like git. Or you tar/zip the directory, transfer the archive with rsync and untar/unzip it on the other end.

Both is counterproductive in terms of using rsync, in my opinion. It's better to use already proofed tools than doing something on your own a way worse with only a small theoretical benefit. But i'm very open to discussions.

more about rsync: http://en.wikipedia.org/wiki/Rsync

tarciozemel commented 11 years ago

Nice, @MaxDaten, thank you for your answer! So, if I correctly undertood, the task already gets the benefits of rsync and transfer only the deltas? If yes, I'll make other tests, maybe in that time something was happening with my internet conection...

Thank you, again!

MaxDaten commented 11 years ago

It should in most cases, but with a pitfall: If you glob in the config for example with files: 'dist/**/*.jpg' grunt will enumerate the files if I remember it correctly. And this plugin will call rsync for each element in the enumeration. Yes this is effective but not efficient, I will take a look at it and how to improve this. Thanks to lead me to this part.

tarciozemel commented 11 years ago

Thank YOU for this great task! I'm anxiously waiting for the news!

Btw, if you can help me again, in the manual of rsync, we can find "delete extraneous files from dest dirs" explanation to the --delete option. But, when I put this in additionalOptions (to delete timestamped file names), nothing happens. Maybe I misunderstood, but what do you recommends me?

MaxDaten commented 11 years ago

i will take a look to the --delete option also, it will take some time for investigations too ;)

MaxDaten commented 11 years ago

sorry for the delay. this project has the clean option, but it is not documented, actually. I will update the doc. In my test-project this option is effectful. It adds --delete and --delete-after to the rsync args, and it worked for me.

MaxDaten commented 11 years ago

its documented now, if you have further issues please feel free to open an issue or comment here. maybe we will find out whats wrong.