search

RsyncProject / rsync

An open source utility that provides fast incremental file transfer. It also has useful features for backup and restore operations among many other use cases.
https://rsync.samba.org
Other
2.72k stars 327 forks source link

Stats may be wrongly computed #7

Closed benrubson closed 4 years ago

benrubson commented 4 years ago

Hi,

In some situations, especially when transfer is aborted, stats are wrongly computed, at both sides.

Here's an example of a CTRL+C interrupted transfer :

# du -b /tmp/source/
3221229568    /tmp/source/

# rsync -a /tmp/source/ ...
sending incremental file list
/tmp/
/tmp/source/
/tmp/source/bf1
          1.07G 100%  147.89MB/s    0:00:06 (xfr#1, to-chk=2/5)
/tmp/source/bf2
          1.07G 100%  132.94MB/s    0:00:07 (xfr#2, to-chk=1/5)
/tmp/source/bf3
        521.34M  48%  124.39MB/s    0:00:04  ^C

# "cat receiver.log"
rsync: connection unexpectedly closed (146 bytes received so far) [generator]
sent 81 bytes  received 146 bytes  total size 3221225472
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.2.0dev]

Results : on receiver side, sent / received numbers are wrong.

And with SIGUSR2 :

# du -b /tmp/source/
3221229568    /tmp/source/

# rsync -a /tmp/source/ ...
sending incremental file list
/tmp/
/tmp/source/
/tmp/source/bf1
          1.07G 100%  149.53MB/s    0:00:06 (xfr#1, to-chk=2/5)
/tmp/source/bf2
          1.07G 100%  133.87MB/s    0:00:07 (xfr#2, to-chk=1/5)
/tmp/source/bf3
        527.70M  49%  125.91MB/s    0:00:04  
Number of files: 5 (reg: 3, dir: 2)
Number of created files: 5 (reg: 3, dir: 2)
Number of deleted files: 0
Number of regular files transferred: 3
Total file size: 3.22G bytes
Total transferred file size: 3.22G bytes
Literal data: 2.15G bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 0
Total bytes received: 0
sent 0 bytes  received 0 bytes  -0.00 bytes/sec
total size is 3.22G  speedup is -9,223,372,036,854,775,808nf

# "cat receiver.log"
rsync: connection unexpectedly closed (150 bytes received so far) [generator]
sent 81 bytes  received 150 bytes  total size 3221225472
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.2.0dev]

Results : on receiver side, sent / received numbers are wrong. Results : on sender side, many of the --stats numbers are wrong.

Thank you 👍

WayneD commented 4 years ago

The receiving side is measuring the stats in 2 procs until the very end, so I disabled early stats except from the sender.

benrubson commented 4 years ago

Thank you Wayne. I've updated my post above with some Results, and, as you can see, issue is not only on receiver side, but also on sender side, where the --stats numbers are wrong. Instead of simply disabling the stats if / when they are wrong, do you think the accounting process could be modified in a manner that stats would be sort of "real-time" computed, so that when transfer is aborted, numbers are accurate, and available at both sides ? Thank you again 👍

WayneD commented 4 years ago

If you're killing processes with internal-only signals, you have to live with the fact that you get crap in the results since the stats info hasn't been updated by the end-of-run data updating. The only way to make (at least some of) the stats constantly updated is a total re-architecture of the 3-process layout, which is something that I'm contemplating for the future but won't happen any time soon.