search

RsyncProject / rsync

An open source utility that provides fast incremental file transfer. It also has useful features for backup and restore operations among many other use cases.
https://rsync.samba.org
Other
2.83k stars 331 forks source link

Broken pipe issue. #449

Open huangyingw opened 1 year ago

huangyingw commented 1 year ago

Bellowing is the running and output. and passwordlessly ssh to mini work. could you help me to debug what happen there?

rsync -aHv --progress ~/transmission_download/movies/ mini:/media/mapper/mirror2/movies/ 
Warning: Permanently added 'mini' (ED25519) to the list of known hosts.
sending incremental file list
./
movie.mp4
    195,035,136   3%   93.03MB/s    0:00:56  client_loop: send disconnect: Broken pipe

rsync: [sender] write error: Broken pipe (32)
rsync error: unexplained error (code 255) at io.c(848) [sender=3.2.7-80-g509b6b59]
WayneD commented 1 year ago

See https://rsync.samba.org/issues.html for debugging steps.

KES777 commented 5 days ago 
debug1: Sending command: rsync --server -vlogDtpAXrxe.iLsfxCIvu --log-format=%i --delete --append --append . /media/kes/B4\\ EMPTY/balkon/sdc/sdc
sending incremental file list
<f.stp.g... sdc
962.408.775.680  48%  112,95MB/s    2:29:34  packet_write_wait: Connection to yyy port xxx: Broken pipe
1.765.917.753.344  88%   80,62MB/s    0:47:20  
rsync: [sender] write error: Broken pipe (32)
rsync error: unexplained error (code 255) at io.c(848) [sender=3.2.7]
Oct 15 19:52:55 work systemd[1]: Condition check resulted in fast remote file copy program daemon being skipped.
░░ Subject: A start job for unit rsync.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit rsync.service has finished successfully.
░░ 
░░ The job identifier is 197.

kes@work /var/log $ sudo systemctl status rsync
○ rsync.service - fast remote file copy program daemon
     Loaded: loaded (/lib/systemd/system/rsync.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Tue 2024-10-15 19:52:55 EDT; 4 days ago
             └─ ConditionPathExists=/etc/rsyncd.conf was not met
       Docs: man:rsync(1)
             man:rsyncd.conf(5)

Oct 15 19:52:55 work systemd[1]: Condition check resulted in fast remote file copy progra>

On both sides I have rsync 3.2.7

Now I removed all -v options. I added --timeout=28000 and changed to --del. Will post here what was changed.

UPD

debug1: Sending command: rsync --server -vlogDtpAXrxe.iLsfxCIvu --log-format=%i --delete --append --append . /media/kes/B4\\ EMPTY/balkon/sdc/sdc
sending incremental file list
<f.stp.g... sdc
962.248.736.768  48%  113,94MB/s    2:28:17  packet_write_wait: Connection to yyy port xxx: Broken pipe
1.765.899.665.408  88%   80,17MB/s    0:47:36  
rsync: [sender] write error: Broken pipe (32)
rsync error: unexplained error (code 255) at io.c(848) [sender=3.2.7]

UPD It looks like connection was restored: image I see broken pipe error, but but it still checking the sum of file. Let's see what happened when already the copied file size will be reached: 1765961465856 Oct 20 00:33 sdc.

UPD

+ sudo rsync -e 'ssh -p yyy -i /home/kes/.ssh/keshome' -i --del -xaXA --copy-devices -D --append-verify --partial --inplace --progress --timeout=28000 /dev/sdc 'xxx:/media/kes/B4 EMPTY/balkon/sdc/sdc'
sending incremental file list
<f.stp.g... sdc
955.245.166.592  47%  114,38MB/s    2:28:43  packet_write_wait: Connection to xxx port yyy: Broken pipe
1.765.935.054.848  88%   80,77MB/s    0:47:14  ^[
rsync: [sender] write error: Broken pipe (32)
rsync error: unexplained error (code 255) at io.c(848) [sender=3.2.7]

It looks very suspicious to fail at the same place every time. Now added: --msgs2stderr -M--msgs2stderr options.

UPD

+ sudo rsync -e 'ssh -p xxx -i /home/kes/.ssh/keshome' -i --del -xaXA --copy-devices -D --append-verify --partial --inplace --progress --timeout=28000 --msgs2stderr -M--msgs2stderr /dev/sdc 'xxx:/media/kes/B4 EMPTY/balkon/sdc/sdc'
sending incremental file list
<f.stp.g... sdc
960.969.211.9packet_write_wait: Connection to xxx port xxx: Broken pipe
1.765.914.607.616  88%   80,26MB/s    0:47:32  
rsync: [sender] write error: Broken pipe (32)
rsync error: unexplained error (code 255) at io.c(848) [sender=3.2.7]

UPD After changes to SSH configuration: https://unix.stackexchange.com/a/327653/129967 It is able to reach current file size for sum calculations:

sending incremental file list
<f.stp.g... sdc
  9.465.528.320   0%  135,20MB/s    4:00:43  
111.411.224.832   5%  134,97MB/s    3:47:47  
1.765.839.896.576  88%   78,74MB/s    0:48:46

After this it is started to calculate sum on the remote host. Curious, why checksum calculation is not done simultaneously?

KES777 commented 4 days ago

Seems fixing connection issue with SSH resolved my problem:

+ sudo rsync -e 'ssh -p xxx -i /home/kes/.ssh/keshome' -i --del -xaXA --copy-devices -D --append-verify --partial --inplace --progress --timeout=28000 --msgs2stderr -M--msgs2stderr /dev/sdc 'xxx:/media/kes/B4 EMPTY/balkon/sdc/sdc'
[sudo] password for kes:         
sending incremental file list
<f.stp.g... sdc
  9.465.528.320   0%  135,20MB/s    4:00:43  
111.411.224.832   5%  134,97MB/s    3:47:47  
1.765.839.896.576  88%   78,74MB/s    0:48:46  
1.779.010.338.816  88%    2,46MB/s   24:24:14