martymac / fpart

Sort files and pack them into partitions
https://www.fpart.org/
BSD 2-Clause "Simplified" License
231 stars 39 forks source link

dont treat rsync warnings as errors #61

Closed milahu closed 2 months ago

milahu commented 3 months ago
$ cat ~/.ssh/config
Host example.com
  LocalForward 3030 127.0.0.1:3030

$ fpsync src/ example.com:dst/
1720357928 <=== Fpsync completed with errors in 1s, see logs:
/tmp/fpsync/log/1720357926-3287005/1.stderr

$ cat /tmp/fpsync/log/1720357926-3287005/1.stderr
bind [127.0.0.1]:3030: Address already in use
channel_setup_fwd_listener_tcpip: cannot listen to port: 3030
Could not request local forwarding.

in this example, port forwarding fails because there is already a ssh session

martymac commented 3 months ago

Hello @milahu,

I presume you are using fpsync's '-w' option. In that mode, every job is run on a remote machine using a dedicated ssh connection.

Enabling port forwarding for each connection will not work as-is and display errors because the local port will be in use after the first successful ssh connection. Thus, next ssh connections running in parallel will not be able to bind to the same port again.

You may want to play with ssh options either to silent those errors (LogLevel option) or maybe (not tested) by using multiplexing (ControlMaster / ControlPath) ?

milahu commented 2 months ago

no fpsync -w here, just fpsync -v -n 4 -f 100 -m rsync

a workaround is to add LogLevel FATAL to ~/.ssh/config

currently fpsync treats any output to stderr as error, and exits nonzero but fpsync should return zero after a successful sync

martymac commented 2 months ago

Hello milahu,

I managed to reproduce the problem with a command like :

$ fpsync -n 2 /src/ localhost:/dst/

to use rsync over ssh.

Indeed, fpsync currently detect errors using stderr output only. I'll see if I can track return codes to improve error detection. Stay tuned!

martymac commented 2 months ago

Hello milahu,

Could you try the fix above please ? It should fix your problem.

Best regards,

Ganael.