search

gokrazy / rsync

gokrazy rsync
BSD 3-Clause "New" or "Revised" License
494 stars 28 forks source link

Is it possible to support concurrent multi-file/multi-block transfering? #1

Closed yuzhichang closed 2 years ago

stapelberg commented 2 years ago

Take a look at the sendFiles implementation to understand how rsync file transfer works: https://github.com/gokrazy/rsync/blob/0ebc038f5c532d141ea52501c76be56d0d4278b0/internal/rsyncd/rsyncd.go#L261

rsync clients request files one after the other (no concurrency, but pipelining). I don’t know if individual chunks within the same file can be sent in any order, but either way, even when sending multiple chunks concurrently, you’ll eventually be limited by the throughput of the TCP connection.

Are you asking because there are concurrency features in other implementations? If so, which ones?

yuzhichang commented 2 years ago

I wish concurrent rsync for long time. I don't find good implementation yet. FYI, I wrote a sftp client in Golang which upload/download a file concurrently(establish multiple sftp connections. Looks like lftp pget -n), which speedup transfer much for slow network.

stapelberg commented 2 years ago

I wish concurrent rsync for long time. I don't find good implementation yet.

Okay :)

The goal of this project (gokrazy/rsync) is to be compatible with what other rsync implementations do, so I’ll close this issue.

Perhaps the code is useful to you regardless, if you decide to build a concurrent rsync version…