Support for rsync

[jovandeginste]

Since you now support scp, I think rsync would be a logical next step :-)

[drakkan]

For SCP I implemented the (undocumented) SCP protocol.

In OpenSSH basically an scp local command is forked for every new scp request and it receive protocol commands and data from the remote scp sender via ssh. I guess rsync works the same way.

I have no problem to do the same inside sftpgo (I already implemented this in my first scp experiments) but executing an external command give no control on what happen, for example which files or directory are created, or if a symlink is pointing outside the home dir so I think this easy method, that works for each system command over ssh, cannot be used here.

The alternative is to implement the rsync protocol too. While for scp I have not found any library, for rsync seems there are some libraries available, but I guess some modifications are needed to be used here, so this is not trivial

[jovandeginste]

Agreed, let's get this project to release 1.0.0 first, maybe (since this might be a large feature).

[drakkan]

I did some tests:

using a command like this one:

 rsync -avvvvvvvv -e "ssh -p 2022" /home/nicola/test.sh nicola@127.0.0.1:/

I get this exec command over ssh rsync --server -vvvvvvvvlogDtpre.iLsfxC . /. So now we have to emulate the behaviour of this command.

After the exec command the rsync client sends these bytes

1f 00 00 00

since 1f=31 this is the protocol version.

We need to answer with the server protocol version, for example if we want to emulate protocol version 31 we need to send something like this:

1f 00 00 00

after the version we need to send this 5 (magic) bytes, probably a serialized c struct:

3f 94 8b b4 5d

after this the rsync client prints:

(Client) Protocol versions: remote=31, negotiated=31
FILE_STRUCT_LEN=24, EXTRA_LEN=4

and then sends the file lists. This is probably another serialized c struct, I can read file names inside received bytes.

rsync CLI does not have documentation about this. There is documentation (and several Go libraries) that implement the rsync protocol but no documentation about rsync CLI itself.

I suppose that if we are able to deserialize files list (and any other rsync CLI specific exchange) we can use one of the existing Go libraries to apply delta and handle rsync protocol itself.

I'm not motivated enough to read the rsync CLI source code and understand its protocol, anyway if someone can help providing some decent documentation I'll implement rsync support myself. Thanks

[drakkan]

as rsync alternative you could consider rclone (https://rclone.org/) or restic (https://restic.net/)

These commits make sftpgo compatibile with restic:

https://github.com/drakkan/sftpgo/commit/08e85f6be9e7fa73e30f4bf5563f70a279868cf0 https://github.com/drakkan/sftpgo/commit/5be1d1be699cd0cbe97218a42d1b011197df900f

This commit improve rclone compatibility

https://github.com/drakkan/sftpgo/commit/ca6cb34d9870b2408c8510df7a639078a657bf9c

[drakkan]

Now that we support system commands over SSH add rsync, executing the system command, is really easy, here is a patch:

0001-add-rsync-support.zip

someone would like to test it?

The patch simply add rsync to the list of the supported SSH commands and add the option --safe-links if not set, so links outside the tree are not recreated. rsync has a lot of options and I'm not sure that there is no way to escape the chroot, this should be carefully tested.

As documented for the other supported system commands quota check is suboptimal: we only see the bytes that the remote command send to the local command over SSH and we check quota size based on these bytes, but this check is wrong, for example:

1) the bytes we see contain both files and protocol commands, so the size of the files is different from the total transferred size 2) rsync support compression, so the bytes we see could be smaller than the real files size 3) a protocol command (few bytes) can delete a big file and we have no way to check this

To mitigate these issues quotas are recalculated at the command end with a full home directory scan.

Maybe in future we can use some filesystem notifications library (for example https://github.com/fsnotify/fsnotify) to get notified about file creation/deletion/overwrite but it should be able to recursively watch a directory tree including the new created dirs

[jovandeginste]

I'm busy with some projects at the moment, I plan to test this later on.

[drakkan]

No hurry, thanks for your time.

It seems to work, the only problem is the quota management. Maybe when we use system commands is better to periodically scan for quota using the REST API and do custom actions (for example disable the user if it is over quota)

[drakkan]

Merged, If you find bugs or have ideas to improve quota management please comment here and/or open separate issues, thanks!

[FrancYescO]

Can someone help me on getting rsync working? i was using the -slim tag on docker that later i've read it lacks the rsync support, so i moved to the latest but i'm still getting on the client

[...]
Authenticated to 172.11.0.11 ([172.11.0.11]:2022) using "publickey".
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: network
debug1: Sending environment.
debug1: channel 0: setting env LANG = "C.UTF-8"
debug1: Sending command: rsync --server -logDtpre.iLsfxCIvu . /
exec request failed on channel 0
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(231) [sender=3.2.7]

am i still missing something in the docker compose to enable rsync or something like that?

[drakkan]

Can someone help me on getting rsync working? i was using the -slim tag on docker that later i've read it lacks the rsync support, so i moved to the latest but i'm still getting on the client

[...]
Authenticated to 172.11.0.11 ([172.11.0.11]:2022) using "publickey".
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: network
debug1: Sending environment.
debug1: channel 0: setting env LANG = "C.UTF-8"
debug1: Sending command: rsync --server -logDtpre.iLsfxCIvu . /
exec request failed on channel 0
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(231) [sender=3.2.7]

am i still missing something in the docker compose to enable rsync or something like that?

make sure the rsync command is enabled in your configuration. It is disabled by default

[FrancYescO]

Thanks for the reply seems what i was searching for was the docker env (and i have the doubt that i should add all the other default values more than just rsync) SFTPGO_SFTPD__ENABLED_SSH_COMMANDS=rsync

i suggest to at least replace the in readme something like: SCP and rsync (must be enabled) are supported.

[FrancYescO]

did something changed recently? i'm using the latest docker image and i'm getting on start unsupported ssh command: "rsync" ignored