Does not work if the target file system lacks support for (some?) metadata

[ppenguin]

Copying files from a mounted ntfs disk to a remote smb share fails for every file, and leaves behind all target files as 0 byte files.

Likely the same underlying issue as here: https://github.com/sxyazi/yazi/issues/1775

But I find it hard to believe that it's not (easily) possible to make a copy operation in rust behave the same as one in c (or a glibc function call for that matter?)

[wykurz]

hey, thanks for reporting! how can I repro this? (I have an access to wsl if that helps)

fyi rcp uses libc. the exact function that is called under the covers for copying file data is copy_file_range: https://man7.org/linux/man-pages/man2/copy_file_range.2.html it's somewhat new -- the system call was added in 2016. that and also we apply file metadata changes after the copy so I suspect it could be that system call and it not being supported that's the cause of this issue.

the copy_file_range is a very efficient way to copy files however, if I can get a repro of what you're seeing I'm happy to take a look if we can provide an alternative method for file sytems that don't support it (or maybe there's a different reason for this failure).

[wykurz]

PS. I read through the issue you linked. I'd be interested in a repro for this. fwiw I am considering using lower-level APIs instead of some library calls being made today to further reduce the number of times we try to read/write file metadata when performing copies, it'd be nice if by doing that we can also address this issue.

[ppenguin]

The conclusion I arrived at is that low level (i.e. also copy_file_range) doesn't work because it doesn't handle metadata (correctly). From what I saw the typical libs used by many/most in rust don't handle metadata sophisticated enough to work reliably across different file systems (i.e. non-supported attributes lead to a hard fail). That's likely the reason why Linux's cp command (from coreutils) is several 1000 loc and not just a simple libc call.

A repro I found to work pretty reliably: use an ntfs drive as source and a Linux smb share (e.g. a NAS) as a target. I'd hazard a guess that anything that in cp gives any type of permission/attribute errors (which are non-fatal in cp) on the target will fail in anything that uses fs::copy or io::copy or related.

Pretty serious if you ask me, because IMO a show stopper for any copy utility.

[wykurz]

I have tried doing just that but didn't see any issues.

I setup an NTFS source (over the 9p WSL 2.0 protocol) and an SMB share:

> mount -l
...
C:\ on /mnt/c type 9p (rw,noatime,dirsync,aname=drvfs;path=C:\;uid=1000;gid=1000;symlinkroot=/mnt/,mmap,access=client,msize=65536,trans=fd,rfd=5,wfd=5)
//localhost/share on /mnt/windows-share type cifs (rw,relatime,vers=3.0,cache=strict,username=mateusz,uid=1000,noforceuid,gid=1000,noforcegid,addr=127.0.0.1,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1)

I'm able to copy files back and forth between the two w/o any problems.

Let me know if you know of a different way to reproduce that?

[ppenguin]

Ah, that looks interesting... TBH I don't really have a solid idea, but intuitively things that might lead to different behaviour are e.g. noforceuid,nounix,mapposix.

In my case I just mounted the SMB share via nemo that uses gvfs, so I didn't do anything to "tune" the mount options. But I suspect that such options may improve permission mapping and such, which might make the setting of the metadata after copying actually work without error.

On the source side I mounted via ntfs3g, which may also behave differently?

Let me know if you know of different way to reproduce that?

I could imagine that your mount commands are "better integrated" with windows (since you're using WSL I guess), and that may be the reason there's no meta data issue in the first place? FYI, my SMB mount has much less options, maybe that could go some way for your repro (to make it "bad behaved"):

> , gio info smb://192.168.1.84/public
display name: public on 192.168.1.84
edit name: /
name: /
type: directory
size:  0
uri: smb://192.168.1.84/public/
local path: /run/user/1000/gvfs/smb-share:server=192.168.1.84,share=public
unix mount: gvfsd-fuse /run/user/1000/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=100

FWIW: the yazi issue was solved by using std::io::copy() for the data only and copying the metadata/permissions separately and allow the latter to fail. (See linked issue)

[wykurz]

Are you able to verify if this PR addresses your issue? https://github.com/wykurz/rcp/pull/27

I'm not comfortable moving to a blocking thread always, that could have significant performance implications and I don't have time to test that right now. However, I think it should be OK as a fallback option if tokio::fs::copy fails.

If that PR doesn't work -- would you mind running rcp -vvv ... and sharing the output?