rsync stopped working after the recent update (only Android 7)

[tigran123]

This is quite a critical issue, because I rely on Termux to provide synchronoisation of all my major databases (200GB+) between desktop and mobile.

On Android 5.1.1 everything is working fine, just as before.

But on Android 7 I get these problems:

$ rsync --delete --size-only -rvW . umi:sd/Books/Religions/Bible/Hebrew/
sending incremental file list
tnk.pdf
rsync: rename "/data/data/com.termux/files/home/sd/Books/Religions/Bible/Hebrew/.tnk.pdf.cV0VoX" -> "tnk.pdf": Permission denied (13)

sent 8,084,033 bytes  received 178 bytes  3,233,684.40 bytes/sec
total size is 3,353,040,691  speedup is 414.76
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1]

Now, let's see if the file was actually transferred:

$ md5sum tnk.pdf
d785319291cd0d6a4c096abcef0c8ef8  tnk.pdf
$ ssh umi md5sum sd/Books/Religions/Bible/Hebrew/tnk.pdf
f4bb285d1421d86d57f4ec29df882e4a  sd/Books/Religions/Bible/Hebrew/tnk.pdf

No, as you see, the checksums do not match, so the file was not transferred.

Now, let's login to the mobile and check the permissions manually:

$ ssh umi
Welcome to Termux!

Online help:     https://termux.com/help
Community forum: https://termux.com/community
IRC channel:     #termux on freenode
Gitter chat:     https://gitter.im/termux/termux
Mailing list:    termux+subscribe@groups.io

Search packages:   packages search <query>
Install a package: packages install <package>
Upgrade packages:  packages upgrade
Learn more:        packages help
$ cd sd/Books/Religions/Bible/Hebrew/
$ touch new
$ mv new tnk.pdf 
mv: cannot move 'new' to 'tnk.pdf': Permission denied
$ l -d .
drwxr-x--- 5 u0_a116 everybody 32768 Mar  6 13:33 .
$ id
uid=10116(u0_a116) gid=10116(u0_a116) groups=10116(u0_a116)
$ l new
-rwxr-x--- 1 u0_a116 everybody 0 Mar  6 13:33 new
$ rm new
$ l new
ls: cannot access 'new': No such file or directory

So, as you see, I can create and remove files, but cannot move them to an existing file, for some reason. But the Unix permission on the parent directory is 0750 and the uid is the same as my uid (and that of the sshd process). So, why doesn't it allow me to rename the file?

I hope Android is not using something stupid like ACL or (even worse!) this nonsense called "selinux" and is relying on the old good (almost venerable --- I could even say "sacred" if I wasn't afraid to be labeled an idol-worshipper) Unix permission mechanism as designed in 1970s... Or is something changed between 1970s and now, especially since Android 7?

I'll see if the problem persists via adb port forwarding and ssh to localhost... I hope not.

[its-pointless]

Selinux is used in android. how this information helps i do not know.

[tigran123]

Btw, rsync to a native (not on sd) filesystem (e.g. a subdirectory of $HOME) is working fine on Android 7. So, it is just the vfat (or whatever --- Android's own "flavour" thereof) that is not working.

@its-pointless Yes, yes, I know. My replic was almost rhetorical --- I know this very well as I've been following all the NSA's clandestine work since 1990s and see how the things which they boldly announce on conferences as "purely theoretical suggestions" to "improve security" (but actually to provide mechanisms for spying on people in order to learn how to brainwash them most efficiently) all get invariably implemented by the actual software providers, one by one, thus removing all our freedoms and "convincing" us that it is ok to spy upon every thing that we do.

[tigran123]

Yes, the problem persists (just as one would expect) when connected via USB with adb port forwarding.

So, we need to find an rsync option which deletes the target file before trying to rename it.... I've tried all --delete-blah options and none of them help (they shouldn't as the manual clearly says they are for something completely different). Rsync command relies on the Unix permission mechanism (and rightly so) and the fools at (insert the manufacturer name here, any will do) bow down to NSA's demands and violate this perfect mechanism (and perfection cannot be further perfected, so why modify what worked fine for many generations?).

So, what do we do now? I will need to begin to actually think rather than keep complaining :)

[tigran123]

Ok, found a workaround: instead of doing rsync in a single step, perform two steps:

1. Save the backup copy of every file to be modified with ".bak" suffix.
2. Rsync again which deletes the backup files.

It looks like this:

$ rsync -b --suffix '.bak' --delete --size-only -rvW . localhost:sd/Books/
sending incremental file list
Religions/Bible/Hebrew/The-Little-Book.pdf

sent 851,305 bytes  received 2,220 bytes  243,864.29 bytes/sec
total size is 53,095,519,000  speedup is 62,207.34
$ rsync --delete --size-only -rvW . localhost:sd/Books/
sending incremental file list
deleting Religions/Bible/Hebrew/The-Little-Book.pdf.bak

sent 700,081 bytes  received 2,245 bytes  156,072.44 bytes/sec
total size is 53,095,519,000  speedup is 75,599.53

Does anyone have a better idea?

[fornwall]

$ mv new tnk.pdf mv: cannot move 'new' to 'tnk.pdf': Permission denied

@tigran123 Hm, are you able to rm tnk.pdf? Or do you get permission denied for moving a file over another for every file on the sd, but otherwise you can delete it? I can't reproduce that on an Android 7.1 device here.

[tigran123]

Yes, I can rm tnk.pdf, look:

$ l tnk.pdf
-rwxr-x--- 1 u0_a116 everybody 8080229 Mar  6 14:00 tnk.pdf
$ pwd
/data/data/com.termux/files/home/sd/Books/Religions/Bible/Hebrew
$ rm tnk.pdf
$ l tnk.pdf
ls: cannot access 'tnk.pdf': No such file or directory
$ u
Linux localhost 3.18.35 #14 SMP PREEMPT Mon Jan 23 18:24:13 CST 2017 aarch64 Android
$ id
uid=10116(u0_a116) gid=10116(u0_a116) groups=10116(u0_a116)
$ l -d .
drwxr-x--- 5 u0_a116 everybody 32768 Mar  7 09:29 .

[fornwall]

@tigran123 Are you able to reproduce the issue without using rsync? That is, can you try writing step-by-step instructions for how one could try reproducing the issue from scratch?

[tigran123]

@fornwall Yes, of course, I have done exactly what you suggest in the very first post. I even provided the terminal session transcript of those instructions. Basically, just go to any directory, create a new file and try to "mv newfile existingfile" --- that is all there is to it. Previously this worked absolutely fine and still works on the internal properly Linux-y filesystems like where $HOME is located, but not on the external microsd card.

[stale[bot]]

This issue/PR has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[twaik]

Is this issue still relevant?