oandbackup can't back up symbolic links, making it impossible to back up and restore Termux

[YannCodes]

I tried some days ago to backup the Termux app, which is a terminal emulator. It took an unusually long time (about 40 minutes) and the backup was 5GB large, while Android showed that the app was 600MB large. I decompressed it and found out that oandbackup followed symbolic links to the emulated storage. Instead of just backing the link up, it backed up everything that was linked. This is what I get inside Termux :

-bash-4.4$ ls -la ~/storage total 8 drwx------ 2 u0_a174 u0_a174 4096 May 25 18:19 . drwx------ 7 u0_a174 u0_a174 4096 May 25 18:37 .. lrwxrwxrwx 1 u0_a174 u0_a174 24 May 25 18:19 dcim -> /storage/emulated/0/DCIM lrwxrwxrwx 1 u0_a174 u0_a174 28 May 25 18:19 downloads -> /storage/emulated/0/Download lrwxrwxrwx 1 u0_a174 u0_a174 26 May 25 18:19 movies -> /storage/emulated/0/Movies lrwxrwxrwx 1 u0_a174 u0_a174 25 May 25 18:19 music -> /storage/emulated/0/Music lrwxrwxrwx 1 u0_a174 u0_a174 28 May 25 18:19 pictures -> /storage/emulated/0/Pictures lrwxrwxrwx 1 u0_a174 u0_a174 19 May 25 18:19 shared -> /storage/emulated/0`

oandbackup version : 0.2.12.1, installed from F-Droid

I think oandbackup should backup the links and not the linked files.

[bdantas]

Thanks for figuring this out. I was puzzled why backing up Termux takes so long and causes my device's storage to max out.

As a temporary workaround, before backing up Termux do this in Termux:

cd /data/data/com.termux/files/home
tar -cvf storage.tar storage/
rm -r storage

Then open oandbackup and backup Termux. After you restore the backup, remember to reverse the above. In Termux:

cd /data/data/com.termux/files/home
tar -xvf storage.tar
rm storage.tar

Now the links are safely out of the way during backup/restore operations.

[YannCodes]

Thank you for your workaround.

I took the time to read the relevant part of the source code. Here is what I found in ShellCommands.java :

        // -L because fat (which will often be used to store the backup files)
        // doesn't support symlinks
        String followSymlinks = prefs.getBoolean("followSymlinks", true) ? "L" : "";
        switch(backupMode)
        {
            case AppInfo.MODE_APK:
                commands.add("cp " + packageApk + " " + backupSubDirPath);
                break;
            case AppInfo.MODE_DATA:
                commands.add("cp -R" + followSymlinks + " " + packageData + " " + backupSubDirPath);
                break;
            default: // defaults to MODE_BOTH
                commands.add("cp -R" + followSymlinks + " " + packageData + " " + backupSubDirPath);
                commands.add("cp " + packageApk + " " + backupSubDirPath);
                break;
        }

Also, I found this in the busybox manual about cp : -H,-L Dereference all symlinks (default)

And I felt dumb, since I found that in the preferences there is a checkbox : "follow symbolic links", which is checked by default. But when I backed up and restored Termux with the option unchecked, I got this in the app :

exec("/data/data/com.termux/files/usr/bin/login"): No such file or directory

[Process completed (code 1) - press Enter]

I compared directories before and after backup and this is what I found :

bullhead:/data/data # diff -r com.termux com.termux.old/                                                                                                                                                           
Only in com.termux.old/: files/home/storage
Only in com.termux.old/: files/usr/bin/applets
Only in com.termux.old/: files/usr/bin/env
Only in com.termux.old/: files/usr/bin/packages
Only in com.termux.old/: files/usr/bin/sh
Only in com.termux.old/: files/usr/bin/xdg-open
Only in com.termux.old/: files/usr/etc/apt/apt.conf.d
Only in com.termux.old/: files/usr/etc/apt/preferences.d
Only in com.termux.old/: files/usr/lib/apt/methods/ssh
Only in com.termux.old/: files/usr/lib/apt/methods/xz
Only in com.termux.old/: files/usr/lib/libapt-pkg.so
Only in com.termux.old/: files/usr/lib/libapt-pkg.so.5.0
Only in com.termux.old/: files/usr/lib/libhistory.so
Only in com.termux.old/: files/usr/lib/libhistory.so.7
Only in com.termux.old/: files/usr/lib/libiconv.so
Only in com.termux.old/: files/usr/lib/libintl.so
Only in com.termux.old/: files/usr/lib/libncurses.so
Only in com.termux.old/: files/usr/lib/libncurses.so.6
Only in com.termux.old/: files/usr/lib/libncurses.so.6.0
Only in com.termux.old/: files/usr/lib/libncursesw.so
Only in com.termux.old/: files/usr/lib/libncursesw.so.6
Only in com.termux.old/: files/usr/lib/libreadline.so
Only in com.termux.old/: files/usr/lib/libreadline.so.7
Only in com.termux.old/: files/usr/lib/terminfo
Only in com.termux.old/: files/usr/share/man/man1/sh.1
Only in com.termux.old/: files/usr/tmp
Only in com.termux.old/: files/usr/var/cache
Only in com.termux.old/: files/usr/var/lib/apt
Only in com.termux.old/: files/usr/var/lib/dpkg/triggers
Only in com.termux.old/: files/usr/var/lib/dpkg/updates
Only in com.termux.old/: files/usr/var/log
--- com.termux/shared_prefs/com.termux_preferences.xml
+++ com.termux.old/shared_prefs/com.termux_preferences.xml
@@ -1,4 +1,4 @@
 <?xml version='1.0' encoding='utf-8' standalone='yes' ?>
 <map>
-    <string name="current_session">267e2de5-e115-47e1-be45-3e3042ca1928</string>
+    <string name="current_session">2638ed1a-c7e6-4734-bd8b-14788a4894b7</string>
 </map>

Turns out that all the missing files are symlinks. So currently oandbackup can either follows symlinks while backing up, or omit them completely. It'd be really nice to have an option to back up the links.

[ildar]

No option. Symlinks should be symlinks. But that's not so trivial to fix. @jensstein , I'd suggest skipping symlink copying to sdcard and adding symlinks to ZIP afterwards.

[Lanchon]

or bundle a more capable native zip tool with the apk. golang (yuck!) can naively cross build to many archs and there is a full go userland in development which might have what's needed.

[ildar]

That's not the issue of zip tool. The app just first copies files to sdcard where symlinks aren't functional. Yes, I'd prefer another approach, e.g. hard linking inside /data . That, BTW is also faster

[Lanchon]

why doesn't it zip in one go? probably limitations of the zip command line? this should be tar|lzop imho, with tar keeping permissions and only a chown afterwards. btw, what would happen with links and chmod/chown if links were actually backed up in the future?

[ildar]

Think what can happen if the is launched in the middle. E.g. be the external trigger. BTW idea with hard links may be not so good.

[Lanchon]

[i think you meant "the app is launched"]

and what if it is launched in the middle of copying? android has a setting for this: the force-stopped setting. it should be set before backing up, then restored to the previous value. normal stuff like intents will not reach and wake a force-stopped app. but i think some less common stuff will. the solution is to retry (or abort/retry) the backup, maybe a limited number of times, if the app does not continue to be force-stopped at the end of its backup process. (the case of starting then force-stopping an app during its backup would be quite obscure.)

[ildar]

Oh! Good point. I wonder what @jensstein thinks about it.

[lameventanas]

I just hit this bug. One of my apps has a symlink to /storage/emulated/0, so the backup took up all the space and never finished. How about using tar instead of zip? tar can save symbolic links.