port to Android

[tex]

Hi,

have anybody tried to run borg on Android?

Or with Jython?

I can create GUI app for Borg that would trigger borg to start backup, but I have no experience with Python on Android...

[ThomasWaldmann]

I didn't try either yet, but I'ld like to point out that borg is not quite the easiest python-project port to android, because it's not just Python (3.4+), but also extensions in C (built with the help of Cython). Also, it needs quite some resources to run and root permissions.

It has been run on Raspberry Pi 2/3 under Linux though, so at least there is no issue with non-x86 CPUs.

[enkore]

It has been run on Raspberry Pi 2/3 under Linux though, so at least there is no issue with non-x86 CPUs.

Yes, it's fairly portable now, having been tested successfully on Linux (x64+86, powerpc, arm7), OSX, the BSDs, Solaris, ...

[axelsimon]

Any news on this? Borg on Android sounds like it could be a great solution for backing up one's data in a hassle-free way (daily and without thinking about it) and restoring. Knowing you could lose your phone and be able to restore all your stuff on another one in minutes would really be great. Actually, I'm a bit baffled that after about 8 years of Android we still don't have a proper FOSS solution to do this, but I digress.

I understand this is neither the original goal of the project nor and easy job to port Borg to Android, but I think it could be extremely useful.

Thanks!

[RonnyPfannschmidt]

no volunteer stepped up ^^ it might help to just be the one that volunteers it

[stonewareslord]

I have gotten Borg to successfully backup all of my data through Linux Deploy (Gentoo) with no issue. I installed Gentoo, emerged borgbackup, added /storage/emulated/0 as a mountpoint and ran cd /mnt/0&&borg create repo:dir::$(uuidgen) .. I backup with lzma-7 compression with keyfile encryption and both of those work fine.

Not a good permanent solution though because the Linux image takes up 4GB on my phone.

[RonnyPfannschmidt]

@stonewareslord still, pretty neat experiment

i wonder if we can find someone to crosscompile directly

[bauerj]

kivy/python-for-android could be useful.

Without the Cython part it should be easy to create an App with borg code. No idea how hard it is to compile Cython with the NDK.

[enkore]

Cython can be built without a C compiler afaik, but the Cython compiler won't work as fast then.

To build cythonized code however a C compiler is required, so building Borg always requires a C compiler.

[textshell]

I got borg to run with a debian armhf pyinstaller build that also ships debian’s glibc. See #1727 Of course this is not a real port. But at least is only needs 16MB or so on the phone. To properly use this an ssh client would be needed. For now i’m using busybox’s netcat via a script for BORG_RSH.

Also this is of course backup only. Trying to restore a system backup with this is likely going to break horribly. At least without deep enough android knowledge to do a partial restore and fixup the file permissions after doing it.

[ThomasWaldmann]

maybe maintain this in a android branch until it is really usable?

[textshell]

sure, although it mostly only adds 2 "script" files. Primarily is needs testing, before we can even say that it works on enough phones/tablets to be useful.

[enkore]

I'd just throw this in master, it doesn't really touch the code (yet?) so I don't see a need for a separate branch.

[ThomasWaldmann]

Well, it doesn't touch the code now but as far as i understood, it is not really usable yet. So, can we predict it can be made usable without needing a lot of changes we might not want in the end? Can we predict it can be made usable at all (at least for advanced android users)?

That's why I was suggesting a branch.

[stonewareslord]

as i understood, it is not really usable yet

I didn’t run any tests, but when I used it with lzma and keyfile encryption, it worked perfectly fine and it's what I use currently for backup of my android.

[flokli]

Whats the current state of this? I found @textshell 's two scripts. Can I already use them to produce a standalone binary, or what's their intended usage?

[textshell]

The state is a bit mixed. It worked for me when i needed phone backups in a pinch.

But for some reason it seems only to work when used from the adb (root) shell not from a shell started from connectbot.

The scripts should be able to produce a working arm standalone binary distribution that uses a bundled libc. But more test result would be very helpful.

I don’t have the laptop where i did this on around at the moment. I build the binaries using qemu on a debian jessie image. I used full system emulation, but user space emulation only should be easier.

[axelsimon]

I was wondering if Borg could be packaged as a package for termux, an advanced terminal emulator. That might be an option.

[jcgruenhage]

@axelsimon Is it possible to package that as a package for termux, and still wrap and use it in a regular android app?

[rugk]

I think here are the instructions for getting a built there.

Is it possible to package that as a package for termux, and still wrap and use it in a regular android app?

Yes, I think so. An app could just run the borg binary provided it fins its path and it is readable. But packaging it in Termux would be the first big step to do.

[rugk]

Also the app/borg binary should be able to run in root mode, so it can backup really everything, and in non-root mode to only backup documents/pictures or so…

[ThomasWaldmann]

Some feedback about whether #1727 is useful would be nice.

As it does not touch the main code, I am positive about merging this, if it is useful. (that does not necessarily apply for heavy android-related changes to main code, though)

Guess some of the comments above should also go into some sort of README for it.

So, who tried it? Did it work? Do you somehow "use it in production"?

[ravenschade]

I got borg 1.0.11 (39f876229ffcdda12764afbfc0e0187aef69a349) to work on Android 7.1.2 (aarch64, Nexus6p) with termux (https://termux.com/, version 0.53). I have only tested backing up the device via ssh since that is the most likely scenario: borg create --debug -p -v -C lz4 --stats --one-file-system backup:/backup/borg/angler::testfull / /system /vendor /cache /persist /firmware /storage /data In the next days I will clean up my build scripts and patches and make them publicly available.

[ThomasWaldmann]

@ravenschade interesting. did you use the code from PR #1727 or start from scratch?

[ravenschade]

I started from scratch. The necessary modifications are very small. Basically just some adjusted paths and some workarounds for stuff that is missing in android. I am building everything directly on the android device.

[ravenschade]

You can find the build-scripts at https://github.com/ravenschade/borgbackup_on_android. Feedback and tests on other devices are very welcome.

[stonewareslord]

@ravenschade Does your borg output the following error?

WARNING: linker: /data/data/com.termux/files/usr/lib/libacl.so.1.1.0: unused DT entry: type 0xf arg 0x449

It appears for me on my OnePlus3T whenever I run borg (even borg -h).

uname -a:
Linux localhost 3.18.31-lineageos-g252a353 #1 SMP PREEMPT Sat Jun 3 14:55:59 CEST 2017 aarch64 Android

[ravenschade]

Yes, but it is not an error according to https://stackoverflow.com/questions/33206409/unused-dt-entry-type-0x1d-arg.

[textshell]

I've made another try(targeting android 4.1+), this time using the ndk (like proper android software should). It seems to work with a simple local test. I'm not sure how much time i will have to refine this in the near future. But maybe someone is interested so i'll attach my current copy and paste script: android-ndk-initial.txt

The commit hashes are just for pinning to the current version when i tested this to make this more reproducable (it's not fully pinned). If should be possible to use newer versions (or remove the checkout by commit hash).

[bpereto]

I'm trying to make it work as termux package (ndk) as work from Neo-Oli here: https://github.com/termux/termux-packages/issues/557

My try: https://github.com/bpereto/termux-packages/tree/borg

At the moment there is some issue to cross compile cython extensions. Borg compiles with gcc x86_64 instead of aarch64 (see paste https://pastebin.com/KEbgfQcN) My guess is that Cython ignores the CC settings at https://github.com/borgbackup/borg/blob/1.1-maint/setup.py#L111 Any hint is appreciated to get this work :)

[textshell]

did you have a look at my comment? I used crossenv which i think should cover that problem.

[bpereto]

Yup, I think termux wants a clean package. Anyway, it worked to cross-compile with specific flags, but I don't know how I can test if the cython part is working :) Plus there is a need for python-msgpack too.

At the moment borgbackup 1.1.8 with borgmatic works on my phone :tada:

[ThomasWaldmann]

@bpereto since borg 1.1.10 msgpack (0.5.6, which is latest supported version) is part of borg and gets built as part of borg build.

[domhaas]

Currently thinking about how nice it would be to have borg on android. @bpereto any news about it?

[rugk]

An awesome idea would also be to use the new backup app/feature, where you can use the usual Android backup menu. (See this thread on how it works/looks like.)

https://github.com/stevesoltys/backup/issues/45

[amaramrahul]

I have ported @bpereto code to the updated code of termux-packages. Also bumped borgbackup to 1.1.10. However, I have not yet tested it. But the package has been built successfully. Here are the instructions to build it:

git clone git@github.com:amaramrahul/termux-packages.git cd termux-packages/scripts docker build -t termux/package-builder . cd .. ./scripts/run-docker.sh ./build-package.sh borgbackup ls debs/

[amaramrahul]

I have ported @bpereto code to the updated code of termux-packages. Also bumped borgbackup to 1.1.10. However, I have not yet tested it. But the package has been built successfully. Here are the instructions to build it:

git clone git@github.com:amaramrahul/termux-packages.git cd termux-packages/scripts docker build -t termux/package-builder . cd .. ./scripts/run-docker.sh ./build-package.sh borgbackup ls debs/

I have been successfully able to run this on my android phone. You need to copy the borgackup and libacl debs from debs directory to your andoid phone and then install them using dpkg command. There will be a few missing dependencies which you can then install using "apt-get -f install".

While running borg, I got libc.so.6 file missing error thrown by /data/data/com.termux/files/usr/lib/python3.7/site-packages/borgbackup-1.1.10-py3.7-linux-x86_64.egg/borg/xattr.py. I simply symlinked libc++_shared.so in /data/data/com.termux/files/usr/lib to libc.so.6. I don't think this was the right thing to do and might cause issues if xattr attributes are backed up. However, I didn't face any issue probably because I was not backing up files with xattr attributes.

[WladyX]

WOW, managed to get borgmatic up&running on the phone. Thank you guys!!

[amaramrahul]

Raised a PR to package into Termux: https://github.com/termux/termux-packages/pull/4644

[amaramrahul]

The package has been merged to termux master: https://github.com/termux/termux-packages/commit/fc6b158f89a3e8cf0f21cd38598e6e171acda69d#diff-2fc9e51174a78dd5bfc57e8e368590b3

Also, borg can be installed via pip as below.

apt-get install libacl liblz4 zstd openssl python
env BORG_OPENSSL_PREFIX="/data/data/com.termux/files/usr" BORG_LIBLZ4_PREFIX="/data/data/com.termux/files/usr" BORG_LIBZSTD_PREFIX="/data/data/com.termux/files/usr" pip install borgbackup

[maxmopp]

You guys are heros! Thanks a million. Works great

[rugk]

Issue solved? :thinking:

[luke-jr]

Not really. Would be nice to have borgbackup run natively, without all the Termux cruft... ;)

(Ideal would be integrated into LineageOS/other ROMs so it can run as root even on unrooted phones)

[Alexander-Shukaev]

Getting

E: Unable to locate package borgbackup

as per

pkg install borgbackup

[rugk]

BTW just FYI, if you want a quite good integrated backup solution for Android, consider using Seedvault. It needs to be integrated into the OS, but on the other hand this is also an advantage, because it uses the same mechanisms like Googles backup solution + one thing it recently got is a beta feature for backing up arbitrary directories on your phone.

[Alexander-Shukaev]

@rugk, I can't agree to that after a trial today. For example, they still have these weird issues, seedvault-app/seedvault#81 and seedvault-app/seedvault#165, which are a showstopper for me. Having a proper "full" backup of a mobile device, by taking advantage of rooting it, is a must I'd say in order to save yourself the hassle of either precise restoration or quick hardware migration. I ended up going with Swift Backup, which does all of that (it also already includes compression and encryption). Since I don't trust most cloud providers and Google, there is an option to use it anonymously without registering an account and therefore forfeiting the possibility to upload backups directly to supported clouds. As a result, you can still back up locally on internal storage, but afterwards, you're on your own. Hence, I'm looking forward to learn how to set up Borg on Android to do that incremental heavy-lifting and upload archives of Swift Backup to my own cloud.

[rugk]

Well… fine, if that works for you no problem and it certainly has advantages, but if you do not have root and e.g. you can dismiss the limitations Google's backup solution imposes (like that apps need to opt-in into backups and allowBackup cannot be [yet] be forced for these) then it's a solid and usable solution that only requires a GUI and no CLI stuff etc. And it has Nextcloud integration, so automatic backups to the cloud are possible, which is just awesome.

Anyway, that's kinda off-topic anyway. Just wanted to provide a hint/help for users, who may still following this issue here. :upside_down_face:

[wshanks]

@Alexander-Shukaev The termux package still exists: https://github.com/termux/termux-packages/tree/master/packages/borgbackup. You might get better help with your error in a termux support forum.

[luke-jr]

My Termux didn't see the borgbackup package either... until I stumbled upon a notice that Termux from Google Play is no longer supported! Removing it and reinstalling from F-Droid got me access to borgbackup, which seems to be working nicely now.

[Alexander-Shukaev]

My Termux didn't see the borgbackup package either... until I stumbled upon a notice that Termux from Google Play is no longer supported! Removing it and reinstalling from F-Droid got me access to borgbackup, which seems to be working nicely now.

@luke-jr, boss answer, thanks a lot!