Package request: git-annex

[matthiasbeyer]

Would be awesome to have git-annex in termux, so I can sync music and movies from my other devices with it.

[joeyh]

I am the developer of git-annex, and I would also like to see it in termux. Am willing to do work to make it happen.

I do have git-annex building for android. As a haskell program, it needs https://github.com/neurocyte/ghc-android to build.

[joeyh]

@fornwall I see you have a disabled attempt to port ghc at https://github.com/termux/termux-packages/tree/master/disabled-packages/ghc -- I think ghc-android should contain all the patches you need, and I am somewhat familiar with it, if I can help.

That is, ghc-android gets you a ghc that can cross-compile for android. It doesn't get a ghc that runs natively on android, and I think that would be a lot of additional work if it's your goal. That's not needed for porting git-annex though.

[pjones]

I'd love to see this happen! If I can help please let me know.

[jamesd]

So the author of git-annex offers to help make a termux package happen over a year ago and still nothing?

[joeyh]

Well, we're all busy and this is probably somewhat low-priority for all of us and involves a complicated compiler port or cross build.

There has, however, been significant work on ghc for android in the past year, by zw3rk.

• http://hackage.mobilehaskell.org/ has prebuilt cross-compiler for Android, and a growing set of patched libraries.
• https://medium.com/@zw3rk has a lot more

All cross-compiling so far, running ghc on android is not on the map that I know of.

[jamesd]

@joeyh, I'm glad you're still involved. I was worried.

I don't follow Haskell development so I did not know the cross compiler is still in development. With the latest work by zw3rk hopefully we are not too far off from a git-annex package for termux.

[matthiasbeyer]

I'm really looking forward to git-annex on termux for my mobile device, though I have to note that git-annex might be really slow on the devices.

For example, I had the git-annex android app installed some time ago (I own a Oneplus One, which works perfectly well for todays apps still). Cloning my music collection (4347 directories, 52794 files) onto the phone and git annex syncing (not putting the data on my phone, simply syncing the repository) took hours! In the end I aborted it because it took too long.

Still, for smaller repositories/collections, it might be worth it!

[janbuchar]

Do I understand correctly that adding git-annex to Termux would allow starting git-annex when the device boots (using Termux::Boot) without having to open a terminal every time (which is the case with the current git-annex Android app)?

[its-pointless]

Yes

[joeyh]

@Jan yes, that would work and would be a nice benefit!

-- see shy jo

[jan]

@jan summons me, @joeyh.

[joeyh]

Turns out that the arm tarball from http://git-annex.branchable.com/install/Linux_standalone/ just works in termux. Just untar inside your termux user's home directory, and cd git-annex.linux && ./runshell will add a working git-annex to your path.

[sorsasampo]

@joeyh

~ $ git-annex.linux/runshell
ERROR: ld.so: object '/data/data/com.termux/files/usr/lib/libtermux-exec.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
Bad system call

[tomty89]

unset LD_PRELOAD

[sorsasampo]

So the git-annex tarball is not compatible with termux-exec?

[joeyh]

@sorsasampo I've made the git-annex wrapper script unset LD_PRELOAD for the next release, so hopefully that will avoid this problem. Although it would be good to get confirmation that unsetting LD_PRELOAD is sufficient to make it work for you; the "Bad system call" could be some other problem.

-- see shy jo

[sorsasampo]

@joeyh, still having some issue:

~ $ unset LD_PRELOAD
~ $ git-annex.linux/runshell
\w $ git-annex
sh: 1: git-annex: not found
\w $ git annex
fatal: 'annex' appears to be a git command, but we were not
able to execute it. Maybe you need to 'pkg install perl'?
\w $ sh -x $(which git-annex)
+ GIT_ANNEX_PROGRAMPATH=/data/data/com.termux/files/home/git-annex.linux/bin/git-annex
+ export GIT_ANNEX_PROGRAMPATH
+ exec /data/data/com.termux/files/home/git-annex.linux/exe/git-annex --library-path /data/data/com.termux/files/home/git-annex.linux//usr/lib/arm-linux-gnueabi/gconv:/data/data/com.termux/files/home/git-annex.linux//usr/lib/arm-linux-gnueabi/audit:/data/data/com.termux/files/home/git-annex.linux//etc/ld.so.conf.d:/data/data/com.termux/files/home/git-annex.linux//lib:/data/data/com.termux/files/home/git-annex.linux//lib/arm-linux-gnueabi:/data/data/com.termux/files/home/git-annex.linux//usr/lib/arm-linux-gnueabi: /data/data/com.termux/files/home/git-annex.linux/shimmed/git-annex/git-annex
Bad system call

(and yes, perl is installed)

[Grimler91]

@sorsasampo what device do you have? What's the output of termux-info?

[sorsasampo]

~ $ termux-info
Updatable packages:
All packages up to date
System information:
Linux localhost 4.4.13-13201289-QB17339056 #1 SMP PREEMPT Thu Mar 15 18:46:17 KST 2018 aarch64 Android
Termux-packages arch:
aarch64
Android version:
8.0.0
Device manufacturer:
samsung
Device model:
SM-G950F

[Grimler91]

Seems like you are trying to run a arm binary on a aarch64 system. Try a aarch64/armv8l binary instead if that is available.

[ghost]

Seems like you are trying to run a arm binary on a aarch64 system.

@Grimler91 It is possible to run arm binary on aarch64 system.

Bad system call

@sorsasampo Your binary is blocked by seccomp (feature introduced in Android 8). Your binary is compiled against glibc which don't know about blacklisted system calls on Android. Tool strace can help to find out which call actually causes this problem.

I have downloaded the arm tarball of git-annex and executed your commands. Output (only part of text as it was too long):

[termux]:~/test:$ git-annex.linux/runshell
ERROR: ld.so: object '/data/data/com.termux/files/usr/lib/libtermux-exec.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
$ unset LD_PRELOAD
$ ls
git-annex.linux
$ git annex
fatal: 'annex' appears to be a git command, but we were not
able to execute it. Maybe you need to 'apt install perl'?
$ sh -x $(which git-annex)
+ GIT_ANNEX_PROGRAMPATH=/data/data/com.termux/files/home/test/git-annex.linux/bin/git-annex
+ export GIT_ANNEX_PROGRAMPATH
+ exec /data/data/com.termux/files/home/test/git-annex.linux/exe/git-annex --library-path /data/data/com.termux/files/home/test/git-annex.linux//usr/lib/arm-linux-gnueabi/gconv:/data/data/com.termux/files/home/test/git-annex.linux//usr/lib/arm-linux-gnueabi/audit:/data/data/com.termux/files/home/test/git-annex.linux//etc/ld.so.conf.d:/data/data/com.termux/files/home/test/git-annex.linux//lib:/data/data/com.termux/files/home/test/git-annex.linux//lib/arm-linux-gnueabi:/data/data/com.termux/files/home/test/git-annex.linux//usr/lib/arm-linux-gnueabi: /data/data/com.termux/files/home/test/git-annex.linux/shimmed/git-annex/git-annex
Missing: COMMAND

Usage: git-annex COMMAND
  git-annex - manage files with git, without checking their contents in

  Commonly used commands:

  add             PATH ...         add files to annex
  addurl          URL ...          add urls to annex
  assistant                        automatically sync changes
  copy            PATH ...         copy content of files to/from another repository
  drop            PATH ...         remove content of files from repository
  edit            PATH ...         same as unlock
  export          TREEISH          export content to a remote
  get             PATH ...         make content of annexed files available
  help            COMMAND          display help
  import          PATH ...         move and add files from outside git working copy
  importfeed      URL ...          import files from podcast feeds

I don't have bad system call as I'm on Android 7.0 aarch64.

[Grimler91]

@xeffyr I stand corrected. Is this true on all aarch64 devices? It obviously won't work for dynamically linked binaries but I was also under the impression that the kernel must have been compiled with certain options, see this so question.

Maybe all android devices are though.

[ghost]

@Grimler91 There is a way to check this:

[termux]:~:$ cat /system/build.prop | grep abi
# ro.product.cpu.abi and ro.product.cpu.abi2 are obsolete,
# use ro.product.cpu.abilist instead.
ro.product.cpu.abi=arm64-v8a
ro.product.cpu.abilist=arm64-v8a,armeabi-v7a,armeabi
ro.product.cpu.abilist32=armeabi-v7a,armeabi
ro.product.cpu.abilist64=arm64-v8a

If list contains armeabi or armeabi-v7a, then 32-bit programs are supported in your OS.

[joeyh]

I don't have a arm64 build of git-annex in this form; it would not be super hard to make one.

It's sometimes possible to run arm binaries on arm64, but it depends on kernel support. The "bad system call" looks like the kernel doesn't support arm system calls.

-- see shy jo

[joeyh]

Seems likely that seccomp might have a whitelist of arm64 system calls that does not include arm system calls?

-- see shy jo

[joeyh]

@sorsasampo I've prepared a test build for arm64: https://downloads.kitenet.net/git-annex/linux/test/git-annex-standalone-arm64.tar.gz

[wshanks]

A path to a working git-annex in Termux is great news. Is the git annex arm build system compatible with the Termux build system? In other words, does this mean git-annex could be packaged in the Termux repository? Or does that still require that the Haskell dependencies be packaged for Termux?

[ghost]

@willsALMANJ These builds are compiled against glibc:

This means that:

1. git-annex arm build system is not compatible with the Termux build system.
2. git-annex will be available when it will be possible to compile it against Android's bionic c library.
3. Haskell dependencies are still needed to build it.

[sorsasampo]

@joeyh

@sorsasampo I've prepared a test build for arm64: https://downloads.kitenet.net/git-annex/linux/test/git-annex-standalone-arm64.tar.gz

Here is output of strace with this arm64 tarball:

~ $ git-annex.linux/runshell
\w $ git-annex
/data/data/com.termux/files/home/git-annex.linux/shimmed/sh/sh: 1: git-annex: not found
\w $ git annex
fatal: 'annex' appears to be a git command, but we were not
able to execute it. Maybe you need to 'pkg install perl'?
\w $ strace -f bash -x $(which git-annex)
<clip>
mprotect(0x5991d22000, 4096, PROT_READ) = 0
set_tid_address(0x788e5c9a70) = 16947
set_robust_list(0x788e5c9a80, 24 <unfinished ...>
--- SIGSYS {si_signo=SIGSYS, si_code=SYS_SECCOMP, si_call_addr=0x7890921958, si_syscall=__NR_set_robust_list, si_arch=AUDIT_ARCH_AARCH64} ---
<... set_robust_list resumed>) = ?
+++ killed by SIGSYS +++
Bad system call

So there's verification for what @xeffyr wrote earlier:

@sorsasampo Your binary is blocked by seccomp (feature introduced in Android 8). Your binary is compiled against glibc which don't know about blacklisted system calls on Android. Tool strace can help to find out which call actually causes this problem.

[ghost]

--- SIGSYS {si_signo=SIGSYS, si_code=SYS_SECCOMP, si_call_addr=0x7890921958, si_syscall=__NR_set_robust_list, si_arch=AUDIT_ARCH_AARCH64} ---

Looks like syscall set_robust_list is not provided by bionic libc. At least it is not exist in: https://github.com/aosp-mirror/platform_bionic/blob/master/libc/SYSCALLS.TXT

Seccomp allows:

1. all the syscalls exposed via bionic
2. syscalls to allow Android to boot
3. syscalls used by popular Android applications, as determined by running Google's full app compatibility suite

More info about seccomp in Android available here: https://android-developers.googleblog.com/2017/07/seccomp-filter-in-android-o.html

[joeyh]

set_robust_list is called by glibc in eg, pthread_create. No easy way to disable it without recompiling; OTOH if the syscall were intercepted with ptrace and NOOPed, it would probably not harm the program (unless it caused it to deadlock when a thread died leaving a futex locked).

proot seems a likely way to get around this, and indeed I see they have dealt with just this in https://github.com/termux/proot/commit/5ed0da18d9055b0428e5823d8a19ecae754fb66a and there are a few other workarounds for some other innocuous syscalls that this IMHO illfounded security policy of Android's blocks.

@sorasampo, you could try pkg install proot and then try proot git-annex.linux/git-annex

[joeyh]

@sorsasampo see my comment above for something to try

[sorsasampo]

@joeyh nice, it indeed seems to work:

~ $ git-annex.linux/git-annex version
Bad system call
~ $ proot git-annex.linux/git-annex version |head -1
git-annex version: 6.20180426-g9926727ee

I'll try to give it a proper spin later.

Can the same workaround be used in git-annex, or do you plan on making the wrapper script wrap git-annex execution in a proot call?

[joeyh]

@sorsasampo I'll have the wrapper script use proot, assuming this works well enough.

[sorsasampo]

So far I have not run into any problems, apart from trying to figure out how to deal with a crippled filesystem (as git-annex kindly denounces it), and v6 origin.

@joeyh, is it possible for the git-annex wrapper to catch the signal and helpfully suggest installing proot? I'm assuming you are not going to require it for everyone, since some devices might not need it.

[quincunx]

@joeyh I've tried to run git-annex according to install_on_Android_in_Termux.mdwn, but this error message is shown:

proot error: execve("/data/data/com.termux/files/home/git-annex.linux/bin/sh"): No such file or directory
proot info: possible causes:
* the program is a script but its interpreter (eg. /bin/sh) was not found;
* the program is an ELF but its interpreter (eg. ld-linux.so) was not found;
* the program is a foreign binary but qemu was not specified;
* qemu does not work correctly (if specified);
* the loader was not found or doesn't work.
fatal error: see `proot --help`. 

The mentionend file /data/data/com.termux/files/home/git-annex.linux/bin/sh does exist, though.

I'm using a Nokia 5 with Android 8.1.0.

[Chobbes]

I'm having the same problem as @quincunx. I am also using an arm64 phone. Could this be the issue?

Can we get arm64 into the build bot?

[chrysn]

I've done some debugging on my LineageOS 14.1 (Android 7.1.2) device and documented it in the git-annex wiki.

[aspiers]

@quincunx @Chobbes I believe you need to run

termux-fix-shebang git-annex.linux/bin/*

as mentioned on install on Android in Termux.

However, if you have an aarch64 phone like me (or arm64, I'm not sure what the exact difference is) then after running that you'll get:

% ./git-annex.linux/runshell
proot info: vpid 1: terminated with signal 11

Some further digging suggests that this is due to the architecture mismatch between my OnePlus 5T which is aarch64, and the linker which is 32-bit:

% PROOT_VERBOSE=9 ./git-annex.linux/runshell |& tail -n3
proot info: vpid 1: translate("/" + "/data/data/com.termux/files/home/git-annex.linux/lib/ld-linux.so.3")
proot info: vpid 1:          -> "/data/data/com.termux/files/home/git-annex.linux/lib/ld-linux.so.3"
proot info: vpid 1: terminated with signal 11
% file /data/data/com.termux/files/home/git-annex.linux/lib/ld-linux.so.3
/data/data/com.termux/files/home/git-annex.linux/lib/ld-linux.so.3: ELF 32-bit LSB pie executable ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=59819e8e8c922b822204d3778cd4d6c843422462, stripped
% termux-info
Updatable packages:
All packages up to date
System information:
Linux localhost 4.4.78-perf+ #1 SMP PREEMPT Wed Jun 13 17:23:58 CST 2018 aarch64 Android
Termux-packages arch:
aarch64
Android version:
8.1.0
Device manufacturer:
OnePlus
Device model:
ONEPLUS A5010

@joeyh What's the next step here?

[aspiers]

Just checked - still broken with latest version.

[joeyh]

arm64 (aka aarch64) is now supported, the git-annex wiki has a termux installer script. It should also work on intel android devices, although I have not tested that.

I wonder if the git-annex installer script could be packaged in termux, so pkg install git-annex would run it as a postinst and download the git-annex binary? I don't know if termux's policy allows such an installer. But it would be a simple way to close this issue. ;-)

FWIW, the linux binaries are very easy to build from source on any debian system. Indeed, I built the new arm64 one in a debian chroot on my phone.

[daraul]

I'm also having this problem. Here's my info:

Termux Variables:
TERMUX_API_VERSION=0.50.1
TERMUX_APK_RELEASE=F_DROID
TERMUX_APP_PACKAGE_MANAGER=apt
TERMUX_APP_PID=32407
TERMUX_IS_DEBUGGABLE_BUILD=0
TERMUX_MAIN_PACKAGE_FORMAT=debian
TERMUX_VERSION=0.118.0
Packages CPU architecture:
aarch64
Subscribed repositories:
# sources.list
deb https://mirrors.njupt.edu.cn/termux/apt/termux-main stable main
Updatable packages:
vim-runtime/stable 9.0.1250 all [upgradable from: 9.0.1200]
vim/stable 9.0.1250 aarch64 [upgradable from: 9.0.1200]
termux-tools version:
1.36.1
Android version:
12
Kernel build information:
Linux localhost 4.19.157-perf-g2cf66443a7ca #1 SMP PREEMPT Sun Nov 13 01:27:09 CST 2022 aarch64 Android
Device manufacturer:
Xiaomi
Device model:
220333QAG