Binfmt on Termux

[toastmod]

As the title implies this is two suggestions in one.

I think it would be really neat if you could find a way to integrate box86/64 directly with linux's set of exec functions, modifying it so that it reads the header of an executable for it's architecture and executes with box86/64 instead of natively. If you can compile box as a library then you could statically link it directly into ld-linux.so

I think this would be especially nice in conjunction with dpkg --add-architecture, such that upon installing a package with a foreign architecture, any execution will be handled correctly no matter the arch.

Of course this might require handling shell scripts more properly which is something I'd like to see added into box86/64. Being able to run scripts would help with environment variable setups and such.

Maybe instead of integrating with ld-linux.so, box could have it's own version of bash that can run scripts, and read headers for architectures instead of at ld-linux.so.

Anyway, just a suggestion I think worth discussing. It might make box86/64 alot more convenient to use.

[ptitSeb]

What you are discribing sound like binfmt. And box already comes with binfmt integration when you install it.

[toastmod]

Oh, so does that mean I should be able to run any binary directly from the shell? Or do I have to enable something for that?

[ptitSeb]

Yes. Once installed, and after a restart of the binfmt service. At least on Debian/Ubuntu/Armbian. Not sure for other distro, but the idea is the same: binfmt already exist and does the magic.

[toastmod]

Hey I just realized I'm not able to use binfmt on termux, is there any workaround for this?

[WheezyE]

The short answer is that Termux proot doesn’t allow for binfmt, though Termux root might. https://github.com/termux/proot/issues/192

One workaround might be to make a /usr/local/bin/wine script that takes input and adds box64 wine to the beginning of anything passed to it or something like that. We were trying to do that in AnBox86 though lowspecman and I ran into some kind of trouble with 32-bit proot environments. I’m not a pro by any means though and if you find a workaround id really like to know

[toastmod]

@WheezyE After looking into things, looks like binfmt on termux will be a bust because it requires systemd, something not available in proot, let alone chroots even. I think the workaround here might be an LD_PRELOAD with code like this for linux's exec system calls. I'm not a pro either haha, but I'll gonna try making this one work.

[WheezyE]

Lmk if you get a command or script together with LD_PRELOAD. I’ll look into that too. This also gives me an idea to try making a launcher script that runs a file through file first and greps output before passing the files name to box86 or box64 depending on file type

[toastmod]

@WheezyE Yeah a launcher script would be super nice, let's post what we come up with in this thread.

[Azkali]

~~Found an interesting read surrounding this issue, maybe this could be implemented in box86 and box64, in the case the host system is not muslc or glibc ? https://www.humprog.org/~stephen//blog/devel/syscall-tracing-in-process.html~~ EDIT: Found an android compatible replacement for LD_PRELOAD at runtime https://github.com/bylaws/liblinkernsbypass

[toastmod]

So far I've been working on an LD_PRELOAD layer, but I have bumped into a few hurdles. One being that box86/64 will also read LD_PRELOAD and attempt to load the layer, potentially causing a loop. But maybe that part of box can be modified to ignore a certain library specified in LD_PRELOAD.

Anyhow, so far when I try to run wine64 with the preload layer I get this:

[BOXEC] Non-ARM ELF detected!
[BOXEC] >>> box64./wine64(null) ...
bash: ./wine64: Bad address

I have to do some digging on Bad address errors to understand what's going on.

@Azkali what you suggested there looks like it could be used to preload from outside of termux if I'm understanding correctly. If that's so, then that could avoid box64 reading the LD_PRELOAD env variable. Since it would be preloaded from outside the proot.

That said, maybe fitting box86/64 into proot itself is worth looking into for the same reason (preloading from outside the proot). I believe you can already use qemu-user or any interpreter of your choice with proot, but I'd rather stay away from hosting an entire x86 rootfs with box86 since there would be more overhead afaik.

So basically, this preload layer thing might not work out, but there are still other workarounds I want to try. Also some scripting stuff like @WheezyE said is still on my mind. Alas, until tomorrow.

[WheezyE]

I haven’t had any time this week to try cobbling anything together yet, but my plan is to use bash-preexec to run a script that takes the commandline input of every terminal command, checks it against file, then if the command points to a 32-bit i386 file, the script will take the terminal input string and append “box86” to it and run it. Something like that maybe idk.

I’m not sure about the “bad address” in wine. Are you running a native apk or running through termux? If apx, could it be related to wine needing a 3G/1G kernel?

[toastmod]

@WheezyE Wow I didn't know something like that existed, using preexec is exactly what you'd need for sure. I'll try making a script when I have time too.

I think the bad address error is coming from some broken pointers in my C code, but bash thinks it's happening in wine64 because that's the program it thinks it's executing. I have been able to run wine in box86 (I'm on termux btw) so I doubt there's any issue. I think the LD_PRELOAD idea might be too complex to finish.

[WheezyE]

What-with holidays, it took me a while to find time to give this a shot, sorry for the slow turn-around. I didn't end up finding a solution, but I'll post my attempts here:

1. Installed box86/box64 with my experimental AnBox86_64_notworking.sh script for Termux:

   curl https://raw.githubusercontent.com/WheezyE/AnBox86/0547b645df25706b26c5d3ab1b3654bf4adbb00d/AnBox86_64_notworking.sh | bash

   and then removed my older attempts at launcher scripts (which are auto-installed with this script)

   rm /usr/local/bin/wineserver
   rm /usr/local/bin/wine
   rm /usr/local/bin/wine64

2. Installed bash-preexec inside the proot-distro ubuntu user account:

   pkg install curl -y
   curl https://raw.githubusercontent.com/rcaloras/bash-preexec/master/bash-preexec.sh -o ~/.bash-preexec.sh
   # Source our file at the END of our bash profile (e.g. ~/.bashrc, ~/.profile, or ~/.bash_profile)
   echo '[[ -f ~/.bash-preexec.sh ]] && source ~/.bash-preexec.sh' >> ~/.bashrc

3. Created this 'launcher' script (to try to act like binfmt)

   
   GNU nano 5.6.1                                                                   binfmtfaux.sh
   #!/bin/bash

Filetype finder

found32arm="$(file "$@" | grep -o 'ELF 32-bit LSB executable, ARM')"

found64arm="$(file "$@" | grep -o 'ELF 64-bit LSB executable, ARM aarch64')"

foundDOS="$(file "$@" | grep -o 'PE32 executable (GUI) Intel 80386')" foundx86="$(file "$@" | grep -o 'ELF 32-bit LSB executable, Intel 80386')" foundx64="$(file "$@" | grep -o 'ELF 64-bit LSB executable, x86-64')"

filepath="$(realpath "$@")"

if [[ $foundx86 ]] || [[ $foundDOS ]]; then

$foundx86 has a value

  /usr/local/bin/box86 /home/user/wine/bin/wine $filepath
  echo "hi"

elif [[ $foundx64 ]]; then

$foundx64 has a value

  /usr/local/bin/box64 /home/user/wine/bin/wine64 $filepath

else : #do nothing fi

4. Set up bash-preexec: `preexec() { ~/./binfmtfaux.sh "$@"; }`
5. Downloaded/installed an old DOS game (that works on [AnBox86.sh](https://github.com/WheezyE/AnBox86/blob/44644ec1a500a829fab4bd0d792455d2af3de869/AnBox86.sh) - controls, audio, and all):

sudo apt install p7zip-full -y wget https://archive.org/download/es2demo/es2demo.exe 7z x es2demo.exe 7z x DATA.EXE ES.EXE

Results:

user@localhost:~$ ES.EXE ERROR: ld.so: object '/usr/lib/aarch64-linux-gnu/libgcc_s.so.1' from /etc/ld.so.preload cannot be preloaded (wrong ELF class: ELFCLASS64): ignored. Box86 with Dynarec v0.2.5 2fc8d9ae built on Jan 7 2022 16:02:48 ERROR: ld.so: object '/usr/lib/aarch64-linux-gnu/libgcc_s.so.1' from /etc/ld.so.preload cannot be preloaded (wrong ELF class: ELFCLASS64): ignored. Box86 with Dynarec v0.2.5 2fc8d9ae built on Jan 7 2022 16:02:48 Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL PageSize:4096 Box64 with Dynarec v0.1.7 667dc36 built on Jan 7 2022 15:42:29 Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/ Using default BOX64_PATH: ./:bin/ Counted 32 Env var Looking for /home/user/wine/bin/wineserver Using native(wrapped) libdl.so.2 Using native(wrapped) libc.so.6 Using native(wrapped) ld-linux-x86-64.so.2 Using native(wrapped) libpthread.so.0 Using native(wrapped) librt.so.1 wine: chdir to /tmp/.wine-1000/server-0-1ef : No such file or directory hi -bash: ES.EXE: command not found

Hardware: Android Amazon Fire HD10 tablet (no root) running Termux (downloaded from F-Droid)

--------

I previously had launcher scripts for wine/wine64 (to put "box86" or "box64" in place behind each respective version of wine before they launched) but they didn't work either:

Make scripts and symlinks to transparently run wine with box86 (since we don't have binfmt_misc available)

        echo -e '#!/bin/bash'"\nDISPLAY=:1 box64 $HOME/wine/bin/wine64" '"$@"' | sudo tee -a /usr/local/bin/wine64 >/dev/null
        echo -e '#!/bin/bash'"\nDISPLAY=:1 box86 $HOME/wine/bin/wine" '"$@"' | sudo tee -a /usr/local/bin/wine >/dev/null
        echo -e '#!/bin/bash'"\nbox64 $HOME/wine/bin/wineserver" '"$@"' | sudo tee -a /usr/local/bin/wineserver >/dev/null
        sudo ln -s $HOME/wine/bin/wineboot /usr/local/bin/wineboot
        sudo ln -s $HOME/wine/bin/winecfg /usr/local/bin/winecfg
        sudo chmod +x /usr/local/bin/wine64 /usr/local/bin/wine /usr/local/bin/wineboot /usr/local/bin/winecfg /usr/local/bin/wineserver

[toastmod]

@WheezyE Great work anyhow. Looks like it could be a recent bug in wine where the workaround is to do ln -s /tmp/.wine-1000/ /run/user/1000/wine according to the links I found. Haven't tried it though.

[WheezyE]

Ah! Thank you for the hint. I've gotten a little farther now with being able to run wine64. The wine 32-bit binary still doesn't work for me for some reason though. Here's another summary of my tinkering:

Old scripts method Using my old scripts (see the very end of my last post for those), and with linking, adding permissions, and adding a directory that wine was looking for, I was able to run wine64, but not wine yet (wineserver is 64bit and I believe isn't playing nicely with 32-bit wine binary for some reason even after running wineserver & wine wineboot using my old scripts):

ln -s /tmp/.wine-1000/ /home/user/wine
mkdir /tmp/.wine-1000/server-0-15c # wine kept asking for this directory so I just created it manually
sudo chmod 700 -R /tmp/.wine-1000/

EDIT: I realized I was running proot --isolated and without --shared-tmp

wine64 winecfg works! But wine winecfg (or wineboot) don't work and give this error due to no binfmt:

user@localhost:~$ wine ES.EXE
ERROR: ld.so: object '/usr/lib/aarch64-linux-gnu/libgcc_s.so.1' from /etc/ld.so.preload cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
Box86 with Dynarec v0.2.5 88642d81 built on Jan  9 2022 13:35:05
ERROR: ld.so: object '/usr/lib/aarch64-linux-gnu/libgcc_s.so.1' from /etc/ld.so.preload cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
Box86 with Dynarec v0.2.5 88642d81 built on Jan  9 2022 13:35:05
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL PageSize:4096
Box64 with Dynarec v0.1.7 9bf7185 built on Jan  9 2022 13:14:15
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 31 Env var
Looking for /home/user/wine/bin/wineserver
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
wine: for some mysterious reason, the wine server never started.

New 'launcher' method Deleting my old wine, wine64, and wineserver scripts (making them symlinks instead) and trying the newer bash-preexec method, I get a similar result when trying to run ES.EXE:

user@localhost:~$ ES.EXE
ERROR: ld.so: object '/usr/lib/aarch64-linux-gnu/libgcc_s.so.1' from /etc/ld.so.preload cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
Box86 with Dynarec v0.2.5 88642d81 built on Jan  9 2022 13:35:05
ERROR: ld.so: object '/usr/lib/aarch64-linux-gnu/libgcc_s.so.1' from /etc/ld.so.preload cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
Box86 with Dynarec v0.2.5 88642d81 built on Jan  9 2022 13:35:05
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL PageSize:4096
Box64 with Dynarec v0.1.7 9bf7185 built on Jan  9 2022 13:14:15
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 32 Env var
Looking for /home/user/wine/bin/wineserver
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL PageSize:4096
Box64 with Dynarec v0.1.7 9bf7185 built on Jan  9 2022 13:14:15
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 32 Env var
Looking for /home/user/wine/bin/wineserver
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL PageSize:4096
Box64 with Dynarec v0.1.7 9bf7185 built on Jan  9 2022 13:14:15
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 32 Env var
Looking for /home/user/wine/bin/wineserver
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL PageSize:4096
Box64 with Dynarec v0.1.7 9bf7185 built on Jan  9 2022 13:14:15
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 32 Env var
Looking for /home/user/wine/bin/wineserver
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL PageSize:4096
Box64 with Dynarec v0.1.7 9bf7185 built on Jan  9 2022 13:14:15
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 32 Env var
Looking for /home/user/wine/bin/wineserver
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL PageSize:4096
Box64 with Dynarec v0.1.7 9bf7185 built on Jan  9 2022 13:14:15
Using default BOX64_LD_LIBRARY_PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/
Using default BOX64_PATH: ./:bin/
Counted 32 Env var
Looking for /home/user/wine/bin/wineserver
Using native(wrapped) libdl.so.2
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libpthread.so.0
Using native(wrapped) librt.so.1
wine: for some mysterious reason, the wine server never started.
hi
bash: ES.EXE: command not found
user@localhost:~$

[bbbruni]

wine: for some mysterious reason, the wine server never started.

The last time I saw this issue it was caused by a broken pre-compiled wine-development *.deb package. You have loaded a pre-compiled deb, right? If yes, see https://forum.winehq.org/viewtopic.php?t=35241&p=132898 And try to compile wine from source with minimal configuration: ./configure --without-x --without-freetype Don't install it! Then run wineboot -u

with stable 6.0, I still have to create manually a directory "/tmp/.wine-1000/server-xxxx" (again the same issue) but when done it works fine ....

[WheezyE]

I'm not using a pre-compiled deb, but good looking out! You can see the steps the installation box86 compilation takes in this script for Termux/Android.

Edit: I misunderstood - I thought you meant box86. I'll check the wine deb file

Edit: I think I'll still need to cross-compile an i386 wine installation regardless. Maybe I'll just try an i386 package from another source

[WheezyE]

Thanks for the wait and sorry if this is a bit off-topic, but I've made a little bit of progress getting wine working on aarch64 PRoot with @bbbruni's hint that the wine deb might be the issue: I tried other wineHQ installs, including an Ubuntu one (which would match my Termux Ubuntu proot-distro better), but still got crashes. So I made a new Termux proot-distro install using Debian instead of Ubuntu (Debian is a new proot-distro option that we didn't have before) and tried again with wine amd64 6.0.2 for Debian, but got crashes again. I downgraded to wine 5.0.0 and the 64bit Debian proot was able to run notepad++ x64 with box64! Here is the current experimental Termux install script to get box64 working on Termux:

Even though box64 is working on the Termux aarch64 Debian PRoot now, I'm still failing to get box86 working. I get this kind of output.

user@localhost:~$ DISPLAY=:1 WINEARCH=win32 /usr/local/bin/box86 /home/user/wine/bin/wine wineboot
-bash: /usr/local/bin/box86: No such file or directory

user@localhost:~$ sudo apt install libc6:armhf libncurses5:armhf libstdc++6:armhf -y
user@localhost:~$ DISPLAY=:1 WINEARCH=win32 /usr/local/bin/box86 /home/user/wine/bin/wine wineboot
Box86 with Dynarec v0.2.5 16b8657a built on Jan 16 2022 15:27:51
Box86 with Dynarec v0.2.5 16b8657a built on Jan 16 2022 15:27:51
Box86 with Dynarec v0.2.5 16b8657a built on Jan 16 2022 15:27:51
proot warning: ptrace(PEEKDATA): I/O error
wine client error:0: sendmsg: Bad address

I'm gonna keep trying to figure this out with logs from the TwisterOS discord "box86-on-aarch64-ubuntu-etc" channel, but just wanted to update current progress and 'work in the open'.

EDIT: Might be related to https://github.com/termux/proot/issues/70#issuecomment-586688823 EDIT2: I'm asking for help here too to see what the proot folks think https://github.com/termux/proot/issues/213

[WheezyE]

Just wanted to post an update that michalbednarski updated Termux so that box86 & box64 can both run from the Debian aarch64 PRoot using multiarch libraries. https://github.com/termux/proot/commit/d4e4a07fbdc16e24868c295d3f7de97eca2a7f80 I'll keep working on binfmt-like behavior and also see if I can make a PR on lowspecman's AnBox86 script for Termux pretty soon (to include box64)

EDIT: AnBox86 PR merged: https://github.com/lowspecman420/AnBox86/pull/8

[Botspot]

Just wanted to post what worked for me. Binfmt is a kernel module. So just because the OS doesn't use systemd doesn't mean it can't use binfmt.

I recently got Box86 binfmt working in a chroot with these commands:

sudo modprobe binfmt_misc
sudo mount -t binfmt_misc none /proc/sys/fs/binfmt_misc
echo ':x86:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x03\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/local/bin/box86:' | sudo tee /proc/sys/fs/binfmt_misc/register

Hopefully this info will help someone.

[WheezyE]

Thanks for giving this a shot and glad to hear that you got it working in chroot!

I tried your suggested commands on Termux proot and got this:

user@localhost:~$ sudo modprobe binfmt_misc
modprobe: FATAL: Module binfmt_misc not found in directory /lib/modules/5.4.0-faked

[Botspot]

Thanks for giving this a shot and glad to hear that you got it working in chroot!

I tried your suggested commands on Termux proot and got this:

user@localhost:~$ sudo modprobe binfmt_misc
modprobe: FATAL: Module binfmt_misc not found in directory /lib/modules/5.4.0-faked

Looks like your kernel doesn't have the binfmt_misc module. It might be built-in to the kernel. You can find out by checking if the /proc/sys/fs/binfmt_misc/register file exists. If it doesn't exist, then you're out of luck.

[WheezyE]

Ah yeah looks like that file is missing in Termux proot (and in regular Termux)

[ptitSeb]

Can this ticket be closed, it's not an issue with box86.

[Botspot]

Can this ticket be closed, it's not an issue with box86.

Go for it.