search

termux / termux-docker

Termux environment packaged as Docker image.
497 stars 70 forks source link

How to execute binaries of multiple platforms (WIP) #46

Closed lars18th closed 1 year ago

lars18th commented 1 year ago

Hi,

First, I want to share a simple test:

This proof-of-concept demonstrates that inside of the containers you aren't virtualizing the runtime. In fact, you're only virtualizing the processes (binaries). Therefore, you CAN execute binaries of both platforms inside the same container.

This works because the busybox binary is fully static (check it with ldd busybox). The problem begins when you want to execute processes that use shared libraries.

For the Termux libraries, a potential solution exist: using patchelf you can change the rpath to something different. For example, I've planed:

And copy on these "root" directories (/data/data/com.termux/<pseudo-arch>/) the patched version of the libraries. In my initial tests this should work. And you only need to patch binaries and libraries with: patchelf --set-rpath /data/data/com.termux/amd32/usr/lib --output tool.amd32 tool. Where the original tool binary is from the termux/termux-docker:latest image (so i686). And the tool.amd32 is the patched version that will work in any container.

The problem arrives when you need to handle with the standard bionic libraries at /system/lib<64>/, like libc.so. Is not a problem the ld-android.so library. You can copy it directly to the corresponding target directory (/data/data/com.termux/<pseudo-arch>/usr/lib) because it's fully static. And the binary will use this version. However, for the rest of the libraries (i.e. libc.so, libm.so, libdl.so, etc.) this doesn't work, because they depend of the ld-android.so and others. Therefore when launching the binary the versions of the running (aka emulated) platform in /system/lib<64>/ will be loaded.

The result is that you CAN'T execute any dynamic linking binary. Only static binaries (like busybox). So my question is: You know how to patch or recompile the bionic libraries to put then on an alternative directory?

My objective is to create a container (mainly for aarch64 arm architecture) with some tools with the x86_64 container. Therefore, I'll have something like: bash and bash.amd64. And I can call to the second to execute the same tool without slow emulation and at native speed. You can extrapolate this to some weighted tools (compilers and linkers?), or any other tool that you call a lot inside a heavy work (compiling?). And in a exceptional cases perhaps you want to remove the "native" (aka emulated) version and replace it with a symbolic link to the "external" (aka native) version.

So, please can you help me solve this problem? Thank you.

Grimler91 commented 1 year ago

patchelf has an option to change linker as well, --set-interpreter:

$ man patchelf
[...]
       --set-interpreter INTERPRETER
              Change the dynamic loader ("ELF interpreter") of executable
              given to INTERPRETER.
[...]

which I guess can help.

libc (and other libraries) are compiled from the sources in platform/bionic. To compile it the "standard" way (without using the Android.{bp,mk} files) you would have to write some Makefiles (maybe someone somewhere has already done this)

lars18th commented 1 year ago

Hi @Grimler91 ,

Thank you for the answer. However, I'm not sure how --set-interpreter can help. Let me to describe my final objective:

/data/data/com.termux/files/usr/bin/bash (target version) /data/data/com.termux/files/usr/bin/bash.arm64 (patched for aarch64) (ln -s /data/data/com.termux/arm64/usr/bin/bash) /data/data/com.termux/files/usr/bin/bash.arm32 (patched for arm) (ln -s /data/data/com.termux/arm32/usr/bin/bash) /data/data/com.termux/files/usr/bin/bash.amd32 (patched for i686) (ln -s /data/data/com.termux/amd32/usr/bin/bash) /data/data/com.termux/files/usr/bin/bash.amd64 (patched for x86_64) (ln -s /data/data/com.termux/amd64/usr/bin/bash)



Therefore, when installing the *.deb packages of the different architectures then it will possible to apply the `patchelf --set-rpath` to all libraries and binaries to point to `/data/data/com.termux/<pseudo-arch>/usr/lib`. One script could to this.

However, I've the problem with the Bionic libraries. So, I need to found a _straightforward_ solution. What you think is the best alternative:
- Recompile the full bionic package changing the target directory.
- Recompile the full bionic package changing the target directory and making all system libraries static.
- Patch the system libraries (how to do it?).
- Modify the system linker tool to search for libraries in different directory and patch the libraries (how to achieve this?).

In fact I'm not sure about the best solution. Please, comment about it.
Thank you.
lars18th commented 1 year ago

Hi,

Anyone interested in this, or is it a waste of time?

licy183 commented 1 year ago

I don't think this is possible... It may be able to run after patching the intercepter and RPATH but the hardcoded path in the binary will not be modified. Maybe you could try to use proot.

sylirre commented 1 year ago

While it indeed possible to run binaries of multiple architectures within same container, it would be problematic to make Termux environment multi-arch. At least not without modifying our patches for packages and recompiling everything. This is not aim of termux-docker images.

As of now you can go through "virtual prefix" approach. For example you have multiple prefixes:

The standard prefix /data/data/com.termux/files/usr is a symbolic link pointing on one of available prefix directories. You can make script that allows easy switching of prefix which simply replaces the symlink.

Or can use proot as alternative to symlink.

lars18th commented 1 year ago

Hi @sylirre ,

The idea for this proposal is to use emulated Termux containers with some native tools. Not to create a standard multi-arch environment. The reason is this: you wan to compile or test some executable in the Termux environment using the container. And the target platform is ARM64 (the current standard for Android devices). OK, so no problem: you configure your environment to run termux-docker:aarch64 with multiarch/qemu-user-static. Until here nothing new and all runs well... With the exception of the performance. If the platform is one x86_64 PC (the typical desktop) then you're amulating entirely the CPU. But this is not necessary if you recompile the tool for your container platform. This could be very useful for compiling inside Termux: imagine the compiler, linker, preprocessor, etc, of the target platform but compiled for the container platform. Then it will run at native speed. Great, don't think so?

As I explained at the start of the issue this could be done without much effort: All Termux tools (for any platform) could be patched to use libraries in an alternative path. And you can copy to these alternative pathes the stock Termux libraries of these other plaftorms. The problem is not this. The problem is with the Android native libraries. The libraries living at /system/lib<64>/. The current Termux environment is linked to use them, and we can't patch these pathes. So, to introduce the multi-arch support we need to copy them to /<arch>/system/lib<64>/ and recompile the full environment. Then it will possible to mix all architectures in the same container only adjusting pathes and patching binaries.

Please, reopen this topic to continue discussing about it with other users. My suggestion is to discuss the most simple way to achieve this: recompile the Termux environment for /system/lib<64>/ and pack all together inside a container. Thank you.