x86_64/aarch64, help making this work with https://gitlab.com/android_translation_layer/android_translation_layer?

I'm trying to make this work with https://gitlab.com/android_translation_layer/android_translation_layer. At first I tried using the build system we have for https://gitlab.com/android_translation_layer/art_standalone (see patches below), but it seems that some files are missing, and arm is harcoded in some paths. I assume there will be some other issues.

Would it be possible to simply use the original build system to build qemu, so as to sidestep these issues? Is __ANDROID__ being defined necessary for the qemu fork to work correctly, or is it only necessary to build for AOSP target?

Seeing that there doesn't seem to be too much interest in this project as-is, would you like to help with getting this running with android_translation_layer (we like this compared to libhoudini, since it's open source, which is always a plus, and it's also practical since it means it can be modified to work more sanely with a standard Linux system)

diff --git a/Android.mk b/Android.mk
index bc2abd7..ddf3972 100644
--- a/Android.mk
+++ b/Android.mk
@@ -17,6 +17,6 @@
 # along with this program; if not, see <http://www.gnu.org/licenses/>.
 #

-QEMU_HOST_ARCH := i386
-QEMU_TARGET_ARCH := arm
+QEMU_HOST_ARCH := x86_64
+QEMU_TARGET_ARCH := aarch64
 include $(call all-subdir-makefiles)
diff --git a/capstone b/capstone
deleted file mode 160000
index 0ea7272..0000000
--- a/capstone
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit 0ea72729858bcefe9edd32bade421278b1d41ca1
diff --git a/glib b/glib
deleted file mode 160000
index 0d2cfc7..0000000
--- a/glib
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit 0d2cfc78da3e1507c2258dc52bb34e6ea9a08e93
diff --git a/libnb-qemu/Android.mk b/libnb-qemu/Android.mk
index 94bb85e..2e38d76 100644
--- a/libnb-qemu/Android.mk
+++ b/libnb-qemu/Android.mk
@@ -40,7 +40,7 @@ LOCAL_STATIC_LIBRARIES := libqemu-glib libqemu-capstone libffi
 LOCAL_CFLAGS := $(LOCAL_QEMU_CFLAGS)
 LOCAL_C_INCLUDES := system/core/libnativebridge/include
 LOCAL_EXPORT_STATIC_LIBRARY_HEADERS := libqemu-core libqemu-target libqemu-glib libffi
-include $(BUILD_SHARED_LIBRARY)
+include $(BUILD_HOST_SHARED_LIBRARY)

 include $(CLEAR_VARS)
 LOCAL_MODULE := libnb-qemu-EGL
@@ -48,8 +48,9 @@ LOCAL_SRC_FILES := libEGL.c libEGL.itf.cpp
 LOCAL_CFLAGS := \
    -fvisibility=hidden $(LOCAL_QEMU_CFLAGS) \
    -DGL_GLEXT_PROTOTYPES -DEGL_EGLEXT_PROTOTYPES
-LOCAL_SHARED_LIBRARIES := libnb-qemu liblog libEGL
-include $(BUILD_SHARED_LIBRARY)
+LOCAL_SHARED_LIBRARIES := libnb-qemu liblog
+LOCAL_LDLIBS := -lEGL
+include $(BUILD_HOST_SHARED_LIBRARY)

 include $(CLEAR_VARS)
 LOCAL_MODULE := libnb-qemu-GLESv3
@@ -57,8 +58,9 @@ LOCAL_SRC_FILES := libGLESv3.c
 LOCAL_CFLAGS := \
    -fvisibility=hidden $(LOCAL_QEMU_CFLAGS) \
    -DGL_GLEXT_PROTOTYPES -DEGL_EGLEXT_PROTOTYPES
-LOCAL_SHARED_LIBRARIES := libnb-qemu liblog libGLESv3
-include $(BUILD_SHARED_LIBRARY)
+LOCAL_SHARED_LIBRARIES := libnb-qemu liblog
+LOCAL_LDLIBS := libEGL
+include $(-lGLESv3)

 include $(CLEAR_VARS)
 LOCAL_MODULE := libnb-qemu-GLESv1_CM
@@ -66,21 +68,24 @@ LOCAL_SRC_FILES := libGLESv1_CM.c
 LOCAL_CFLAGS := \
    -fvisibility=hidden $(LOCAL_QEMU_CFLAGS) \
    -DGL_GLEXT_PROTOTYPES -DEGL_EGLEXT_PROTOTYPES
-LOCAL_SHARED_LIBRARIES := libnb-qemu liblog libGLESv1_CM
-include $(BUILD_SHARED_LIBRARY)
+LOCAL_SHARED_LIBRARIES := libnb-qemu liblog
+LOCAL_LDLIBS := -lGLESv1_CM
+include $(BUILD_HOST_SHARED_LIBRARY)

 include $(CLEAR_VARS)
 LOCAL_MODULE := libnb-qemu-OpenSLES
 LOCAL_SRC_FILES := libOpenSLES.c libOpenSLES.itf.cpp
 LOCAL_CFLAGS := \
    -fvisibility=hidden $(LOCAL_QEMU_CFLAGS)
-LOCAL_SHARED_LIBRARIES := libnb-qemu liblog libOpenSLES
-include $(BUILD_SHARED_LIBRARY)
+LOCAL_SHARED_LIBRARIES := libnb-qemu liblog
+LOCAL_LDLIBS := -lOpenSLES
+include $(BUILD_HOST_SHARED_LIBRARY)

 include $(CLEAR_VARS)
 LOCAL_MODULE := libnb-qemu-android
 LOCAL_SRC_FILES := libandroid.c libandroid.itf.cpp
 LOCAL_CFLAGS := \
    -fvisibility=hidden $(LOCAL_QEMU_CFLAGS)
-LOCAL_SHARED_LIBRARIES := libnb-qemu liblog libandroid
-include $(BUILD_SHARED_LIBRARY)
+LOCAL_SHARED_LIBRARIES := libnb-qemu liblog
+LOCAL_LDLIBS := -landroid
+include $(BUILD_HOST_SHARED_LIBRARY)
diff --git a/qemu b/qemu
--- a/qemu
+++ b/qemu
@@ -1 +1 @@
-Subproject commit 7b06ff77ab9d048d7ed3cdb08a136e97c03f43f4
+Subproject commit 7b06ff77ab9d048d7ed3cdb08a136e97c03f43f4-dirty

diff --git a/Android.mk b/Android.mk
index 7b0933cc72..645fc561d7 100644
--- a/Android.mk
+++ b/Android.mk
@@ -226,14 +226,14 @@ LOCAL_MODULE := libqemu-core
 LOCAL_SRC_FILES := $(QEMU_CORE_SRC_FILES)
 LOCAL_CFLAGS := $(QEMU_CORE_CFLAGS)
 LOCAL_C_INCLUDES := $(QEMU_CORE_C_INCLUDES)
-LOCAL_STATIC_LIBRARIES := libqemu-glib libqemu-capstone
+LOCAL_LDLIBS := -lcapstone -lglib
 LOCAL_EXPORT_C_INCLUDE_DIRS := \
    $(LOCAL_PATH) \
    $(LOCAL_PATH)/tcg \
    $(LOCAL_PATH)/tcg/$(QEMU_HOST_ARCH) \
    $(LOCAL_PATH)/accel/tcg \
    $(LOCAL_PATH)/include
-include $(BUILD_STATIC_LIBRARY)
+include $(BUILD_HOST_STATIC_LIBRARY)

 QEMU_TARGET_SRC_FILES := \
    disas.c \
@@ -315,20 +315,19 @@ LOCAL_MODULE := libqemu-target
 LOCAL_SRC_FILES := $(QEMU_TARGET_SRC_FILES)
 LOCAL_CFLAGS := $(QEMU_TARGET_CFLAGS)
 LOCAL_C_INCLUDES := $(QEMU_TARGET_C_INCLUDES)
-LOCAL_STATIC_LIBRARIES := libqemu-glib libqemu-capstone
-LOCAL_SHARED_LIBRARIES := libz
+LOCAL_LDLIBS := -lcapstone -lglib -lz
 LOCAL_EXPORT_C_INCLUDE_DIRS := \
    $(LOCAL_PATH)/target/$(QEMU_TARGET_ARCH) \
    $(LOCAL_PATH)/linux-user/$(QEMU_TARGET_ARCH) \
    $(LOCAL_PATH)/linux-user/host/$(QEMU_HOST_ARCH) \
    $(LOCAL_PATH)/linux-user/
-include $(BUILD_STATIC_LIBRARY)
+include $(BUILD_HOST_STATIC_LIBRARY)

 include $(CLEAR_VARS)
 LOCAL_MODULE := qemu-arm
-LOCAL_SRC_FILES := linux-user/android-main.c
+LOCAL_SRC_FILES := linux-user/main.c
 # We need the functions marked with __attribute__((constructor)), so include whole archives
 LOCAL_WHOLE_STATIC_LIBRARIES := libqemu-target libqemu-core
-LOCAL_STATIC_LIBRARIES := libqemu-capstone libqemu-glib
-LOCAL_SHARED_LIBRARIES := libz liblog
-include $(BUILD_EXECUTABLE)
+LOCAL_LDLIBS := -lcapstone -lglib -lz
+LOCAL_SHARED_LIBRARIES := liblog
+include $(BUILD_HOST_EXECUTABLE)
diff --git a/roms/qboot b/roms/qboot
index cb1c49e0cf..8ca302e86d 160000
--- a/roms/qboot
+++ b/roms/qboot
@@ -1 +1 @@
-Subproject commit cb1c49e0cfac99b9961d136ac0194da62c28cf64
+Subproject commit 8ca302e86d685fa05b16e2b208888243da319941-dirty

As you can see, the project has been stalled since May 2020. I don't know how to write code, but I have been interested in Android native bridge. I hope this information is helpful to you and this project.

At present, there is a native bridge implementation called berberis on the Android mainline source code, which is actually a riscv version of the closed source ndk_translation developed by Google for ARM to x86 (located in frameworks/libs/binary_translation). Starting from Android 11, some code for building system proxy libraries for ndk_translation has been added to the source code located in frameworks/libs/native_bridge_support. Recently, in the mainline source code, this part of the code has been supplemented with a lot of content on building proxy libraries for Berberis to build (in fact, most functions and the implementation of ndk_translation are the same, including some design of Berberis).

I suggest that you can refer to Berberis and native_bridge_support to refactor or improve this project. Moreover, QEMU can also achieve translation between other architectures, and this project will be very helpful.

I'm looking forward to using this with waydroid. I can code in C/C++(just can), however I haven't any experience in the Android world.

well, I added aarch64 support and made this work with ATL: https://gitlab.com/android_translation_layer/qemu-based-native-bridge/nb-qemu if anyone is interested in fixing the AOSP support in a reasonable way, be my guest, but building dependencies (glib, capstone, qemu) for android should not be part of this repo.

fwiw, for waydroid, which runs a full AOSP system, there are no practical downsides to using the proprietary libhoudini, so I doubt they really care all that much, unless someone else does all the work for them to bring this to feature parity

I tested native tunnel (ndk example) and smashhit v1.4.3 and they run at 60 fps with occasional drops

well, I added aarch64 support and made this work with ATL: https://gitlab.com/android_translation_layer/qemu-based-native-bridge/nb-qemu if anyone is interested in fixing the AOSP support in a reasonable way, be my guest, but building dependencies (glib, capstone, qemu) for android should not be part of this repo.

Oh you've gone further. Is the guest/bionic part build on aosp-q? I will try it.

And I've come up with some new ideas. Build qemu-static separately, add binder/ashmem support to qemu. Then I think I can let the bridge run guest-support-program with qemu-static by exec() in the subprocess, then use some IPC implementation (like unix domain socket, shm or binder) to interact with the guest-program in qemu. This bridge implementation could easily switch the qemu-static to houdini-static or etc.. It also allows qemu to be built independently of bridge. But the efficiency of IPC may slow down the trampoline/thunk conversion.

Currently I have few time and try to patch on new qemu. But I think once the two solutions are both ready then I could test the benchmark between them :)

I don't think that makes much sense, we need non-upstreamable patches to qemu anyway. Houdini is also in-process afaik, and it's closed source so you can't exactly make it not be. And there's no way it wouldn't be slower, it would also make thunking pointers more annoying. Also, binder/ashmem are not a thing with ATL so I'm certainly not interested in maintaining that.

The guest part is cross-compiled with no outside dependencies, so it should work as-is on android. It probably doesn't even matter what the host android version is.

I don't think that makes much sense, we need non-upstreamable patches to qemu anyway. Houdini is also in-process afaik, and it's closed source so you can't exactly make it not be. And there's no way it wouldn't be slower, it would also make thunking pointers more annoying. Also, binder/ashmem are not a thing with ATL so I'm certainly not interested in maintaining that.

Yes I am aware of the issues. But I have been tried the original nb-qemu build, patching libffi as instructed. the result is that it doesn't work in the android emulator, maybe I'm missing something. And the qemu-arm binfmt_misc function is broken. cant find way to fix the problem that causes Android to freeze when starting up. The original qemu's Android makefile and the dependencies on capstone, glib, etc. are very annoying in Android builds. And my goal is not to run on ATL, but the current Android platform.

So in my design I don't care about performance, I just want to be able to minimally modify qemu so that I don't have to port it to the Android build system. Because the original qemu-static can be used directly as a binfmt_misc in Android without patch. Just changing the linker path or LD_PREFIX, qemu can run ndk's ELF smoothly. So it would be possible for me to simply just have the bridge execute a designed guest program with qemu-static, communicating via local socket IPC and wrapping the thunk with libffi & IPC.

Houdini-static (exist in houdini 12 and below, no static programs above houdini 13) fully supports all ioctl for android bionic, including binder/ashmem IPC. therefore, the thunk performance of bridge can be improved with the help of better IPC design. And since the translator runs in a subprocess, Houdini or even other linux translators (preferably static) such as box64 have the opportunity to run the guest program instead of qemu, and provide performance comparisons. This will help in improving and debugging other translators and also compare various issues during the runtime of the guest program.

BTW I will try your nb-qemu (not very soon) to see if it runs on android since I do want to know its performance in nativebridge.

it seems that termux was able to compile qemu for android: https://github.com/termux/termux-packages/tree/e358cd8718dd9ef9d7f1194535cf86cc20f0eb9e/packages/qemu-system-x86-64-headless

Termux compiles everything natively against bionic, so it should work properly in-process. You could also use their glib/capstone etc I imagine.

It seems they are using a newer version of Qemu, if you decide to forward port the patches I would be very grateful if you didn't break ATL support (which I guess shouldn't be that hard, all I did to make that work properly was remove #ifdef __ANDROID__ around parts which are actually needed unconditionally (originally they seemed to be used such that compiling for Linux would produce a "normal" qemu, which is kinda pointless since if someone wanted that they would just use upstream qemu))

Alternatively could probably go through the git history, seems they had that package for quite a while

I suppose you could try to use termux's build system to build the native bridge, just need to link against art in addition to termux-provided stuff like glib/capstone/libffi but that should be fine (or against nativehelper, maybe it makes a difference on AOSP). I assume their build system should be able to cope with my Makefiles, maybe with some trivial changes.

goffioul / ax86-nb-qemu

x86_64/aarch64, help making this work with https://gitlab.com/android_translation_layer/android_translation_layer? #2