Cross-compiling Executorch SDK for Android

[Aeranthes]

Hi, I'm interested in operator profiling data from running Executorch + XNNPack on Android devices. For this purpose, I've been trying to cross compile the Executorch SDK for Android using the r26c Android NDK on Ubuntu 22.04 x86.

When building with cmake with -DEXECUTORCH_BUILD_SDK on, the project attempts to build the flatcc_cli tool to generate the etdump headers. Although it seems there's a clause implemented for cross compilation that runs a separate flatcc host installation, this still results in an exec format error as it tries to use a binary compiled for android on my host machine.

I'm running roughly as below:

git submodule sync
git submodule update --init

python3 -m venv executorch-env
source executorch-env/bin/activate

./install_requirements.sh --pybind xnnpack

cmake -DCMAKE_INSTALL_PREFIX=cmake-android-out \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_TOOLCHAIN_FILE="${ANDROID_NDK}/build/cmake/android.toolchain.cmake" \
  -DANDROID_ABI="${ANDROID_ABI}" \
  -DANDROID_PLATFORM="${ANDROID_PLATFORM}" \
  -DEXECUTORCH_BUILD_SDK=ON \
  -DEXECUTORCH_SEPARATE_FLATCC_HOST_PROJECT=ON \
  -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
  -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
  -DEXECUTORCH_BUILD_XNNPACK=ON \
  -DXNNPACK_ENABLE_ARM_BF16=OFF \
  -DEXECUTORCH_ENABLE_LOGGING=ON \
  -DEXECUTORCH_ENABLE_EVENT_TRACER=ON \
  -Bcmake-android-out .

cmake --build cmake-android-out -j3 --target install --config Release

00:13:10.352  [  5%] Generating etdump headers
00:13:10.352  /bin/sh: 1: /home/samwot/executorch/sdk/../third-party/flatcc/bin/flatcc: Exec format error
00:13:10.352  gmake[2]: *** [sdk/CMakeFiles/etdump_schema.dir/build.make:71: sdk/include/executorch/sdk/etdump/etdump_schema_flatcc_reader.h] Error 126
00:13:10.352  gmake[1]: *** [CMakeFiles/Makefile2:857: sdk/CMakeFiles/etdump_schema.dir/all] Error 2
00:13:10.352  gmake[1]: *** Waiting for unfinished jobs....

I've gotten around this by installing and building a separate x86 instance of flatcc manually (and changing /sdk/CMakeLists.txt::179 to use FLATCC_EXECUTABLE instead of ${_flatcc_source_dir}/bin/flatcc so I can give the path when running the cmake command). I was able to build the sdk_example_runner project this way and collect an etdump on my android device, albeit I haven't tried linking the delegate yet.

Is there a flow/flag I'm missing for cross compiling the SDK smoothly, or is this unintended/under development? I understand that flatcc has an issue in not being able to output its build items to a specific directory.

[lucylq]

@Aeranthes, thanks for trying it out! I can repro the error you're seeing. @Olivia-liu, can you take a look on the sdk side?

[kimishpatel]

labelling triaged to Olivia

[Olivia-liu]

Hey thanks a lot for the question! I just want to let you know that I'm looking into it and will get back to you.

[Olivia-liu]

I'm still looking at this but I think I've made an interesting observation. I was able to reproduce this consistently: If I run these Android cross-compile commands first (just run all the cmake commands, not including the "adb" commands), then run your commands, it builds with no problems. But if I just run your commands, I got the same error as you did or sometimes a different flatcc error like this one

ld: archive member '/' not a mach-o file in '/Users/olivia/executorch/third-party/flatcc/lib/libflatcc.a'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[5]: *** [/Users/olivia/executorch/third-party/flatcc/bin/flatcc] Error 1
make[4]: *** [src/cli/CMakeFiles/flatcc_cli.dir/all] Error 2
make[3]: *** [all] Error 2
make[2]: *** [_host_build/src/flatcc_project-stamp/flatcc_project-build] Error 2
make[1]: *** [sdk/CMakeFiles/flatcc_project.dir/all] Error 2

I tested on the main branch and used conda, but should be similar with venv.

[Olivia-liu]

Hi sorry I'm working on some other high priority items right now, so I haven't got the time to root cause this. Just to make sure, you're not blocked by this, do you? I'll come back to work on this issue in 3 weeks.

[Aeranthes]

No worries - thanks for looking into it. This is lower priority. I can work around this by creating two copies of flatcc - interestingly I don't seem to be able to follow the pattern you mentioned with running a build without flags first.