Error while generating android extension

[agunapal]

🐛 Describe the bug

I am following the instructions in https://pytorch.org/executorch/stable/demo-apps-android.html

Getting an error when I tried to install the android extension

cmake extension/android \
  -DCMAKE_TOOLCHAIN_FILE="${ANDROID_NDK}"/build/cmake/android.toolchain.cmake \
  -DANDROID_ABI="${ANDROID_ABI}" \
  -DCMAKE_INSTALL_PREFIX=cmake-android-out \
  -Bcmake-android-out/extension/android
CMake Deprecation Warning at /Users/agunapal/Library/Android/sdk/ndk/21.1.6352462/build/cmake/android.toolchain.cmake:35 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.
Call Stack (most recent call first):
  /Users/agunapal/miniforge3/envs/executorch/lib/python3.10/site-packages/cmake/data/share/cmake-3.31/Modules/CMakeDetermineSystem.cmake:146 (include)
  CMakeLists.txt:9 (project)

-- ANDROID_PLATFORM not set. Defaulting to minimum supported version
16.
CMake Deprecation Warning at /Users/agunapal/Library/Android/sdk/ndk/21.1.6352462/build/cmake/android.toolchain.cmake:35 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.
Call Stack (most recent call first):
  /Users/agunapal/Documents/pytorch/executorch/cmake-android-out/extension/android/CMakeFiles/3.31.0/CMakeSystem.cmake:6 (include)
  CMakeLists.txt:9 (project)

-- Detecting C compiler ABI info
CMake Deprecation Warning at /Users/agunapal/Library/Android/sdk/ndk/21.1.6352462/build/cmake/android.toolchain.cmake:35 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.
Call Stack (most recent call first):
  /Users/agunapal/Documents/pytorch/executorch/cmake-android-out/extension/android/CMakeFiles/3.31.0/CMakeSystem.cmake:6 (include)
  /Users/agunapal/Documents/pytorch/executorch/cmake-android-out/extension/android/CMakeFiles/CMakeScratch/TryCompile-NlufAA/CMakeLists.txt:4 (project)

-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Users/agunapal/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
CMake Deprecation Warning at /Users/agunapal/Library/Android/sdk/ndk/21.1.6352462/build/cmake/android.toolchain.cmake:35 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.
Call Stack (most recent call first):
  /Users/agunapal/Documents/pytorch/executorch/cmake-android-out/extension/android/CMakeFiles/3.31.0/CMakeSystem.cmake:6 (include)
  /Users/agunapal/Documents/pytorch/executorch/cmake-android-out/extension/android/CMakeFiles/CMakeScratch/TryCompile-gQVBGb/CMakeLists.txt:4 (project)

-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Users/agunapal/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Deprecation Warning at /Users/agunapal/Documents/pytorch/executorch/examples/third-party/fbjni/CMakeLists.txt:15 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

CMake Warning (dev) at CMakeLists.txt:30 (find_package):
  Policy CMP0144 is not set: find_package uses upper-case <PACKAGENAME>_ROOT
  variables.  Run "cmake --help-policy CMP0144" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  CMake variable EXECUTORCH_ROOT is set to:

    /Users/agunapal/Documents/pytorch/executorch/extension/android/../..

  For compatibility, find_package is ignoring the variable, but code in a
  .cmake module might still use it.
This warning is for project developers.  Use -Wno-dev to suppress it.

etdump library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
bundled_program library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
flatccrt library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
coremldelegate library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
mpsdelegate library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
qnn_executorch_backend library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
vulkan_backend library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
optimized_kernels library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
cpublas library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
eigen_blas library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
optimized_ops_lib library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
optimized_native_cpu_ops_lib library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
quantized_kernels library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
quantized_ops_lib library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
quantized_ops_aot_lib library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
-- Configuring done (1.5s)
-- Generating done (0.1s)
-- Build files have been written to: /Users/agunapal/Documents/pytorch/executorch/cmake-android-out/extension/android
(executorch) agunapal@agunapal-mbp executorch % cmake --build cmake-android-out/extension/android -j16
[  5%] Building CXX object third-party/fbjni/CMakeFiles/fbjni.dir/cxx/fbjni/ByteBuffer.cpp.o
[ 23%] Building CXX object third-party/fbjni/CMakeFiles/fbjni.dir/cxx/fbjni/OnLoad.cpp.o
[ 23%] Building CXX object third-party/fbjni/CMakeFiles/fbjni.dir/cxx/fbjni/ReadableByteChannel.cpp.o
[ 23%] Building CXX object third-party/fbjni/CMakeFiles/fbjni.dir/cxx/lyra/lyra_breakpad.cpp.o
[ 47%] Building CXX object third-party/fbjni/CMakeFiles/fbjni.dir/cxx/fbjni/detail/Environment.cpp.o
[ 47%] Building CXX object third-party/fbjni/CMakeFiles/fbjni.dir/cxx/fbjni/detail/Meta.cpp.o
[ 47%] Building CXX object third-party/fbjni/CMakeFiles/fbjni.dir/cxx/lyra/lyra_exceptions.cpp.o
[ 47%] Building CXX object third-party/fbjni/CMakeFiles/fbjni.dir/cxx/lyra/cxa_throw.cpp.o
[ 52%] Building CXX object third-party/fbjni/CMakeFiles/fbjni.dir/cxx/fbjni/detail/Hybrid.cpp.o
[ 58%] Building CXX object third-party/fbjni/CMakeFiles/fbjni.dir/cxx/fbjni/fbjni.cpp.o
[ 64%] Building CXX object third-party/fbjni/CMakeFiles/fbjni.dir/cxx/fbjni/detail/Exceptions.cpp.o
[ 76%] Building CXX object third-party/fbjni/CMakeFiles/fbjni.dir/cxx/fbjni/detail/References.cpp.o
[ 76%] Building CXX object third-party/fbjni/CMakeFiles/fbjni.dir/cxx/lyra/lyra.cpp.o
[ 82%] Building CXX object third-party/fbjni/CMakeFiles/fbjni.dir/cxx/fbjni/detail/utf8.cpp.o
[ 88%] Linking CXX shared library libfbjni.so
[ 88%] Built target fbjni
[ 94%] Building CXX object CMakeFiles/executorch_jni.dir/jni/jni_layer.cpp.o
[100%] Linking CXX shared library libexecutorch_jni.so
/Users/agunapal/Documents/pytorch/executorch/cmake-android-out/lib/libextension_module.a(module.cpp.o): In function `executorch::extension::Module::load(executorch::runtime::Program::Verification)':
/Users/agunapal/Documents/pytorch/executorch/extension/module/module.cpp:89: undefined reference to `executorch::extension::FileDataLoader::from(char const*, unsigned long)'
/Users/agunapal/Documents/pytorch/executorch/extension/module/module.cpp:92: undefined reference to `executorch::extension::MmapDataLoader::from(char const*, executorch::extension::MmapDataLoader::MlockConfig)'
/Users/agunapal/Documents/pytorch/executorch/extension/module/module.cpp:97: undefined reference to `executorch::extension::MmapDataLoader::from(char const*, executorch::extension::MmapDataLoader::MlockConfig)'
/Users/agunapal/Documents/pytorch/executorch/extension/module/module.cpp:100: undefined reference to `executorch::extension::MmapDataLoader::from(char const*, executorch::extension::MmapDataLoader::MlockConfig)'
/Users/agunapal/Documents/pytorch/executorch/cmake-android-out/lib/libextension_module.a(module.cpp.o): In function `~Result':
/Users/agunapal/Documents/pytorch/executorch/extension/module/../../../executorch/runtime/core/result.h:85: undefined reference to `executorch::extension::FileDataLoader::~FileDataLoader()'
/Users/agunapal/Documents/pytorch/executorch/extension/module/../../../executorch/runtime/core/result.h:85: undefined reference to `executorch::extension::MmapDataLoader::~MmapDataLoader()'
/Users/agunapal/Documents/pytorch/executorch/cmake-android-out/lib/libextension_module.a(module.cpp.o): In function `FileDataLoader':
/Users/agunapal/Documents/pytorch/executorch/extension/module/../../../executorch/extension/data_loader/file_data_loader.h:(.text._ZN10executorch9extension14FileDataLoaderC2EOS1_[_ZN10executorch9extension14FileDataLoaderC2EOS1_]+0xc): undefined reference to `vtable for executorch::extension::FileDataLoader'
/Users/agunapal/Documents/pytorch/executorch/extension/module/../../../executorch/extension/data_loader/file_data_loader.h:(.text._ZN10executorch9extension14FileDataLoaderC2EOS1_[_ZN10executorch9extension14FileDataLoaderC2EOS1_]+0x10): undefined reference to `vtable for executorch::extension::FileDataLoader'
/Users/agunapal/Documents/pytorch/executorch/cmake-android-out/lib/libextension_module.a(module.cpp.o): In function `std::__ndk1::default_delete<executorch::extension::FileDataLoader>::operator()(executorch::extension::FileDataLoader*) const':
/Users/agunapal/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/c++/v1/memory:2338: undefined reference to `executorch::extension::FileDataLoader::~FileDataLoader()'
/Users/agunapal/Documents/pytorch/executorch/cmake-android-out/lib/libextension_module.a(module.cpp.o): In function `MmapDataLoader':
/Users/agunapal/Documents/pytorch/executorch/extension/module/../../../executorch/extension/data_loader/mmap_data_loader.h:(.text._ZN10executorch9extension14MmapDataLoaderC2EOS1_[_ZN10executorch9extension14MmapDataLoaderC2EOS1_]+0xc): undefined reference to `vtable for executorch::extension::MmapDataLoader'
/Users/agunapal/Documents/pytorch/executorch/extension/module/../../../executorch/extension/data_loader/mmap_data_loader.h:(.text._ZN10executorch9extension14MmapDataLoaderC2EOS1_[_ZN10executorch9extension14MmapDataLoaderC2EOS1_]+0x10): undefined reference to `vtable for executorch::extension::MmapDataLoader'
/Users/agunapal/Documents/pytorch/executorch/cmake-android-out/lib/libextension_module.a(module.cpp.o): In function `std::__ndk1::default_delete<executorch::extension::MmapDataLoader>::operator()(executorch::extension::MmapDataLoader*) const':
/Users/agunapal/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/c++/v1/memory:2338: undefined reference to `executorch::extension::MmapDataLoader::~MmapDataLoader()'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [libexecutorch_jni.so] Error 1
make[1]: *** [CMakeFiles/executorch_jni.dir/all] Error 2
make: *** [all] Error 2

I am not sure if the documentation is out-dated. Can we add a generic script to automate all these steps?

Versions

Collecting environment information...
PyTorch version: 2.5.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 15.1 (arm64)
GCC version: Could not collect
Clang version: 16.0.0 (clang-1600.0.26.4)
CMake version: version 3.31.0
Libc version: N/A

Python version: 3.10.0 | packaged by conda-forge | (default, Nov 20 2021, 02:27:15) [Clang 11.1.0 ] (64-bit runtime)
Python platform: macOS-15.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] executorch==0.4.0a0+6a085ff
[pip3] numpy==1.21.3
[pip3] torch==2.5.0
[pip3] torchaudio==2.5.0
[pip3] torchsr==1.0.4
[pip3] torchvision==0.20.0
[conda] executorch                0.4.0a0+6a085ff          pypi_0    pypi
[conda] numpy                     1.21.3                   pypi_0    pypi
[conda] torch                     2.5.0                    pypi_0    pypi
[conda] torchaudio                2.5.0                    pypi_0    pypi
[conda] torchsr                   1.0.4                    pypi_0    pypi
[conda] torchvision               0.20.0                   pypi_0    pypi

[metascroy]

cc @kirklandsign for android build issue

[kirklandsign]

Can we add a generic script to automate all these steps?

Good suggestions!

[kirklandsign]

Hi @agunapal

Did you do these steps correctly?

cmake . -DCMAKE_INSTALL_PREFIX=cmake-android-out \
  -DCMAKE_TOOLCHAIN_FILE="${ANDROID_NDK}/build/cmake/android.toolchain.cmake" \
  -DANDROID_ABI="${ANDROID_ABI}" \
  -DEXECUTORCH_BUILD_XNNPACK=ON \
  -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
  -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
  -DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
  -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
  -Bcmake-android-out

cmake --build cmake-android-out -j16 --target install

[kirklandsign]

Also you used NDK 21 but we recommend the latest version r27c

https://developer.android.com/ndk/downloads

[agunapal]

Thanks @kirklandsign Issue was resolved after upgrading NDK version!