search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.31k stars 986 forks source link

[question] How to properly cross-compile for Android with android-ndk package? #11096

Closed nmgwddj closed 1 month ago

nmgwddj commented 2 years ago

In the official documentation, I found the following link: https://docs.conan.io/en/latest/integrations/cross_platform/android.html#using-android-ndk-package-tool-require, through which I learned that android-ndk can be referenced through tool_requires. The documentation states that when we use this package, the toolchain file and the variables required by CMake are generated. But when I execute conan install and then pass CMake include conanbuildinfo.cmake, I get the following error when I execute CMake initialization:

[cmake] CMake Error at /opt/homebrew/Cellar/cmake/3.23.0/share/cmake/Modules/Platform/Android-Determine.cmake:209 (message):
[cmake]   Android: Neither the NDK or a standalone toolchain was found.
[cmake] Call Stack (most recent call first):
[cmake]   /opt/homebrew/Cellar/cmake/3.23.0/share/cmake/Modules/CMakeDetermineSystem.cmake:160 (include)
[cmake]   CMakeLists.txt:24 (project)

Obviously, Android NDK related environment variables are not injected into CMake. Then I saw in the description of the android-ndk package that CMakeDeps and CMakeToolchain need to be specified in generators, but when I After setting up both generators and doing conan install I get the following error:

[cmake] conanfile.py (conan-cross-platform-build/None): Generator 'CMakeToolchain' calling 'generate()'
[cmake] conanfile.py (conan-cross-platform-build/None): ERROR: Traceback (most recent call last):
[cmake]   File "/usr/local/Cellar/conan/1.47.0/libexec/lib/python3.10/site-packages/conans/client/generators/__init__.py", line 177, in write_generators
[cmake]     generator.generate()
[cmake]   File "/usr/local/Cellar/conan/1.47.0/libexec/lib/python3.10/site-packages/conan/tools/cmake/toolchain/toolchain.py", line 164, in generate
[cmake]     save(self.filename, self.content)
[cmake]   File "/usr/local/Cellar/conan/1.47.0/libexec/lib/python3.10/site-packages/conan/tools/cmake/toolchain/toolchain.py", line 157, in content
[cmake]     context = self._context()
[cmake]   File "/usr/local/Cellar/conan/1.47.0/libexec/lib/python3.10/site-packages/conan/tools/cmake/toolchain/toolchain.py", line 144, in _context
[cmake]     blocks = self.blocks.process_blocks()
[cmake]   File "/usr/local/Cellar/conan/1.47.0/libexec/lib/python3.10/site-packages/conan/tools/cmake/toolchain/blocks.py", line 43, in process_blocks
[cmake]     content = b.get_rendered_content()
[cmake]   File "/usr/local/Cellar/conan/1.47.0/libexec/lib/python3.10/site-packages/conan/tools/cmake/toolchain/blocks.py", line 66, in get_rendered_content
[cmake]     context = self.values
[cmake]   File "/usr/local/Cellar/conan/1.47.0/libexec/lib/python3.10/site-packages/conan/tools/cmake/toolchain/blocks.py", line 58, in values
[cmake]     self._context_values = self.context()
[cmake]   File "/usr/local/Cellar/conan/1.47.0/libexec/lib/python3.10/site-packages/conan/tools/cmake/toolchain/blocks.py", line 308, in context
[cmake]     raise ConanException('CMakeToolchain needs tools.android:ndk_path configuration defined')
[cmake] conans.errors.ConanException: CMakeToolchain needs tools.android:ndk_path configuration defined

I've been tossing around for days, and I haven't been able to find a solution by constantly searching for issues, google, conan index example, etc.

The requirements are very simple. I hope to manage NDK through conan tool requires, and correctly inject various environment variables and toolchain information of NDK into my existing CMakeLists.txt and then perform cross-compilation. Because I have implemented various cross-compilation scenarios for Windows, macOS, Linux, and iOS through the existing Conan capabilities, and now it is only Android, I am very distressed.

Can someone give a complete example and tell me how to use it?

memsharded commented 2 years ago

Hi @nmgwddj

I'd probably recommend trying to use the new build system integrations, that includes some explicit Android integrations:

That would be a good starting point. Then from there, I would try to use the NDK from a tool_require, but I suggest doing that later.

nmgwddj commented 2 years ago

Thanks to @memsharded for your patient reply, I can successfully compile a hello package and a test program through the example you gave. But when I apply these in my existing project, I still can't finish cross-compiling.

First of all, in this case, I am not a package maker, and some Conan center packages are referenced in my existing CMake project. We will not make this CMake project into a Conan package, we just hope that the current CMake project can be compiled under different platforms (the practice of other platforms is to introduce conan.cmake to automatically perform the conan install process by passing different profiles)

I have installed the NDK and specified the absolute path of the profile in the host profile

[settings]
arch=armv8
compiler=clang
compiler.libcxx=libc++
compiler.version=14
os=Android
os.api_level=21

[conf]
tools.android:ndk_path=/Users/admin/Library/Android/sdk/ndk/23.1.7779620

The conanfile.py of my current CMake project is as follows:

from conans import ConanFile
from conan.tools.cmake import CMake, cmake_layout

class NebaseConan(ConanFile):
    name = "conan-cross-platform-build"
    author = "Dylan <2894220@gmail.com>"
    description = "An example of a C++ app cross-platform build by Conan"
    settings = "os", "compiler", "build_type", "arch"
    generators = "CMakeDeps", "CMakeToolchain", "VirtualBuildEnv", "VirtualRunEnv"
    exports_sources = "*"

    def requirements(self):
        self.requires("jsoncpp/1.9.5")

    def config_options(self):
        if self.settings.os == "Windows":
            del self.options.fPIC

    def layout(self):
        cmake_layout(self)

    def build(self):
        cmake = CMake(self)
        cmake.configure()
        cmake.build()

I simply cited a jsoncpp as an example, then created a buil directory and executed conan install in the build directory

conan install .. --profile:build=default --profile:host=/Users/admin/Documents/temporary/conan-cross-platform-build/.profiles/android-armv8 --build=missing

Then I get the error below, is there any step I'm missing?

Configuration (profile_host):
[settings]
arch=armv8
compiler=clang
compiler.libcxx=libc++
compiler.version=14
os=Android
os.api_level=21
[options]
[build_requires]
[env]
[conf]
tools.android:ndk_path=/Users/admin/Library/Android/sdk/ndk/23.1.7779620

Configuration (profile_build):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=13.1
os=Macos
os_build=Macos
[options]
[build_requires]
[env]
CFLAGS=-fvisibility=hidden -fvisibility-inlines-hidden -mmacosx-version-min=10.13
CXXFLAGS=-fvisibility=hidden -fvisibility-inlines-hidden -mmacosx-version-min=10.13
jsoncpp/1.9.5: WARN: Package binary is corrupted, removing: 0a745a0fa934c8048a9bc8af0ba9e3afefff82ff
conanfile.py (conan-cross-platform-build/None): Installing package
Requirements
    jsoncpp/1.9.5 from 'conancenter' - Cache
Packages
    jsoncpp/1.9.5:0a745a0fa934c8048a9bc8af0ba9e3afefff82ff - Build

Cross-build from 'Macos:x86_64' to 'Android:armv8'
Installing (downloading, building) binaries...
jsoncpp/1.9.5: WARN: Build folder is dirty, removing it: /Users/admin/.conan/data/jsoncpp/1.9.5/_/_/build/0a745a0fa934c8048a9bc8af0ba9e3afefff82ff
jsoncpp/1.9.5: Copying sources to build folder
jsoncpp/1.9.5: Building your package in /Users/admin/.conan/data/jsoncpp/1.9.5/_/_/build/0a745a0fa934c8048a9bc8af0ba9e3afefff82ff
jsoncpp/1.9.5: Generator cmake created conanbuildinfo.cmake
jsoncpp/1.9.5: Aggregating env generators
jsoncpp/1.9.5: Calling build()
CMake Error at /usr/local/Cellar/cmake/3.21.3_1/share/cmake/Modules/Platform/Android-Determine.cmake:207 (message):
  Android: Neither the NDK or a standalone toolchain was found.
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.21.3_1/share/cmake/Modules/CMakeDetermineSystem.cmake:160 (include)
  CMakeLists.txt:2 (project)

CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
jsoncpp/1.9.5: 
jsoncpp/1.9.5: ERROR: Package '0a745a0fa934c8048a9bc8af0ba9e3afefff82ff' build failed
jsoncpp/1.9.5: WARN: Build folder /Users/admin/.conan/data/jsoncpp/1.9.5/_/_/build/0a745a0fa934c8048a9bc8af0ba9e3afefff82ff
ERROR: jsoncpp/1.9.5: Error in build() method, line 74
        cmake = self._configure_cmake()
while calling '_configure_cmake', line 69
        self._cmake.configure(build_folder=self._build_subfolder)
        ConanException: Error 1 while executing cd '/Users/admin/.conan/data/jsoncpp/1.9.5/_/_/build/0a745a0fa934c8048a9bc8af0ba9e3afefff82ff/build_subfolder' && cmake -G "Unix Makefiles" -DCMAKE_SYSTEM_NAME="Android" -DCMAKE_SYSTEM_VERSION="21" -DCMAKE_ANDROID_ARCH_ABI="arm64-v8a" -DANDROID_ABI="arm64-v8a" -DANDROID_PLATFORM="android-21" -DANDROID_TOOLCHAIN="clang" -DANDROID_STL="libc++" -DCONAN_IN_LOCAL_CACHE="ON" -DCONAN_COMPILER="clang" -DCONAN_COMPILER_VERSION="14" -DCONAN_LIBCXX="libc++" -DBUILD_SHARED_LIBS="OFF" -DCMAKE_INSTALL_PREFIX="/Users/admin/.conan/data/jsoncpp/1.9.5/_/_/package/0a745a0fa934c8048a9bc8af0ba9e3afefff82ff" -DCMAKE_INSTALL_BINDIR="bin" -DCMAKE_INSTALL_SBINDIR="bin" -DCMAKE_INSTALL_LIBEXECDIR="bin" -DCMAKE_INSTALL_LIBDIR="lib" -DCMAKE_INSTALL_INCLUDEDIR="include" -DCMAKE_INSTALL_OLDINCLUDEDIR="include" -DCMAKE_INSTALL_DATAROOTDIR="share" -DCONAN_CMAKE_POSITION_INDEPENDENT_CODE="ON" -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY="ON" -DCONAN_EXPORTED="1" -DJSONCPP_WITH_TESTS="False" -DJSONCPP_WITH_WARNING_AS_ERROR="False" -DJSONCPP_WITH_CMAKE_PACKAGE="False" -DJSONCPP_WITH_STRICT_ISO="False" -DJSONCPP_WITH_PKGCONFIG_SUPPORT="False" -DBUILD_STATIC_LIBS="True" -DJSONCPP_WITH_EXAMPLE="False" -DBUILD_OBJECT_LIBS="False" -Wno-dev '/Users/admin/.conan/data/jsoncpp/1.9.5/_/_/build/0a745a0fa934c8048a9bc8af0ba9e3afefff82ff'
memsharded commented 2 years ago

Hi @nmgwddj

First of all, in this case, I am not a package maker, and some Conan center packages are referenced in my existing CMake project. We will not make this CMake project into a Conan package, we just hope that the current CMake project can be compiled under different platforms (the practice of other platforms is to introduce conan.cmake to automatically perform the conan install process by passing different profiles)

I think the main issue you are finding is that existing packages in ConanCenter are using the old build system integrations, they haven't upgraded yet to the new ones (CMakeDeps and CMakeToolchain), so this functionality of cleanly cross-compiling to Android without any special thing besides specifying the path to the NDK in the conf will not work in those recipes yet.

The good news is that recipes in ConanCenter will need to upgrade to the new build systems integrations, because that is necessary for Conan 2.0, which is already in alpha, will get into beta soon. It will take some time, but this is a priority for the following months.

nmgwddj commented 2 years ago

Looking forward to the transformation of Conan 2.0 and old packages. Before that, I can choose to install locally and specify CONAN_CMAKE_TOOLCHAIN_FILE to temporarily satisfy the cross-compilation.

But it is worth mentioning that the introduction of android_ndk in tool_requires is a very good solution, which does not require CI Agent to actively install NDK to the local environment, and the directory where different developers install NDK may also be inconsistent. Manually specify tools.android:ndk_path The way might be a step backwards, so hopefully you guys will consider this situation.

Thanks again for your patient reply.

memsharded commented 2 years ago

But it is worth mentioning that the introduction of android_ndk in tool_requires is a very good solution, which does not require CI Agent to actively install NDK to the local environment, and the directory where different developers install NDK may also be inconsistent. Manually specify tools.android:ndk_path The way might be a step backwards, so hopefully you guys will consider this situation.

Yes, absolutely, we like this approach. My recommendations above was in order to eliminate moving parts while checking what is working and what not. Once the tooling and integration is there, adding the ndk as a tool_requires is very doable and convenient. It basically needs to define something like a self.conf_info.define("tools.android:ndk_path", self.package_folder) in order to inject its path into the consumers and it will work. But that change in the ConanCenter package won't work until the consumer packages modernize themselves, so it hasn't happened yet. But we will certainly support it.

sbellus commented 1 year ago

My solution how to use conan 2.0 and tool_requires for android-ndk. The trick is to use enviroment variable ANDROID_NDK_ROOT in cmake generated toolchain file. Here is host profile for conan.

[settings]
arch=armv7
arch_build=x86_64
build_type=Release
compiler=clang
compiler.libcxx=c++_shared
compiler.version=12
os=Android
os.api_level=21
[build_requires]
android-ndk/r23
[options]
[env]
CONAN_RUN_TESTS=False
CONAN_CMAKE_GENERATOR=Ninja
[conf]
tools.android:ndk_path=$ENV{ANDROID_NDK_ROOT}
tools.cmake.cmaketoolchain:system_processor=armv7-a
memsharded commented 1 year ago

Hi @sbellus

Thanks for the trick of $ENV{ANDROID_NDK_ROOT}, it might indeed work if you have the env-var defined. Recall also that as Conan profiles in 2.0 are jinja templates, it is also possible:

[conf]
tools.android:ndk_path={{os.getenv("ANDROID_NDK_ROOT")}}

Also, a clarification about the above for other readers: The section [env] and env-vars like CONAN_RUN_TESTS and CONAN_CMAKE_GENERATOR do not work in Conan 2.0, only 1.X.

sbellus commented 1 year ago

Hi @memsharded The ANDROID_NDK_ROOT is set by build requirement android-ndk automatically (in host profile). Does conan parse profile at time when build requirement is already applied? If not the tools.android:ndk_path={{os.getenv("ANDROID_NDK_ROOT")}} will not work.

memsharded commented 1 year ago

The android-ndk could define self.conf_info.define("tools.android:ndk_path", self.package_folder...) or something like that, directly, and then you wouldn't need to define it in profiles or anywhere else.

memsharded commented 1 month ago

I am closing this ticket as resolved, now that we even have a docs example in https://docs.conan.io/2/examples/cross_build/android/ndk.html. Please create new updated tickets if necessary if there is any further question. Thanks!