Improve Android package management for the NDK

[DanAlbert]

This bug tracks the Easier access to common open-source libraries item on our roadmap.

There are many other commonly-used libraries (such as Curl and BoringSSL) that are currently difficult to build/package, let alone keep updated. We should offer (a) a tool to build open source projects, (b) a repository of prebuilts, (c) a command-line tool to add prebuilts to an ndk-build/cmake project, and (d) Studio integration to add prebuilts via a GUI.

We currently have cdep, but it doesn't have a very large corpus and could use some build system integration polish. If not cdep, adding support for exposing native libraries from AARs would be an alternative.

Some questions about this came up on another bug, specifically in what we were planning to use and why, so I wanted to publish the requirements that we had for this, show what we'd considered, and explain the current plan.

Note that this isn't the full design document. I've spared the nitty gritty details since I don't think they're particularly interesting at this time, but if anyone has questions just ask and I can elaborate :)

Requirements

1. Works on Linux, Darwin, and Windows build hosts.
2. Supports arbitrary build systems.
3. Able to deliver projects employing a variety of architectures:
   • Header only libraries.
   • Single library projects.
   • Multi library packages.
4. Able to describe complex library use:
   • Export include paths to library users.
   • Export cflags and ldflags to library users.
   • Complex include/link ordering requirements such as those found the C++ standard library.
5. Supports binary distributions.
6. Able to fetch from arbitrary sources.
7. Support IDE integration.
8. Supports cross package dependencies.
9. Support for non-Android targets.
10. Support for multi-language projects.

Options

With these requirements, none of the existing options we know of will work as-is. We considered cdep, AAR (Android packages via Maven), vcpkg, and hunter. cdep is C/C++ only, which means packages like GTestJNI can't be distributed with it. AAR doesn't support C/C++ well (it supports C/C++ implementation, but the interface to the library must be Java), which obviously won't work for the NDK. vcpkg doesn't seem to support more complex build system features such as exporting required flags to dependents. Hunter is CMake only.

Of these options, AAR seems like our best bet. It doesn't meet the single most obvious need (being able to actually expose a C++ interface), but unlike the other options it's the only one that could be reasonably modified to fit these requirements. It also is the extant method for distributing Java dependencies for Android, and reusing existing infrastructure is a big plus.

Design

The plan is to build some low level tooling (which I'm currently calling Prefab) for generating build system integration for prebuilt packages and some metadata describing usage requirements and then extending gradle and AAR to make use of that.

When I say low level, what I mean is that Prefab only solves the build system integration part of the problem. For an Android gradle project, gradle is still responsible for resolving and fetching dependencies. When gradle fetches an AAR that contains Prefab artifacts, it extracts those to a directory and calls prefab to have it generate build system integration. The generated output is then passed to ndk-build or CMake to be used as you would any other dependency. This means that if your organization uses some other mechanism for distributing dependencies, Prefab artifacts can be distributed that way as well (including something as simple as a git submodule).

For example, your build.gradle would contain something like the following:

dependencies {
    implementation 'com.google.gtest:gtest:1.0'
    implementation 'com.google.gtestjni:gtestjni:1.0'
}

and your CMakeLists.txt would contain:

find_package(gtest 1.0 REQUIRED MODULE)
find_package(gtestjni 1.0 REQUIRED MODULE)

# Additional targets for building the actual app library...

add_library(apptest
  SHARED
    src/test/cpp/app_test.cpp
)

target_link_libraries(apptest
  PRIVATE
    app
    gtest::gtest
    gtestjni::gtestjni
)

And that's it. Gradle will fetch the dependencies, call Prefab to generate the CMake package definitions, and find_package will find the dependencies.

Build system integration

Build system integration is provided by plugins for Prefab, and we'll include CMake and ndk-build plugins by default. If you're using a custom build system, you can provide a path to a plugin jar on the command line to teach Prefab how to generate what you need.

Non-Android targets

Support for non-Android targets will be simple to add (basically just a class recognizing and perhaps disambiguating libraries for the platform, such as Android needs to do with multiple ABIs, minSdkVersion targets, etc). We're only planning on implementing support for Android, but patches adding other platforms are welcome.

So, that's the rough outline. If I've missed any major requirements that make this approach not viable for your project, please speak up. We're still in the very early phases of this so it's not too late for large changes.

[enh-google]

(nice! here's a direct link to alexcohn's slides for that video: https://corecppil.github.io/Meetups/2020-03-37_Core_C++_in_Cyberspace_Online/Package_Management_for_Android_C++_Alex.pdf )