Roadmap for supporting Rust in apps

DanAlbert commented 2 years ago

Following up on https://github.com/android/ndk/issues/1487, this bug outlines the work we'd need to do to support Rust in apps.

This is not being worked on, nor is it planned. This bug documents the work that needs to be done if that changes. Use the subscribe button on the right side of the page to be notified of updates.

Toolchain support

We need to get Rust to promote Android to Tier 1. Android is currently Tier 2, which means that it is guaranteed to build, but what's built is not guaranteed to work.

[ ] Rust CI for testing android targets. Because all existing Tier 1 targets are able to self-host, Rust's CI likely does not include cross-testing support. We'd either need to enable self-hosting for rustc/cargo/etc on Android, or would need to build that cross-testing support. The former might be easier but the latter more accurately reflects the developer workflow.
[ ] Provide test devices or adequate emulators
[ ] Fix tests that have already regressed
[ ] Commit to maintaining Android support upstream
[ ] Distribute a prebuilt toolchain as a package in the Android SDK (this would not be the only nor even the recommended way to use rust, tier 1 support means you can just go use rustup, but we need SDK packages for AGP/AS integration)

Android APIs

We need to support the maintenance of the ndk-sys crate. This is the Rust equivalent of the NDK's sysroot. It provides definitions for all the platform APIs.

[ ] Auto-generate this crate as part of the Android platform release process

Rust changes

I'm probably wrong about some of the current state here. Will update once Cunningham's Law kicks in.

[ ] rustc needs to be taught about minSdkVersion. All Android code is currently built for the minimum API level supported by rustc.
[ ] Need a way to write things like #if __ANDROID_MIN_SDK_VERSION__ >= 21. aiui Rust's #cfg doesn't support integer inequalities.
[ ] Need to support using APIs via weak symbols when the API is not available in the app's minSdkVersion. This also means diagnostics to protect users from calling weak APIs unguarded.
[ ] Compiler changes for minSdkVersion dependent binary features (gnu hashes, relocation packing, etc)

Build support in AGP

Take this section with a big grain of salt. I do not work on the IDE or AGP. That team might think of things I haven't.

[ ] Integration with AGP

This is the least clear part of the roadmap. There are some existing tools in the rust ecosystem for this. We might be able to reuse those, but there is probably still some integration work required to smooth out that workflow.

This might be a cargo backend for externalNativeBuild, a gradle DSL for rust, or a workflow for integrating rust into android apps that keeps cargo as a separate workflow. Possibly all three.

Steps most likely required regardless of the approach taken:

[ ] Integration with SDK packaged rust toolchains
[ ] Support for using arbitrary toolchains from rustup
[ ] Support for pulling dependencies from crates.io
[ ] Support for publishing AARs
[ ] Cooperation with Android Studio's gradle sync phase (needs to introspect on build configuration without requiring a full build; most build systems are not cooperative with introspection)
[ ] Integration with Java and C++ code
[ ] Support for tools like bindgen
[ ] Build caching

That last one (build caching) has been a major stumbling block for C++, FWIW. The gradle build cache cannot really support anything built by a subordinate build system (I have never met a build system that behaves well in a multi-build system environment), so this likely requires a gradle DSL for Rust, or a separate caching system for Rust.

Android Studio support

CLion already supports Rust, so a lot of this might Just Work, but we need to verify that. Some of these features do overlap with AGP support.

Take this section with a big grain of salt. I do not work on the IDE or AGP. That team might think of things I haven't.

[ ] Basic editing features (make sure all the standard CLion features work for Android targets)
[ ] Android extensions for clion (JNI inspections, for example) probably need to be reworked for rust
[ ] Test support
[ ] Profiling support
[ ] Coverage support
[ ] Code completion and documentation

Ecosystem support

[ ] Identify essential crates, support them so Android is a maintained target. Anything that uses the cc crate or a build.rs is likely to be a fair amount of work
[ ] Need commonly available test infrastructure so that projects can check Android in CI
[ ] Crates that handle file systems, SSL certificates, audio, etc might need porting to work with Android's systems
[ ] Crash handling support in Play (the tombstone will already be captured, but symbolization probably needs work)

Non-goals

Rust in the NDK

This comes up a lot. What "Rust in the NDK" actually means is "clang and rustc should couple their release cycles and come from the same zip file." I don't think anyone actually wants that. The NDK is already a large package, and there's little value in making Rust-only developers download Clang, or C++-only developers download Rust.

There's probably little value to getting rustc from Google in the first place. Once Android is a Tier 1 target, it's probably easier for everyone to just get the toolchain straight from rustup. You'll have access to new releases much quicker that way, and you can use the same process for every platform you support rather than needing to handle Android differently.

There is one edge case where it would be valuable to have Clang and Rust coupled: cross-language LTO. Rust and Clang must use the same version of LLVM for that to work, so to support that we'll likely ship a Google-built rustc that aligns with the NDK's LTS Clang.

A parallel Android Framework for Rust

Just as with C/C++, there will be no "pure" Rust apps. Rust apps will be a zygote process that loads a native library via System.loadLibrary. A large portion of Android itself is written in Java, so any native code in an app must transit JNI to access those parts of the system. Rust apps will have access to the same set of APIs as C/C++ do via the NDK. All APIs not exposed to the NDK must be called via JNI.

jomof commented 2 years ago

A few things to think about:

Do we need to mix Rust and C++ in the same AGP module?
Do we need to consume Prefab AAR from Rust? (If it's even possible)
Do we need to produce Prefab AAR from AGP modules that build Rust? (If that's even possible)
If Prefab isn't possible, do we need and how will we manage module-to-module references?

DanAlbert commented 2 years ago

Do we need to consume Prefab AAR from Rust? (If it's even possible)

I think this is actually quite doable if we're already including bindgen support in the list.

Do we need to produce Prefab AAR from AGP modules that build Rust? (If that's even possible)

I'd be curious to see how this works for any existing rust libraries, if any exist. If Rust libraries with C consumers are a trivial case of manually curated (or even generated) headers plus some shared libraries, Prefab won't have any issue with it, but AGP might struggle to assemble those packages?

If Prefab isn't possible, do we need and how will we manage module-to-module references?

For Rust <-> Rust interfaces, Prefab is probably the wrong answer. Prefab makes sense for C++ where dependencies can have complex usage requirements, and where there are an unbounded number of build systems that need each need custom support. Rust doesn't have those problems aiui, since configuration requirements are generally expressed as code, and the "build system" doesn't need anything more than to know where the rlib is.

CAD97 commented 2 years ago

(hi, just wandering by)

Need a way to write things like #if __ANDROID_MIN_SDK_VERSION__ >= 21. aiui Rust's #cfg doesn't support integer inequalities.

That's correct; #[cfg] currently only supports string equality or set membership.

The normal way you'll see comparison done is a buildscript that sets e.g. has_android_sdk_version_20, has_android_sdk_version_21, has_android_sdk_version_22, etc.

A less known trick, though, is that you can set any given --cfg more than once, and then testing for cfg presence is checking for set membership. For example, #[cfg(target_feature = "")] is a commonly used multivalued configuration key.

Whatever is in charge of providing the cfg keys (a buildscript or the build system) will still need to output a cfg for 1..=set, but then code can use e.g. #[cfg(has_android_sdk_version = "21")] to check if --cfg=has_android_sdk_version="21" is set (and the build system will set it #if __ANDROID_MIN_SDK_VERSION__ >= 21), and this works for any key of multiple, simultaneously.

If you want to get an actual runtime-usable integer, --cfg is not sufficient; the standard way to provide build system information in this way is env! and/or include!.

togetherwithasteria commented 2 years ago

Hiiiiiii <3

In addition to the ndk-rs crates developed by the Rust Windowing team, it would be great to also support the maintenance of the jni-rs library, at the moment they seem to be in need of dedicated maintainers.. See jni-rs/jni-rs#347

togetherwithasteria commented 2 years ago

There are some lovely works that have been done to generate bindings to the Android SDK's JNI interface too!!

The most working one is https://github.com/MaulingMonkey/jni-bindgen, but it depends on its own JNI bindings crate called jni-glue and not jni, and it's not as popular as the jni crate.

The jni crate equivalent to the jni_glue! macro provided by jni-glue crate would be https://github.com/giovanniberti/robusta, which is a JNI integration library for Rust that is uses the jni crate instead!!

With Robusta, I have been working on porting jni-bindgen to use robusta and jni. It's still not usable yet, for now! ^^

I have a busy school schedule so yeah!! The work is here for anyone interested..

togetherwithasteria commented 2 years ago

If anyone would be interested to take over its development, just let me know!! <3

larsbergstrom commented 2 years ago

Here's an example of a spot shared on twitter where coordination between NDK changes and some of the Rust crates should be grown if we do invest in this area: https://github.com/rust-windowing/android-ndk-rs/pull/189

keith commented 2 years ago

That last one (build caching) has been a major stumbling block for C++, FWIW. The gradle build cache cannot really support anything built by a subordinate build system (I have never met a build system that behaves well in a multi-build system environment), so this likely requires a gradle DSL for Rust, or a separate caching system for Rust.

Just want to throw out that at Lyft we are now shipping some rust on iOS and Android, but we're building it with bazel which allows us to easily include rust, C++, java, kotlin, swift, and objective-c all in the same build. I created an example repo for the bazel setup here: https://github.com/keith/bazel-rust-mobile-demo

montekki commented 2 years ago

This might be a cargo backend for externalNativeBuild, a gradle DSL for rust, or a workflow for integrating rust into android apps that keeps cargo as a separate workflow. Possibly all three.

We have been using tools such as mozilla/uniffi-bindgen and mozilla/rust-android-gradle in the latest versions of paritytech/parity-signer. Since Signer app atm has quite a huge codebase in Rust for an Android application it is probably worth looking at as an example Rust app on Android.

vasishath commented 2 years ago

This is not specific to rust but is the team considering something like what GraalVM does for native code, i.e compiling to LLVM bytecode and then linking to java, eliminating JNI entirely?

DanAlbert commented 2 years ago

Please raise that on our discussion forum. afaik it has nothing to do with rust support.

atsushieno commented 1 year ago

I am not really a Rust app developer but curious - is Tier 1 support particularly required for NDK uses? Since AOSP already incorporates Rust, it seems to me that this kind of requirement must have already met for overall Android ecosystem, or there is different set of requirements particularly for NDK.

DanAlbert commented 1 year ago

AOSP can't test all the things that rustc will ever compile. If there's a bug in the compiler, we can change the code to work around it. That's not the case for the NDK. We need to be reasonably confident that the rustc we ship for apps will work for anything an app might throw at it, and we need to be ready to fix rather than work around compiler bugs. It's of course impossible to be 100% bug free, but we need to do at least the due diligence of reaching tier 1 support, which is defined as "we ran tests, and we'll fix it if it breaks".

If you're an app developer that is willing to accept the risks of a less verified compiler, rustup can get you that toolchain already. This bug is about what we need to do before we're willing to call this "supported".

CAD97 commented 1 year ago

Need to support using APIs via weak symbols when the API is not available in the app's minSdkVersion. This also means diagnostics to protect users from calling weak APIs unguarded.

Just for reference, here's std's functionality for weak! symbols, roughly reproduced here (may include nightly-only functionality):

// for ELF targets only
macro_rules! weak {
    (fn $name:ident($($t:ty),*) -> $ret:ty) => (
        let ref $name: ExternWeak<unsafe extern "C" fn($($t),*) -> $ret> = {
            extern "C" {
                #[linkage = "extern_weak"]
                static $name: Option<unsafe extern "C" fn($($t),*) -> $ret>;
            }
            #[allow(unused_unsafe)]
            ExternWeak::new(unsafe { $name })
        };
    )
}

struct ExternWeak<F: Copy> {
    weak_ptr: Option<F>,
}

impl<F: Copy> ExternWeak<F> {
    #[inline]
    pub(crate) fn new(weak_ptr: Option<F>) -> Self {
        ExternWeak { weak_ptr }
    }

    #[inline]
    pub(crate) fn get(&self) -> Option<F> {
        self.weak_ptr
    }
}

A very recent change now allows directly using Option<fn()> with #[linkage = "extern_weak"] rather than just *mut c_void. Once the bootstrap cycle is through, I believe this will result in it being feasible to just use the static directly rather than requiring the ExternWeak wrapper.

There's no nice solution for upgrading to a normal (extern) function when the minimum API version is high enough to guarantee presence of the weak symbol (perhaps have fn symbol and static weak_symbol, so higher API requirements get the nice API), but this does offer a reasonably misuse-resistant solution for exposing weak function symbols.

DanAlbert commented 1 year ago

A https://github.com/rust-lang/rust/pull/104799 https://github.com/rust-lang/compiler-team/issues/565 now allows directly using Option<fn()> with #[linkage = "extern_weak"] rather than just *mut c_void. Once the bootstrap cycle is through, I believe this will result in it being feasible to just use the static directly rather than requiring the ExternWeak wrapper.

There's no nice solution for upgrading to a normal (extern) function when the minimum API version is high enough to guarantee presence of the weak symbol (perhaps have fn symbol and static weak_symbol, so higher API requirements get the nice API), but this does offer a reasonably misuse-resistant solution for exposing weak function symbols.

Given that Android Rust support is still well in the future, the newness of that feature won't be an issue. On the surface that looks like it's probably enough to check that box. Thanks for the heads up.

Agreed that it's sufficiently misuse-resistant. It's Option, so you can't unintentionally call the missing function, and it's not terribly annoying to deal with the guaranteed-available case anyway. I suppose some sort of #[availability = "android=30"] or whatever could be added, but it might be overkill (though potentially very useful for diagnostics!)

xanahopper commented 1 year ago

About cfg with number, I think something like autocfg may works? A prefab buildscript (build.rs) to provide all config features like android_16, android_17...

complexspaces commented 7 months ago

Crates that handle file systems, SSL certificates, audio, etc might need porting to work with Android's systems

rustls-platform-verifier came out earlier this year and handles verifying SSL certificates on the Android platform 👍

android / ndk