Android rules can't use external repo for toolchain

[AustinSchuh]

Currently, we require developers to install a package with the SDK and NDK. This makes upgrades almost impossible to version and track.

I want to be able to tar up the SDK and NDK and point Bazel to them. An external repo would be awesome and bazely. A URL that the rules fetch from and unpack would also work.

It looks like we might be able to check the toolchains into our repo and it would work that way. That would blow the size of our repo up in a way that's not scalable long term.

[aj-michael]

Hi Austin, just to clarify, your feature request is to allow android_sdk_repository and android_ndk_repository to point to other repositories that "fetch" the SDK/NDK like http_archive?

If so, I think this may be a specific case of a more general feature request which is "the ability to compose repository functions". android_{s,n}dk_repository are currently repository functions that take as input the path to a local directory. If they accepted a label to a filegroup, then we could compose them with new_http_archive, but at the moment I don't think it's possible.

/cc @ahumesky @jin @dkelmer

[aj-michael]

To expand a little, there is currently no way to express the order of repository function evaluation in Bazel. Repository functions are executed the first time that Bazel loading phase notices something in the main repository that references the given repository function.

E.g., when building an android_binary, there is an implicit dependency on @androidsdk//:sdk which provides tools like aapt and dx. So bazel recognizes that android_sdk_repository creates @androidsdk//... so it runs it. When android_sdk_repository runs, it scans the directory containing the Android SDK and creates targets based on the files available.

What we would need for your feature request is a mechanism to express "before running android_sdk_repository, run this new_http_archive runs which creates the directory that android_sdk_repository will act on".

Today, you could try something like this (warning: don't do this):

# WORKSPACE
http_archive(
    name = 'downloads_android_sdk',
    url = '...',
)
android_sdk_repository(
    name = 'androidsdk',
    path = __workspace_dir__ + './bazel-out/downloads_android_sdk',
)

Then if you ran the following sequence of commands, I think it would work:

$ bazel build @downloads_android_sdk//...
$ bazel build @androidsdk//...

However, if you just ran bazel build @androidsdk//..., it would fail, because the http_archive would not have been run.

[AustinSchuh]

Hi Michael,

The implementation doesn't matter to me (hence me not being super specific on purpose to not over-constrain your solution space). My use case is that I can't currently control which SDK/NDK is used through the current mechanisms. I can't check it into our repo (which I bet is the Google solution), and controlling the host version is problematic long term.

Your proposed solution (assuming it doesn't have problems with remote execution and the race is fixed) is fine. Accepting 'http://example.com/sdk.tar.xz' as a "path" would also work for us. I'd love to be able to point the NDK at '@android_ndk//:ndk_filegroup', but I'm really not picky.

[kamalmarhubi]

I was just writing up a feature request for this and decide to search issues before posting. I was working towards a solution like the "don't do this" version above. I think the jankiness might be acceptable for me right now, so I'll give it a shot.

[kamalmarhubi]

Was just thinking of another alternative: a list of sdkmanager packges to install. So something like:

android_sdk_repository(
    name = "androidsdk",
    packages = [
        "build-tools;27.0.2",
        "system-images;android-27;google_apis_playstore;x86",
        # ...
    ],
)

It could have a repositories.cfg override for customizing locations / providing local addons.

[kamalmarhubi]

Looking into it a tiny bit, this seems fairly possible. The Android sdklib looks pretty nicely broken up, and includes a Downloader interface. This should make it possible to wrap bazel's HttpDownloader to get the repository caching niceness. @AustinSchuh @aj-michael thoughts?

[AustinSchuh]

@kamalmarhubi That all sounds good to me. My big one is that our safety guys really like to know exactly which version of the tools are being used, so we want to be able to verify that the toolchains are correct and haven't changed compared to what they should be. It's more annoying to do that with more "automagic" involved, but not impossible. We also need to re-host our dependencies so we are in control of the lifetime of the compilers.

That being said, android isn't part of the safety story so we can be more lax there, but it's good food for thought.

[ahumesky]

For what it's worth, it's entirely possible to tar up the Android SDK and retrieve it using an http_archive, and use that in your build. You can set that SDK as the one to use using a bind() in the workspace file or by setting --android_sdk in your tool/bazel.rc file. The downside is that you'd have to manually write a BUILD file for it, which today the repository rules generate. Namely this would be an android_sdk rule for the SDK and a CROSSTOOL for the NDK. You could run the repository rules once to get the BUILD files and base your own BUILD files on those. Another idea is to provide a url or similar attribute to android_sdk_repository/android_ndk_repository telling it to fetch the sdk/ndk from that location, but it's unlikely we'll have time to work on such a feature in the near future.

[AustinSchuh]

@ahumesky Works for me! I didn't know that was possible. We'll switch over to doing that.

[ahumesky]

Great, let us know if you run into any trouble wiring it up.

[kamalmarhubi]

@AustinSchuh

My big one is that our safety guys really like to know exactly which version of the tools are being used, so we want to be able to verify that the toolchains are correct and haven't changed compared to what they should be. That is very fair. The package metadata includes checksums, which would presumably be validate by sdklib though I'd want to check. If you supplied your own trusted copy of the that could work.

But for the easiest implementation, sounds like a urls property would be good. Will take a look to see how hard this is. I really want to reduce our manual-steps-to-first-build.

[jin]

I stumbled upon https://github.com/quittle/bazel_android_sdk_downloader - may be useful for some of you.

[aj-michael]

@jin note that that solution has the same problems as the http_archive solution I posted in November.

See https://github.com/quittle/bazel_android_sdk_downloader/blob/master/rules.bzl#L90:

def android_sdk_repository(name = None, workspace_name = None, api_level = None, build_tools_version = None):
    # Support downloading the SDK as a repository (inspired by `@yarn//yarn` )
    _android_sdk_repository(
        name = name,
        api_level = api_level,
        build_tools_version = build_tools_version,
    )

    # Create an android_sdk_repository targetting the downloaded repo
    # The path is long and convoluted because the SDK is downloaded into the runfiles of a bin target
    native.android_sdk_repository(
        name = "android_sdk_repository_" + name,
        path = "bazel-bin/external/{name}/{install_target_name}/{install_output_name}.runfiles/{workspace_name}/{sdk_root_path}".format(
                name = name,
                install_target_name = _INSTALL_TARGET_NAME,
                install_output_name = _INSTALL_OUTPUT_NAME,
                workspace_name = workspace_name,
                sdk_root_path = _SDK_ROOT_PATH),
        api_level = api_level,
        build_tools_version = build_tools_version,
    )

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 3 years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[gregoryT5]

@bazelbuild/triage this issue is still relevant, please reopen