[FR] Bionic support for module maps

[finagolfin]

I recently tried building the llbuild C++ project on Android in the Termux app with the Swift package manager, which adds -fmodules by default. While compiling llbuild like this works on linux with glibc and macOS, it doesn't work on Android (with errors like the size_t one explained in the clang docs) or Windows, so I submitted a pull to remove that flag on Android for now, apple/swift-package-manager@00c7768a.

Are there any plans to get the Bionic headers working with that flag? If so, maybe this can be the tracking issue till it's in.

[DanAlbert]

It's not something we currently have planned. We need to take another look at how the various modules implementations finally settled and see if this is something worth doing now.

[finagolfin]

Any thoughts about supporting this now? It would probably help with interoperability issues like apple/swift#60272.

[DanAlbert]

I don't think it would help you. From that bug:

/data/data/com.termux/files/usr/include/bits/pthread_types.h:104:14: note: declaration here is not visible. Why isn't it visible? Who knows.

Because it's a private header. If we were to make an explicit module map it would be even more hidden.

[enh-google]

well, (assuming their copy of that file looks like ours), pthread_t would be in the module.

i suspect it's just that -fmodules + no module definitions => nothing visible, so the fix here is to stop the swift build system from suppling that flag when targeting Android?

[finagolfin]

Because it's a private header. If we were to make an explicit module map it would be even more hidden.

I know very little about C/C++ modules, but wouldn't that make them useless if common types like pthread_t are not accessible that way?

i suspect it's just that -fmodules + no module definitions => nothing visible, so the fix here is to stop the swift build system from suppling that flag when targeting Android?

I think that's not the problem here, as the Swift compiler has a module map for Bionic, though it is simply named SwiftGlibc for now and reused on the BSDs and Android. These are all the libc headers it includes. It works well for importing Bionic declarations into Swift, but this particular recent usage tries to import LLVM declarations using the new C++ interoperability and falls over with this error.

Note that I originally opened this request issue for a different scenario: the Swift package manager supports building mixed C/C++/Swift packages, only it applied -fmodules when building C++ files with clang++ and that would fail because Bionic doesn't have a built-in module map that clang++ could detect and use. Whereas this recent issue is when the Swift compiler itself tries to import declarations from LLVM along with its own Bionic module map, which I guessed might benefit from an official Bionic modulemap instead but is just a stab in the dark.

[finagolfin]

More than four years later, some Swift devs are working on a properly modularized Bionic modulemap, apple/swift#72161. Is that something worth upstreaming to the NDK?

[enh-google]

seems like they're lacking an answer to the "what about API levels?" question. (well, beyond "we pick one and pretend that one size fits all", which isn't an answer we can use :-) )

[finagolfin]

Is that something that needs to be addressed by the Bionic modulemap?

In their case, they build the Swift stdlib against a base API level, by specifying the target triple as aarch64-unknown-linux-android21, then you can build user code against any higher level by specifying the triple for such Swift packages as aarch64-unknown-linux-android29, which passes in the right Android API to the Bionic headers and Swift modules. Unless the compiled Swift stdlib for API 21 relies on some old behavior that is somehow incompatible with API 29, I think that should work.

In your case, since you supply the relevant C libraries compiled against all supported API levels in the NDK, I don't see how the API level will matter in the Bionic modulemap, other than maybe newly added headers.

[DanAlbert]

We're still talking about the non-standard variant of C++ modules, right? afaict moduelmaps aren't a part of the thing that was standardized. https://en.cppreference.com/w/cpp/language/modules does not use the word "map" once, and doesn't appear to have any file that looks similar to the one you linked.

[finagolfin]

I haven't dug down to the level of the modules standard. I assume modulemaps are needed in practice because everybody is using them. I think the page you linked doesn't mention them because it's aimed at C++ users, whereas the libc modulemap is an issue for toolchain implementers.

[finagolfin]

Looking into it more, you may be right that modulemaps are an LLVM-specific feature. Since the NDK uses clang, I'll leave it up to you if you'd want to add such a Bionic modulemap.

[DanAlbert]

No, that's exactly why we haven't done anything about this yet. The ecosystem is fragmented and we're waiting for it to settle before going down what may be the wrong path and not being able to go back. We were told at one point that LLVM and the standards committee were working to find a compromise that didn't result in two competing standards. If that ever happened, I've seen no information about it. That bionic doesn't have one tells you that AOSP hasn't found this worth investigating yet either. @pirama-arumuga-nainar is probably more up to date on that than I am.

"Everybody is using Clang modules" is the opposite of the anecdotes I've heard. Aside from internal google stuff, you're the only person that I've heard try (that thread you linked adds fb to the mix, but given that their infra was heavily inspired by google's that's not surprising). OTOH I've seen quite a few bug reports where the repro case used C++ modules (it's possible they've all been from the same person, idr).

[finagolfin]

What I meant is that everybody I see using modules is using maps, but then I noted these maps might be specific to LLVM.

You know much better than me how much demand there is from NDK users, so if you want to wait for standards finalization, I understand.

[DanAlbert]

That's our plan for now. We used to say that on our roadmap, but we added that ten years ago and nothing's improved, so it was removed.

We're well aware that build speed is pretty much the number 1 metric for users, and that modules (some form of them) will help with that, but we need to make sure we pick the one that won't become a maintenance nightmare.

[DanAlbert]

so if you want to wait for standards finalization

Mild pedantry: modules have been standardized, just not this kind. The kind described on the cppreference page above is the standardized form of modules. That's the thing we're more likely to do. If modulemaps become standardized as well, that'd change things for us.

[pirama-arumuga-nainar]

libc++ is adding support for "standard C++ modules" - see here. But it is still very experimental. Beyond that, I don't have much insight about which module type is better for bionic - esp with regards to @enh-google's question about Android API levels.