search

google / prefab

Prefab is a tool for generating build system integrations for prebuilt C/C++ libraries.
https://google.github.io/prefab/
Apache License 2.0
209 stars 32 forks source link

[PKG] SDL2 #41

Closed Zingam closed 1 month ago

Zingam commented 4 years ago

Package name SDL2

Package URL https://www.libsdl.org/index.php

Package source location https://www.libsdl.org/release/ https://www.libsdl.org/release/SDL2-2.0.10.zip

NOTE: vcpkg seems to use this unofficial repository to pull the sources - https://github.com/SDL-mirror/SDL/releases

madebr commented 2 years ago

Hello, I'm currently looking into creating a sdl2 aar. I'm using oboe's prefab_build.sh script as reference.

My goal is to add this build script to the SDL2 repo and, let the SDL2 maintainers push new SDL2 releases to some maven repo.

What I've got so far is this zip: sdl2-2.23.1.zip

I've got a few problems + questions:

DanAlbert commented 2 years ago

Hello, I'm currently looking into creating a sdl2 aar. I'm using oboe's prefab_build.sh script as reference.

My goal is to add this build script to the SDL2 repo and, let the SDL2 maintainers push new SDL2 releases to some maven repo.

🎉

No SDL2::SDL2 is generated (which should be a shared imported target)

The abi.json for the SDL2::SDL2 module (the arm64-v8a one) is the following:

cat prefab/modules/SDL2/libs/android.arm64-v8a/abi.json
{
  "abi": "arm64-v8a",
  "api": 19,
  "ndk": 21,
  "stl": "c++_shared",
  "static": false
}

Most likely the consuming build is not compatible with that. Prefab prints messages that explains why certain modules are omitted from the generated build scripts, but Android Studio will not show you those messages. My guess though is that your externalNativeBuild is using c++_static (the default), which cannot be combined with a shared library that uses c++_shared.

If that's the case, this is all working correctly: prefab is preventing you from using a module that would introduce ODR issues in your app.

When using this in an Android project, source code is missing for the java sources. How can I include java sources such that people can peek into the SDL2's java support code? Right now, Android Studio decompiles the class files.

I don't believe AAR supports source distributions based on https://developer.android.com/studio/projects/android-library#aar-contents. You'd have to file an AGP bug to ask though; I have no idea how the Java side of this works.

madebr commented 2 years ago

Most likely the consuming build is not compatible with that. Prefab prints messages that explains why certain modules are omitted from the generated build scripts, but Android Studio will not show you those messages. My guess though is that your externalNativeBuild is using c++_static (the default), which cannot be combined with a shared library that uses c++_shared.

Yup, that was the case. The error message was burried deep in the messages emitted by gradle.

But I am wondering, SDL2 has a C-only interface (it used c++ for a single android-specific source: hid.cpp. When I would build libSDL2.so with c++_static, could I get away with not adding the "stl" key? Or am I seeing things wrong and does the stl also encompass C things (such as the stdio functions)?

I don't believe AAR supports source distributions based on https://developer.android.com/studio/projects/android-library#aar-contents. You'd have to file an AGP bug to ask though; I have no idea how the Java side of this works.

Thanks, I will open an issue for this.

DanAlbert commented 2 years ago

But I am wondering, SDL2 has a C-only interface (it used c++ for a single android-specific source: hid.cpp. When I would build libSDL2.so with c++_static, could I get away with not adding the "stl" key? Or am I seeing things wrong and does the stl also encompass C things (such as the stdio functions)?

It depends. https://developer.android.com/ndk/guides/middleware-vendors is worth a read.

If you've cautiously hidden all the non-public parts of the library (the only reliable way to do this is with a version script as that page directs, with everything hidden by default and the public interface explicitly exposed), yes, you can set "stl": "none" for the shared library.

If you're also distributing a static library (it's certainly better for users to have both, but that's still on the backlog for even our official ndkports packages), that one must identify the dependency, since the consumer will need to link something. For that case the distinction between c++_shared and c++_static is actually meaningless, since the library hasn't been linked yet. The two are synonymous for static libraries, so just pick one :)

madebr commented 2 years ago

Thanks for the answer!

It depends. https://developer.android.com/ndk/guides/middleware-vendors is worth a read.

If you've cautiously hidden all the non-public parts of the library (the only reliable way to do this is with a version script as that page directs, with everything hidden by default and the public interface explicitly exposed), yes, you can set "stl": "none" for the shared library.

SDL currently uses -fvisibility=hidden + selective __attribute__ ((visibility("default"))). There was discussion about introducing a version script for SDL. bit it's something that will maybe be used in SDL3. From your answer and the docs, I conclude that because hid.cpp does not use any STL feature (only operator new and operator delete are used), "stl" for the shared library can be "none".

If you're also distributing a static library (it's certainly better for users to have both, but that's still on the backlog for even our official ndkports packages), that one must identify the dependency, since the consumer will need to link something. For that case the distinction between c++_shared and c++_static is actually meaningless, since the library hasn't been linked yet.

I plan to provide both in one .aar package, modeled after the CMake targets provided by the the SDL2 package on desktop. There we provide 4 targets: SDL2::SDL2 for the shared library, SDL2::SDL2-static for the static library, SDL2::SDL2main for a static library providing an entry point (=no-op library on Android) and SDL2::SDL2test (static library providing test routines).

The two are synonymous for static libraries, so just pick one :)

Good to know, otherwise the build matrix would be increased for no gain. Can't it be "none" for a static library, like I will do for the shared library?

DanAlbert commented 2 years ago

SDL currently uses -fvisibility=hidden

-fvisibility=hidden is not sufficient because it only affects the code that you compile. Symbols provided by libc++_static.a will still be public. A version script really is the only way to do this safely. If you can't use a version script, libc++_shared.so is your best choice.

new and delete usages are part of this, FWIW. C++ allows replacing those and those behaviors do not work well if more than one library in the app defines public definitions of those (there are other quirks that happen if there are multiple private definitions, but they only appear if your API is bad), so those need to be hidden in libsdl.

I plan to provide both in one .aar package, modeled after the CMake targets provided by the the SDL2 package on desktop. There we provide 4 targets: SDL2::SDL2 for the shared library, SDL2::SDL2-static for the static library, SDL2::SDL2main for a static library providing an entry point (=no-op library on Android) and SDL2::SDL2test (static library providing test routines).

👍

Good to know, otherwise the build matrix would be increased for no gain. Can't it be "none" for a static library, like I will do for the shared library?

No, because the library will have references to libc++ that must be resolved during the user's build. The stl field is less about what you use and more about what you require from consumers. libsdl.a will have unresolved references to libc++ functions that must be provided during the user's build, so if you claim that isn't required with none, and the user also chooses none (maybe they're a C library), the build will fail.

madebr commented 2 years ago

Thanks for the great help!

I went with your advice and now use c++_shared throughout everything. By adding the compiled classes in classes.jar and the sources in classes-sources.jar, Android Studio is able to show the sources when clicking on an SDL class in the gui (this was one of my questions in a previous message).

Before I bring this up to SDL's project leadership, I would like to know more about how publishing works.

I suppose we need to publish a (signed) zip containing a .pom and .aar file to some maven repository. But as a non-Java developer I don't know what maven repo to choose or how this process works. Google supplies some libraries on its own maven repo. Are we supposed to add SDL through this process?

sdl2-2.25.0.zip

DanAlbert commented 2 years ago

The plan sounds right to me. I'm only familiar with Google's (which only googlers can publish to, so it won't work for you), so I'm not 100% sure, but yes, I think maven central is what you want. That's the repository that's available by default in new gradle projects, anyway. It's pretty trivial to use other repositories in gradle, so you could even use GitHub's hosting for example, but I don't know of any reasons to prefer that over Maven Central.

madebr commented 2 years ago

Hello! We're getting there. I got a (first draft) for a build script merged and the project lead is okay with publishing Android packages.

I've got a few questions left. (I'm asking them here because I don't know where else):

Current state: SDL2-2.25.0.zip.

DanAlbert commented 2 years ago

I saw :) Glad to see they were enthusiastic about it.

Unfortunately I don't know the answer to either question. The publishing system I have to use is entirely different.

madebr commented 2 years ago

Hey @DanAlbert ,

I think providing SDL ourselves is a hornets nest. Providing SDL2 is easy, but the satellite libraries (SDL2_image/SDL2_ttf/SDL2_mixer) have 3rd party dependencies. So perhaps it's best to leave packaging SDL2 to others.

Is ndkports open to accepting extra packages? Or does it package only the "bare essential"?

DanAlbert commented 2 years ago

Providing SDL2 is easy, but the satellite libraries (SDL2_image/SDL2_ttf/SDL2_mixer) have 3rd party dependencies.

Is making packages for those not viable?

Is ndkports open to accepting extra packages? Or does it package only the "bare essential"?

We'd love to, but we haven't had the bandwidth to maintain the packages we have to my standards, so not yet.

madebr commented 2 years ago

Is making packages for those not viable?

Well yes it is possible. We do this already in a way: SDL_mixer/SDL_ttf/SDL_image have a possibility to vendor 3rd party libraries instead of using system libraries. It just feels like a lot of work for a single project, which can be useful for many projects.

In the past, I have contributed to ports for conan (conan center) & vcpk, and had a good experience there.

We'd love to, but we haven't had the bandwidth to maintain the packages we have to my standards, so not yet.

Indeed, any such initiative requires commitment which sucks away time and resources from other projects.

DanAlbert commented 2 years ago

Vendoring those third-party libraries works up to a point. It causes conflicts whenever a library appears in the app multiple times. I'm not sure I understand what conan or vcpkg do that makes that not an issue. aiui the only difference is that those dependencies have already been published by someone else.

If vcpkg already has these dependencies solved, there's your answer. vcpkg can product AARs containing prefab packages. If those get published your SDL package can just depend on those.

madebr commented 2 years ago

The idea is still on the table. SDL_image (and others) libraries already contain support to vendor external libraries, so it ended up not being so hard to re-use that. (I was worried a bit that modelling dependencies would become hard)

leleliu008 commented 1 year ago

This package had been published to my GitHub-Hosted Maven Repository: https://github.com/leleliu008/ndk-pkg-prefab-aar-maven-repo

You’re welcome to give it a try.

It was built with android-21 API level.

It provides arm64-v8a armeabi-v7a x86_64 x86 four abis.

It includes both static and shared libraries.

If it doesn't meet your needs, you could try to mannully build it with ndk-pkg, you could use it via GitHub Actions https://github.com/leleliu008/ndk-pkg-package-manually-build
madebr commented 1 year ago

@leleliu008 Without the SDL java classes, a SDL jni-only package is useless.

leleliu008 commented 1 year ago

Uh, I usually do not use java or kotlin write codes. This is for people who use native language only. If you want jni wrapper, I'm sorry, my tool do not do that.

leleliu008 commented 1 year ago

you want a aar contains both jni and prefab ? I think that need a custom processing, my tool is Unified processing for all packages, if do a custom processing, there will be a mount of work to do.

madebr commented 1 year ago

The CMake script of SDL3 does this. It is very simple: just compile the java sources with the classpath of android.jar added.

Our cmake script is also capable to build a apk without gradle. Albeit for a single arch (but that is enough for testing purposes).

leleliu008 commented 1 year ago

wait, if I understand correctly, prefab is nothing to do with jni, it is only for native build, jni is not involved in native build, isn't it?

madebr commented 1 year ago

Prefab is just a way to describe a binary package. Jni is a way to interface between java and native code, so it's only needed when building the SDL binary. SDL provides java sources, which can be found in the android-project subdirectory.

leleliu008 commented 1 year ago

I think your needs should ask for the SDL2 project, not ask for this project. because this project only provides a protocal for native building on Android.

leleliu008 commented 1 year ago

Mybe you guys could write a tool specially designed for managing jni wrappers.

madebr commented 1 year ago

I'm just saying your maven sdl2 packages are incomplete, and not 100% useful for android developers: you need to provide jars alongside the aar. libsdl-org will probably (no promises) do this starting with SDL3 (assuming we can easily automatize the release process).

leleliu008 commented 1 year ago

IMO, jni and prefab should published in different AARs, because they have different purposes. pack all in one would increase the aar file size, the common case is developers only need one of them, if developers really need both of them, that's no problem.

leleliu008 commented 1 year ago

Maybe someone could write a tool called jni-pkg that is used to manage jni wrappers. The first package would be SDL2.

DanAlbert commented 1 year ago

IMO, jni and prefab should published in different AARs, because they have different purposes

I'm not familiar with the details of SDL, but that is the deciding factor here. junit-gtest is an example of an AAR where they should be bundled: the Java and native components are coupled and must be used together.

If there's both a JNI interface to SDL and a native interface to SDL, and an app will use one or the other but not both, yes, separate AARs are the way to go. Prefab is overkill for shipping implementation details; the only thing you need for that is the libs directory in the AAR.

leleliu008 commented 1 year ago

If there's both a JNI interface to SDL and a native interface to SDL, and an app will use one or the other but not both, yes, separate AARs are the way to go. Prefab is overkill for shipping implementation details; the only thing you need for that is the libs directory in the AAR.

Yes, that's what I mean.

A traditional AAR file for jni is classes.jar + libs/<ABI>/lib*.so, now we could use another way to archive that, like what junit-gtest does, classes.jar + prefab/modules/<ABI>/lib*.a

I don't known how common this case is, if it is commonly used, I will take it into account.

madebr commented 2 months ago

It's very likely SDL3 will provide SDL3-x.y.z.aar aar-chives among its release binaries. An example of such an archive can be downloaded from this github workflow.

madebr commented 1 month ago

Update: SDL3 has an "official" SDL3 android archive with prefab support. See https://github.com/libsdl-org/SDL/releases/tag/preview-3.1.3