Build instructions still valid?

[IzzySoft]

Build instructions in the Readme state:

Set JAVA_HOME environment variable to JDK 11.

Is this still correct? Because doing so, trying to build v1.27 I ended up with:

Execution failed for task ':buildSrc:generateExternalPluginSpecBuilders'.
> Could not resolve all files for configuration ':buildSrc:compileClasspath'.
   > Could not resolve org.lsposed.lsparanoid:gradle-plugin:0.6.0.
     Required by:
         project :buildSrc
      > Dependency requires at least JVM runtime version 17. This build uses a Java 11 JVM.

Further, I tried to build only the SelfFoss variant, but assembleSelfFossRelease failed saying it could not be found.

So reverting to exactly what the Readme says, just omitting the JDK-11 part, I get

> Task :app:buildNativeFoss FAILED
+ make cap_names.h
./build_native.sh: 64: make: not found

Most folks might have build-essential installed – I tried building with a minimal image in Podman, as all the RBs at IzzyOnDroid are built that way. So maybe while on it, maybe also include the requirements (spotted so far: make, gcc)? There must be more, because at this point I'm stuck with:

+ make cap_names.h
_makenames.c:8:10: fatal error: stdio.h: No such file or directory
    8 | #include <stdio.h>
      |          ^~~~~~~~~
compilation terminated.
make: *** [Makefile:76: _makenames] Error 1

In a last try, I've simply added build-essential – but still the build fails, this time with

./build_native.sh: 79: /opt/sdk/ndk/25.1.8937393/toolchains/llvm/prebuilt/linux-x86_64/bin/armv7a-linux-androideabi24-clang: not found

I also could not figure how to specify the variant to build (here: selfFoss), might be good to point that out as well – no need to build them all, then only pick one and throw the others away :wink:

[mirfatif]

Yes it needs JDK 17, make, and NDK. Set the NDK version here. Run ./gradlew assembleFdroidFossRelease.

[IzzySoft]

That's what I installed, but still get the above error. OK, I ran it with ./gradlew :app:assembleRelease -Pself -Pfoss, thinking like with MyLocation I'll need the selfFoss, not the fdroidFoss release. Well… let me specify the NDK explicitly then… ugh. That builds. And despite "selfFoss" being specified it builds the "fdroidFoss" one. Hm. I don't mind if that matches the APK at your releases. Which it does not:

       ATTR: http://schemas.android.com/apk/res/android:versionCode=127
-      ATTR: http://schemas.android.com/apk/res/android:versionName='v1.27'
+      ATTR: http://schemas.android.com/apk/res/android:versionName='v1.27-fd'
       ATTR: http://schemas.android.com/apk/res/android:compileSdkVersion=34

And that's not the only difference:

  -rw-r--r--  0.0 unx      120 b-      118 defN 1981-01-01 01:01:02 8647e885 META-INF/version-control-info.textproto
- -rw-r--r--  0.0 unx     1918 b-     1918 stor 1981-01-01 01:01:02 5224bb4c assets/dexopt/baseline.prof
+ -rw-r--r--  0.0 unx     1918 b-     1918 stor 1981-01-01 01:01:02 90788a2e assets/dexopt/baseline.prof
  -rw-r--r--  0.0 unx      187 b-      187 stor 1981-01-01 01:01:02 31ab17c2 assets/dexopt/baseline.profm
- -rw-r--r--  0.0 unx  1794232 b-   887805 defN 1981-01-01 01:01:02 8c827d71 classes.dex
+ -rw-r--r--  0.0 unx  1794364 b-   887894 defN 1981-01-01 01:01:02 bc505d3d classes.dex
  -rw-r--r--  0.0 unx   168352 b-    69680 defN 1981-01-01 01:01:02 393a229f classes2.dex
- -rw-r--r--  0.0 unx     7512 b-     2882 defN 1981-01-01 01:01:02 0093ac4f lib/arm64-v8a/libpmxd.so
+ -rw-r--r--  0.0 unx     7800 b-     2953 defN 1981-01-01 01:01:02 b355030d lib/arm64-v8a/libpmxd.so
  -rw-r--r--  0.0 unx    27080 b-    11274 defN 1981-01-01 01:01:02 af16dfb5 lib/arm64-v8a/libpmxe.so
  -rw-r--r--  0.0 unx    30224 b-    19307 defN 1981-01-01 01:01:02 e9f9b14e lib/arm64-v8a/libspake2.so
- -rw-r--r--  0.0 unx     5388 b-     2363 defN 1981-01-01 01:01:02 ee21066c lib/armeabi-v7a/libpmxd.so
+ -rw-r--r--  0.0 unx     5644 b-     2434 defN 1981-01-01 01:01:02 a42498a1 lib/armeabi-v7a/libpmxd.so
  -rw-r--r--  0.0 unx    22056 b-    10077 defN 1981-01-01 01:01:02 33d7e1d7 lib/armeabi-v7a/libpmxe.so
  -rw-r--r--  0.0 unx    25864 b-    17618 defN 1981-01-01 01:01:02 9b326212 lib/armeabi-v7a/libspake2.so
- -rw-r--r--  0.0 unx     6300 b-     2888 defN 1981-01-01 01:01:02 2b743280 lib/x86/libpmxd.so
+ -rw-r--r--  0.0 unx     6556 b-     2962 defN 1981-01-01 01:01:02 f2306e2d lib/x86/libpmxd.so
  -rw-r--r--  0.0 unx    24500 b-    12082 defN 1981-01-01 01:01:02 7f39c999 lib/x86/libpmxe.so
  -rw-r--r--  0.0 unx    46292 b-    24638 defN 1981-01-01 01:01:02 8af8c277 lib/x86/libspake2.so
- -rw-r--r--  0.0 unx     7272 b-     2790 defN 1981-01-01 01:01:02 73377181 lib/x86_64/libpmxd.so
+ -rw-r--r--  0.0 unx     7560 b-     2865 defN 1981-01-01 01:01:02 f69c9698 lib/x86_64/libpmxd.so
  -rw-r--r--  0.0 unx    27480 b-    12104 defN 1981-01-01 01:01:02 6fbd0049 lib/x86_64/libpmxe.so
  -rw-r--r--  0.0 unx    38680 b-    21111 defN 1981-01-01 01:01:02 f738153f lib/x86_64/libspake2.so
- -rw-r--r--  0.0 unx    23284 b-    23173 defN 1981-01-01 01:01:02 b1395f8d assets/daemon.dx
+ -rw-r--r--  0.0 unx    23284 b-    23173 defN 1981-01-01 01:01:02 7c2de250 assets/daemon.dx
  -rw----     2.0 fat     1738 b-      782 defN 1981-01-01 01:01:02 7cded4df DebugProbesKt.bin
  -rw----     2.0 fat        6 b-        8 defN 1981-01-01 01:01:02 38ca8087 META-INF/androidx.activity_activity-ktx.version
  -rw----     2.0 fat        6 b-        8 defN 1981-01-01 01:01:02 38ca8087 META-INF/androidx.activity_activity.version
@@ -93,7 +93,7 @@
  -rw----     2.0 fat     2426 b-     1286 defN 1981-01-01 01:01:02 7e3e3d18 kotlin/reflect/reflect.kotlin_builtins
  -rw----     2.0 fat    42868 b-     6038 defN 1981-01-01 01:01:02 71c94b37 org/bouncycastle/x509/CertPathReviewerMessages.properties
  -rw----     2.0 fat    49608 b-     6732 defN 1981-01-01 01:01:02 8357ba7f org/bouncycastle/x509/CertPathReviewerMessages_de.properties
- -rw----     0.0 fat    10968 b-     2762 defN 1981-01-01 01:01:02 97a59635 AndroidManifest.xml
+ -rw----     0.0 fat    10976 b-     2767 defN 1981-01-01 01:01:02 020b588c AndroidManifest.xml
  -rw----     0.0 fat      352 b-      208 defN 1981-01-01 01:01:02 df4f27a5 res/-5.xml

The .so files are not RB, and classes.dex neither. Full diffs (incl. diffoscope): diff.zip

The versionSuffix was one reason I need the selfFoss instead of the fdroidFoss build. How so you build the APK you attach to releases (the non-pro one)? My full recipe:

---
repository: https://github.com/mirfatif/PermissionManagerX.git
updates: releases
versions:
  - tag: v1.27
    apks:
      - apk_pattern: PMX_v[0-9.]+\.apk
        apk_url: https://github.com/mirfatif/PermissionManagerX/releases/download/v1.27/PMX_v1.27.apk
        build:
          - echo "sdk.dir=/opt/sdk" >> local.properties
          - chmod +x gradlew
          - ./gradlew :app:assembleRelease -Pself -Pfoss
          - find . -name '*.apk'
          - mv app/build/outputs/apk/fdroidFoss/release/app-fdroid-foss-release-unsigned.apk /outputs/unsigned.apk
        build_cpus:
        build_home_dir: /build
        build_repo_dir: /build/repo
        build_timeout:
        build_user: build
        provisioning:
          android_home: /opt/sdk
          build_tools:
          cmake:
          cmdline_tools:
            version: '12.0'
            url: https://dl.google.com/android/repository/commandlinetools-linux-11076708_latest.zip
            sha256: 2d2d50857e4eb553af5a6dc3ad507a17adf43d115264b1afc116f95c92e5e258
          extra_packages:
            - make
            - gcc
            - libc-dev
          image: debian:bookworm-slim
          jdk: openjdk-17-jdk-headless
          ndk: 25.1.8937393
          platform:
          platform_tools:
          tools:
          verify_gradle_wrapper: true

Btw, there are 2 APKs, maybe I should take the other one? The name does not really suggest that:

+ find . -name '*.apk'
./app/build/outputs/apk/fdroidFoss/release/app-fdroid-foss-release-unsigned.apk
./priv_daemon/build/outputs/apk/foss/release/priv_daemon-foss-release-unsigned.apk

[mirfatif]

diff --git a/app/build.gradle b/app/build.gradle
index 1d29614..9c13ee9 100644
--- a/app/build.gradle
+++ b/app/build.gradle
@@ -51,7 +51,7 @@ android {
     variantFilter { variant ->
         List names = variant.flavors*.name
         if (noPro) {
-            if (!names.contains('fdroid') || !names.contains('foss')) {
+            if (!names.contains('self') || !names.contains('foss')) {
                 variant.setIgnore(true)
             }
         } else if (names.contains('ps') && names.contains('foss')) {
@@ -306,7 +306,7 @@ if (!noPro) {
 }

 afterEvaluate {
-    setExtSrcDependencies('Fdroid', 'Foss')
+    setExtSrcDependencies('Self', 'Foss')

     if (!noPro) {
         setExtSrcDependencies('Self', 'Foss')

Run ./gradlew assembleSelfFossRelease.

[IzzySoft]

That was the first thing I tried, and failed – but well, maybe that was also solved by explicitly specifying the NDK version, let me try again… and get the same error again:

FAILURE: Build failed with an exception.
15 actionable tasks: 15 executed

* What went wrong:
Task 'assembleSelfFossRelease' not found in root project 'repo' and its subprojects.

and before you ask: no, ./gradlew :app:assembleSelfFossRelease doesn't work either:

FAILURE: Build failed with an exception.

* What went wrong:
Cannot locate tasks that match ':app:assembleSelfFossRelease' as task 'assembleSelfFossRelease' not found in project ':app'.

Listing up available tasks (./gradlew tasks) shows nothing with "Self":

Build tasks
-----------
assemble - Assembles the outputs of this project.
assembleAndroidTest - Assembles all the Test applications.
assembleDebug - Assembles main outputs for all Debug variants.
assembleFdroid - Assembles main outputs for all Fdroid variants.
assembleFdroidFoss - Assembles main outputs for all FdroidFoss variants.
assembleFoss - Assembles main outputs for all Foss variants.
assemblePro - Assembles main outputs for all Pro variants.
assembleRelease - Assembles main outputs for all Release variants.
assembleUnitTest - Assembles all the unit test applications.
build - Assembles and tests this project.
buildDependents - Assembles and tests this project and all projects that depend on it.
buildKotlinToolingMetadata - Build metadata json file containing information about the used Kotlin tooling
buildNeeded - Assembles and tests this project and all projects it depends on.
bundle - Assemble bundles for all the variants.
bundleDebug - Assembles bundles for all Debug variants.
bundleFdroid - Assembles bundles for all Fdroid variants.
bundleFdroidFoss - Assembles bundles for all FdroidFoss variants.
bundleFoss - Assembles bundles for all Foss variants.
bundlePro - Assembles bundles for all Pro variants.
bundleRelease - Assembles bundles for all Release variants.
clean - Deletes the build directory.
…

Nothing with "self" there. That only leaves assembleFoss (as I need an APK, not a bundle). And funnily builds a signle APK only: priv_daemon/build/outputs/apk/foss/release/priv_daemon-foss-release-unsigned.apk (I've already seen that when building the Fdroid variant but ignored it as "daemon" sounded not like what I was looking for). And well, I don't even need to run a diff to see it's not the one I'm after: it's just 35 kB instead of the expected 2.6 MB.

There must be something we both miss, huh? :man_shrugging:

According to foss-pro-flavors.gradle, there are only 2 flavors defined: foss and pro. Only one dimension (version). I see no "self" nor "fdroid" there. A dimension type is defined in build.gradle. I'm no Android dev, but I guess the 0 means it's inserted before any existing dimension? Then selfFoss should be valid. Still, gradle claims not to know of any matching task…

[mirfatif]

Did you apply the diff patch I provided?

[IzzySoft]

Oof, I mistook it for a quote. Will try again once the "daily run" is through here. Thanks for the pointer!

[IzzySoft]

OK, this builds:

build:
  - echo "sdk.dir=/opt/sdk" >> local.properties
  - sed -r "s/!names.contains\('fdroid'/!names.contains('self'/ ; s/setExtSrcDependencies\('Fdroid', 'Foss'\)/setExtSrcDependencies('Self', 'Foss')/" -i app/build.gradle
  - chmod +x gradlew
  - ./gradlew assembleSelfFossRelease
  - find . -name '*.apk'
  - mv app/build/outputs/apk/selfFoss/release/app-self-foss-release-unsigned.apk /outputs/unsigned.apk

but is not RB. APK Diff:

  -rw-r--r--  0.0 unx   168352 b-    69680 defN 1981-01-01 01:01:02 393a229f classes2.dex
- -rw-r--r--  0.0 unx     7512 b-     2882 defN 1981-01-01 01:01:02 0093ac4f lib/arm64-v8a/libpmxd.so
+ -rw-r--r--  0.0 unx     7800 b-     2953 defN 1981-01-01 01:01:02 b355030d lib/arm64-v8a/libpmxd.so
  -rw-r--r--  0.0 unx    27080 b-    11274 defN 1981-01-01 01:01:02 af16dfb5 lib/arm64-v8a/libpmxe.so
  -rw-r--r--  0.0 unx    30224 b-    19307 defN 1981-01-01 01:01:02 e9f9b14e lib/arm64-v8a/libspake2.so
- -rw-r--r--  0.0 unx     5388 b-     2363 defN 1981-01-01 01:01:02 ee21066c lib/armeabi-v7a/libpmxd.so
+ -rw-r--r--  0.0 unx     5644 b-     2434 defN 1981-01-01 01:01:02 a42498a1 lib/armeabi-v7a/libpmxd.so
  -rw-r--r--  0.0 unx    22056 b-    10077 defN 1981-01-01 01:01:02 33d7e1d7 lib/armeabi-v7a/libpmxe.so
  -rw-r--r--  0.0 unx    25864 b-    17618 defN 1981-01-01 01:01:02 9b326212 lib/armeabi-v7a/libspake2.so
- -rw-r--r--  0.0 unx     6300 b-     2888 defN 1981-01-01 01:01:02 2b743280 lib/x86/libpmxd.so
+ -rw-r--r--  0.0 unx     6556 b-     2962 defN 1981-01-01 01:01:02 f2306e2d lib/x86/libpmxd.so
  -rw-r--r--  0.0 unx    24500 b-    12082 defN 1981-01-01 01:01:02 7f39c999 lib/x86/libpmxe.so
  -rw-r--r--  0.0 unx    46292 b-    24638 defN 1981-01-01 01:01:02 8af8c277 lib/x86/libspake2.so
- -rw-r--r--  0.0 unx     7272 b-     2790 defN 1981-01-01 01:01:02 73377181 lib/x86_64/libpmxd.so
+ -rw-r--r--  0.0 unx     7560 b-     2865 defN 1981-01-01 01:01:02 f69c9698 lib/x86_64/libpmxd.so
  -rw-r--r--  0.0 unx    27480 b-    12104 defN 1981-01-01 01:01:02 6fbd0049 lib/x86_64/libpmxe.so
  -rw-r--r--  0.0 unx    38680 b-    21111 defN 1981-01-01 01:01:02 f738153f lib/x86_64/libspake2.so
- -rw-r--r--  0.0 unx    23284 b-    23173 defN 1981-01-01 01:01:02 b1395f8d assets/daemon.dx
+ -rw-r--r--  0.0 unx    23284 b-    23173 defN 1981-01-01 01:01:02 3fa242f2 assets/daemon.dx
  -rw----     2.0 fat     1738 b-      782 defN 1981-01-01 01:01:02 7cded4df DebugProbesKt.bin

I've run diffoscope over it, but cannot make much out in its 1.5 MB output. My guess would be different compiler versions – but I've specified the NDK you named, and other things are not mentioned (e.g. a specific version of cmake, or whatever). But going by the longs, everything there seems to be called with a full path starting with /opt/sdk/ndk/25.1.8937393/toolchains/llvm/prebuilt/linux-x86_64/bin (NDK version taken from here).

diffoscope.html.gz

Any idea from your side?

[mirfatif]

No idea. Only the daemon files differ, not the app files. Did you do a clean build (./gradlew clean assembleSelfFossRelease)?

[IzzySoft]

I can add that, but what effect would that have in a "clean container"? Our builds here always run in a "virgin environment", so there should be nothing to "clean" as there was no previous build. Please correct me if I'm understanding something wrong there.

Edit: Give me a few minutes, looks like I just got something wrong with what I've just tried…

[IzzySoft]

OK, so this recipe:

- echo "sdk.dir=/opt/sdk" >> local.properties
- sed -r "s/!names.contains\('fdroid'/!names.contains('self'/ ; s/setExtSrcDependencies\('Fdroid', 'Foss'\)/setExtSrcDependencies('Self', 'Foss')/" -i app/build.gradle
- chmod +x gradlew
- ./gradlew assembleSelfFossRelease

You see there's just a single gradlew call. As this runs in a "virgin container", there's nothing to clean up, so ./gradlew clean assembleSelfFossRelease should not make a difference (I can try it if you say I didn't understand clean correctly as that it's supposed to clean up remains from previous builds as first step). And this is the diff in the resulting APKs:

  -rw-r--r--  0.0 unx   168352 b-    69680 defN 1981-01-01 01:01:02 393a229f classes2.dex
- -rw-r--r--  0.0 unx     7512 b-     2882 defN 1981-01-01 01:01:02 0093ac4f lib/arm64-v8a/libpmxd.so
+ -rw-r--r--  0.0 unx     7800 b-     2953 defN 1981-01-01 01:01:02 b355030d lib/arm64-v8a/libpmxd.so
  -rw-r--r--  0.0 unx    27080 b-    11274 defN 1981-01-01 01:01:02 af16dfb5 lib/arm64-v8a/libpmxe.so
  -rw-r--r--  0.0 unx    30224 b-    19307 defN 1981-01-01 01:01:02 e9f9b14e lib/arm64-v8a/libspake2.so
- -rw-r--r--  0.0 unx     5388 b-     2363 defN 1981-01-01 01:01:02 ee21066c lib/armeabi-v7a/libpmxd.so
+ -rw-r--r--  0.0 unx     5644 b-     2434 defN 1981-01-01 01:01:02 a42498a1 lib/armeabi-v7a/libpmxd.so
  -rw-r--r--  0.0 unx    22056 b-    10077 defN 1981-01-01 01:01:02 33d7e1d7 lib/armeabi-v7a/libpmxe.so
  -rw-r--r--  0.0 unx    25864 b-    17618 defN 1981-01-01 01:01:02 9b326212 lib/armeabi-v7a/libspake2.so
- -rw-r--r--  0.0 unx     6300 b-     2888 defN 1981-01-01 01:01:02 2b743280 lib/x86/libpmxd.so
+ -rw-r--r--  0.0 unx     6556 b-     2962 defN 1981-01-01 01:01:02 f2306e2d lib/x86/libpmxd.so
  -rw-r--r--  0.0 unx    24500 b-    12082 defN 1981-01-01 01:01:02 7f39c999 lib/x86/libpmxe.so
  -rw-r--r--  0.0 unx    46292 b-    24638 defN 1981-01-01 01:01:02 8af8c277 lib/x86/libspake2.so
- -rw-r--r--  0.0 unx     7272 b-     2790 defN 1981-01-01 01:01:02 73377181 lib/x86_64/libpmxd.so
+ -rw-r--r--  0.0 unx     7560 b-     2865 defN 1981-01-01 01:01:02 f69c9698 lib/x86_64/libpmxd.so
  -rw-r--r--  0.0 unx    27480 b-    12104 defN 1981-01-01 01:01:02 6fbd0049 lib/x86_64/libpmxe.so
  -rw-r--r--  0.0 unx    38680 b-    21111 defN 1981-01-01 01:01:02 f738153f lib/x86_64/libspake2.so
- -rw-r--r--  0.0 unx    23284 b-    23173 defN 1981-01-01 01:01:02 b1395f8d assets/daemon.dx
+ -rw-r--r--  0.0 unx    23284 b-    23173 defN 1981-01-01 01:01:02 7e1e03c0 assets/daemon.dx
  -rw----     2.0 fat     1738 b-      782 defN 1981-01-01 01:01:02 7cded4df DebugProbesKt.bin

I've run diffoscope against the two to see where the differences in the libraries are. Looking at the details for arm64-v8a/libpmxd.so, it seems the build here e.g. inserts an additional hash:

I'm far from understanding the rest, but it looks like that could have caused addresses to be shifted (I'm really not home in that area, so please forgive my ignorance in this concern). Here's the gzipped diffoscope output, maybe you can identify something there: diffoscope.html.gz

Build is running on Debian bookworm here with JDK 17. I could switch to Ubuntu jammy and JDK 21 if that's closer to your environment (shouldn't as you explicitly said "JDK 17"), Debian bullseye would also be an option here. Or specifying an explicit version for make, gcc, libc-dev, idk – the best bet is that different versions in one of the latter produce different output.

[mirfatif]

daemon.dx is a ZIP file. Its diff should be due to difference in ZIP timestamp or other metadata. Hopefully its contents are identical. But libpmxd.so also differs in size. One diff is the extra .hash section as you've shown. But there must be others. I don't have much idea about it.

[IzzySoft]

daemon.dx: confirm via diffoscope on the extracted ZIPs, timestamp only. But can that timestamp be avoided? If not, that prevents RB. Moot to ask if we cannot solve the other part, of course.

*.so: same here, that's beyond me. I've learned a lot in the past half year from my sensei (more than I thought possible for 2 years), but not to that extent. That's compiled from C, right? Maybe a different compiler or compiler version? I was running this on Debian bookworm. I can also run on bullseye or on Ubuntu jammy, if that would help. What does your build run on? I could also try to match the versions of make, gcc and libc-dev to yours.

[mirfatif]

But can that timestamp be avoided?

Yes. By hard-coding a timestamp in ZipEntry.

That's compiled from C, right?

Yes. The source code. And the build script.

Maybe a different compiler or compiler version?

Use the same NDK version.

What does your build run on?

Mostly Ubuntu.

I could also try to match the versions of make, gcc and libc-dev to yours.

make is required to build libpmxe.so, not libpmxd.so. NDK has it own toolchain including clang compiler, Android's libc and other standard headers. It doesn't use host's compiler or standard libraries.

[IzzySoft]

Yes. By hard-coding a timestamp in ZipEntry.

Good, thanks!

Yes. The source code. And the build script.

That script is called by gradle, right? So my above "build recipe" isn't missing anything in this regard (well, we'd not have the libs if it did I guess)?

Use the same NDK version.

jdk: openjdk-17-jdk-headless
ndk: 25.1.8937393

matches I'd say.

Mostly Ubuntu.

I can try again on Ubuntu jammy.

make is required to build libpmxe.so, not libpmxd.so.

OK, libpmxe.so is identical here (no differences between our builds).

---

Now, that's funny. I've tried with Ubuntu jammy: libs are identical now, only the timestamps in daemon.dx remain! Would you agree to your suggestion of hardcoding them so they are deterministic? If yes, please provide me an APK built from a commit having that change (I can pick the commit hash to build against from META-INF/version-control-info.textproto), I'd then check again if we can have the next release RB :partying_face: