Build question / Reproducible Builds

[IzzySoft]

Hi Irfan, you might have noticed that the IzzyOnDroid repo is now publicly featuring Reproducible Builds (see: Reproducible bulds, special client support and more in our repo). So of course I wanted to get your apps in, but have a problem with MyLocation: How do I build SelfFossRelease – assuming that is the one listed at IzzyOnDroid? I failed miserably, only succeeding with FdroidFossRelease.

We'd appreciate if you could help making your build reproducible. We've prepared some hints on reproducible builds for that.

Looking forward to your reply!

[mirfatif]

Hi,

Reproducible Builds sound great for open-source community. And you are correct. selfFossRelease task is disabled on the public repo. self builds are those built by us and signed with our own keys. fdroid builds are meant to be built by F-Droid (or the open-source community) and signed by their keys. This is to identify the builds e.g. when we get crash logs or support requests for different builds.

To enable the self builds one can simply change:

if (!names.contains('fdroid') || !names.contains('foss')) {
  variant.setIgnore(true)
}

to

if ((!names.contains('fdroid') && !names.contains('self')) || !names.contains('foss')) {
  variant.setIgnore(true)
}

in build.gradle.

[IzzySoft]

Thanks! But oh, now I need to get hold of all the RegEx I can gather… oof:

's/if (\(\!names\.contains\('"'"'fdroid'"'"'\)) (..) \!names\.contains\('"'"'foss'"'"'\)\).+/if (\1 \&\& !names.contains('"'"'self'"'"')) \2 !names.contains('"'"'foss'"'"')) {/'

Took me a while to cobble that together. Single quotes and pipes are tricky. OK, now let's see, build is running… Well, actually it's taking ages to download all those gradle plugins, at around 100kB/s (my line would easily give 35MB/s, so this looks like throttling) :disappointed: The "durian" one is downloading now for several minutes already.

(3 builds later)

OK, now the build succeeds (had a typo in the original RegEx, already corrected above). 4th build running now as the output file wasn't where it was expected.

Finally, there we go: it's not RB. Here comes the APK diff:

-------------------------------
--- /dev/fd/63  2024-08-04 16:37:12.983054700 +0200
+++ /dev/fd/62  2024-08-04 16:37:12.983054700 +0200
@@ -1,11 +1,11 @@
   META-INF/com/android/build/gradle/app-metadata.properties
   32-bit CRC value (hex):                         41228e1e
   assets/dexopt/baseline.prof
-  32-bit CRC value (hex):                         3a96f3c4
+  32-bit CRC value (hex):                         27206cc5
   assets/dexopt/baseline.profm
   32-bit CRC value (hex):                         ab338062
   classes.dex
-  32-bit CRC value (hex):                         86203828
+  32-bit CRC value (hex):                         24b49127
   classes2.dex
   32-bit CRC value (hex):                         35d55042
   META-INF/androidx.activity_activity.version
@@ -1844,9 +1844,3 @@
   32-bit CRC value (hex):                         9f343fb7
   resources.arsc
   32-bit CRC value (hex):                         1705207c

Putting that in order: baseline.prof differs because of classes.dex does. Let's start with that… hm, two pages of diff. Comparing red (from your APK) vs. green (from my build) it seems the APK was either built from a later commit with things added, or from an earlier one and things got dropped. Example:

 |: invoke-static {}, Lv2/a;.i:()Z
 |: move-result v0
-|: if-eqz v0, 0022 // +0014
-|: invoke-static {}, Ly1/i;.u:()Z
-|: move-result v0
-|: if-eqz v0, 0022 // +000e
+|: if-eqz v0, 001c // +000e
 |: iget-object v0, v2, Lcom/mirfatif/mylocation/MainActivity;.v:Lu2/a;
 |: iget-object v0, v0, Lu2/a;.d:Landroid/widget/TextView;

Looking at the timestamps I bet the former: tag is from November 2023, APK from January 2024. But there's no commit after the tagged one – so maybe you didn't push that change? §1 of Izzy's RB hints:

§1: Always build the APK from exactly the commit your release-tag will point to!

:wink:

So how can we get that fixed? Any hints from your side?

[mirfatif]

Looking at the timestamps I bet the former: tag is from November 2023, APK from January 2024. But there's no commit after the tagged one – so maybe you didn't push that change?

There's no change in source code after Nov'23. But APK was rebuilt in Jan'24 with v1 signing scheme enabled as you had reported an issue earlier (https://github.com/mirfatif/MyLocation/issues/16).

So how can we get that fixed? Any hints from your side?

Compiled code for selfFoss and fdroidFoss variants won't ever be identical, even if built at the same time. The files generated by Android build system could be different for different variants. In our case, the contents of BuildConfig.java file, for instance, are different due to this and this. So the resulting dex files would also differ.

[IzzySoft]

Compiled code for selfFoss and fdroidFoss variants won't ever be identical

No, and I would not expect them to be. That's why I asked you how to build the selfFoss variant, as that's what is listed at IzzyOnDroid. Here's the recipe so you can see what is compared:

---
repository: https://github.com/mirfatif/MyLocation.git
updates: releases
versions:
  - tag: v1.05
    apks:
      - apk_pattern: MyLocation.*\.apk
        apk_url: https://github.com/mirfatif/MyLocation/releases/download/v1.05/MyLocation_v1.05.apk
        build:
          - sed -r 's/if (\(\!names\.contains\('"'"'fdroid'"'"'\)) (..) \!names\.contains\('"'"'foss'"'"'\)\).+/if (\1 \&\& !names.contains('"'"'self'"'"')) \2 !names.contains('"'"'foss'"'"')) {/' -i app/build.gradle
          - ./gradlew --info assembleSelfFossRelease
          - find . -name '*.apk'
          - mv app/build/outputs/apk/selfFoss/release/app-self-foss-release-unsigned.apk /outputs/unsigned.apk
        build_home_dir: /build
        build_repo_dir: /build/repo
        build_user: build
        provisioning:
          android_home: /opt/sdk
          build_tools:
          cmake:
          cmdline_tools:
            version: '12.0'
            url: https://dl.google.com/android/repository/commandlinetools-linux-11076708_latest.zip
            sha256: 2d2d50857e4eb553af5a6dc3ad507a17adf43d115264b1afc116f95c92e5e258
          extra_packages: []
          image: debian:bookworm-slim
          jdk: openjdk-17-jdk-headless
          ndk:
          platform:
          platform_tools:
          tools:
          verify_gradle_wrapper: true

It compares the APK attached to your release here (apk_url) with the one created by the recipe. You see I've used the above RegEx to make sure it can build selfFoss, then called it with ./gradlew assembleSelfFossRelease (the --info was just in for debugging). Finally used is app/build/outputs/apk/selfFoss/release/app-self-foss-release-unsigned.apk – which should be the selfFoss build. Isn't that what is attached to your release here? If not, maybe I wasn't clear enough with my question: How would I build the APK you've attached to releases here? Apologies for any confusion I might have caused!

[mirfatif]

You were perfectly clear. Probably I was sleepy. :smile:

Ok. I've synced my private branch (on which I work and build APKs, and which contains extra code for non-FOSS variants) with the public one (which I push to GitHub) and ran assembleSelfFossRelease on both branches. The resulting classes.dex still differs. At the moment I don't know why. And don't have time to investigate. Probably there's something wrong with Java API desugaring.

For now I've built the APK on public branch and released v1.06. See if it fixes the problem.

[IzzySoft]

What shall I say? Welcome to the collection of Reproducible Builds at IzzyOnDroid then, Irfan – and thanks for your help! This was a full success.

Err… but a flaky one, second try failed. Lemme see what the 3rd one says… OK, success again. The failure was because some short network hick-up it seems. Should that happen more often, I guess I could simply alter the relevant build step to

- ./gradlew --info assembleSelfFossRelease || ./gradlew --info assembleSelfFossRelease

i.e. repeat the build if it fails :man_shrugging: But for now: v1.06 will show up as RB with the next sync around 6 pm UTC :smiley:

Feel free to close this ticket. I left it open in case you think there's something left to do on your end (e.g. making a note to process future releases like this one :stuck_out_tongue_winking_eye:)