Closed IzzySoft closed 2 months ago
Acclorite
commented
2 months ago Ehm, how would my app be on IzzyOnDroid in the first place, if I didn't even do anything for it?
Also, the build is Reproducible. It passed FDroid build successfully in both merge request and FDroid build for app update later.
So care to explain what's going on here?
Acclorite
commented
2 months ago Well I appreciate it, but honestly am not sure what is going on? I am sure that the apk from Release v1.1.0 is exact as Tag v1.1.0, if you can provide me with an apk of your build, I can test reproducibility myself (like I did with F-Droid merge).
Edit: Okay. I read about reproducible builds on IzzyOnDroid, so on F-Droid official repo it's about having same apk signature, and here it is about security check? Anyway, thanks for your reply.
Edit2: Built APK from v1.1.0, compared with tag v1.1.0 and there is indeed huge difference, not even sure why. A couple of days back on the same commit results were different. I already made a lot of changes to reproducibility on test branch, meaning for now this issue is frozen. Please wait for the next release, where I will test it all thoroughly, as of now, for Release v1.1.0, it passed checks on F-Droid, therefore APKs are identical.
Not really sure what went wrong and why, are you sure you use same Build Tools(34.0.0) and Java version (17)? Release Tag -- Release version
IzzySoft
commented
2 months ago Ehm, how would my app be on IzzyOnDroid in the first place, if I didn't even do anything for it?
Eh – I hope you're not opposed? It comes under a free license allowing free distribution, so $someone found it and it was added. Also to increase visibility and make it easier for people to find it, install it and keep it updated without hassle – additional security scans and more included (which all confirmed your APK is fine, as you can meanwhile see as it's live now already).
It passed FDroid build successfully in both merge request and FDroid build for app update later.
It's not listed at F-Droid, and not in their index. But let me check their metadata… Ah, OK, it was just added there. Funny, I wonder how it's reproducible for them. Let me compare the build recipes… Recipes are identical, too. Might be a "flaky build" then (i.e. one that's non-deterministic, or depends on the number of CPU cores used). I've tried the build only once, and with the diff being that huge didn't consider it might be just a flaky one.
Well I appreciate it, but honestly am not sure what is going on?
Thanks! Well, what's going on is that your app was being added to the IzzyOnDroid repo – which is independent from F-Droid but "uses the same protocol" so to say. Differences are e.g. that IoD directly takes the APKs the developers build (so updates are much faster: less than 24h instead of 2..5 days there) and applies several additional scans, see e.g. Ramping up security: additional APK checks are in place with the IzzyOnDroid repo. Reproducible Builds went public here only recently (August 1st, see Reproducible Builds, special client support and more in our repo, though we already have them since February).
if you can provide me with an apk of your build, I can test reproducibility myself (like I did with F-Droid merge).
I can try to attach it here, though I'm not sure if Github supports that size… Nope, it doesn't:
Okay. I read about reproducible builds on IzzyOnDroid, so on F-Droid official repo it's about having same apk signature, and here it is about security check?
The principle is the same in both places. Just with F-Droid, if RB fails, there will be no APK shipped at all – while at IoD the check goes asynchronously and indeed shows up as an additional "security attribute", without the release being stopped or postponed.
Anyway, thanks for your reply.
Gladly, thanks for yours, too!
Built APK from v1.1.0, compared with tag v1.1.0 and there is indeed huge difference, not even sure why.
Ah! Thanks for confirming – I was already mightily puzzled. You could compare the build logs. As RB failed here right from the beginning, ours are not yet published (but would most likely be similar to your latest). But F-Droid's, which say they were RB, should be public: right at the MR for ~30d in the GitLab artifacts, but also on their server (should be linked from your app's page there once it goes online, but let me check if I can find it by hand… There it is: https://f-droid.org/repo/ua.acclorite.book_story_5.log.gz
Not really sure what went wrong and why, are you sure you use same Build Tools(34.0.0) and Java version (17)?
I'm not sure either – but I can confirm Java-17 – and whatever your build instructions say. Here's our recipe:
---
repository: https://github.com/Acclorite/book-story.git
updates: releases
versions:
- tag: v1.1.0
apks:
- apk_pattern: book-story\.apk
apk_url: https://github.com/Acclorite/book-story/releases/download/v1.1.0/book-story.apk
build:
- sed -r '/signingConfigs.getByName/d' -i app/build.gradle.kts
- chmod +x gradlew
- ./gradlew assembleRelease
- mv app/build/outputs/apk/release/*unsigned.apk /outputs/unsigned.apk
build_home_dir: /build
build_repo_dir: /build/repo
build_user: build
provisioning:
android_home: /opt/sdk
build_tools:
cmake:
cmdline_tools:
version: '12.0'
url: https://dl.google.com/android/repository/commandlinetools-linux-11076708_latest.zip
sha256: 2d2d50857e4eb553af5a6dc3ad507a17adf43d115264b1afc116f95c92e5e258
extra_packages: []
image: debian:bookworm-slim
jdk: openjdk-17-jdk-headless
ndk:
platform:
platform_tools:
tools:
verify_gradle_wrapper: trueI can give it another run to see what Build Tools it used – but with all builds I've seen so far, it pulls in several including 34. Will do that now and attach the logs with the next comment then, give me a few minutes…
obfusk
commented
2 months ago https://github.com/Acclorite/book-story/blob/master/app/build.gradle.kts has a bunch of latest.release instead of pinning specific versions. Which means two builds from the same tag just a few days apart will likely be different if any of those dependencies released a new version in the mean time. For Reproducible Builds you need to pin specific versions instead.
IzzySoft
commented
2 months ago OK, Build tools 34.0.0, Platform tools 35.0.2 (build still running). – Ah, Fay chiming in – and as always, to the point: that perfectly explains it. Could call it "build time dependencies": depending on when you buid, you get different versions of dependencies. So I basically can cancel the build here. But as I "promised" logs, I'll wait it out and attach them… ah, there they come: ua.acclorite.book_story.json (you can extract the build logs themselves using
jq -rs '.[]["tags"][][].build_log' ua.acclorite.book_story.json.
obfusk
commented
2 months ago And indeed, looking at the embedded aboutlibraries JSON file we see Izzy's build picked up dagger/hilt 2.52 released yesterday:
Acclorite
commented
2 months ago Thank you all for the help, but issue is not that I believe. Comparing file to file it has much more difference, a lot of files just do not exist.
I will specify all versions instead of latest.release, thanks for notice. As for now, I guess it's fine to leave it as is until next version. I was able to enable minify and stuff that caused problems for reproducibility, which greatly decreased app size (29mb > 12mb), so more changes expected, do not see much sense in trying to fix already outdated code.
I am waiting for app appear in F-Droid to merge all developments up until now (to prevent any issues, if something went wrong).
And still in the end, if it is not you, who added the app to the repo? xD Some secret volunteer is here, huh?
Acclorite
commented
2 months ago And indeed, looking at the embedded aboutlibraries JSON file we see Izzy's build picked up dagger/hilt 2.52 released yesterday:
Still won't believe one version can cause 100+ differences in files, but who knows...
Acclorite
commented
2 months ago Eh – I hope you're not opposed?
I am not, just don't understand how that happened, even before F-Droid release. Although I had some plans to add it there, but someone predicted and added it before me.
obfusk
commented
2 months ago Unfortunately, the aboutlibraries JSON (and nothing else) also differs between two of Izzy's builds just hours apart:
I'm not sure what caused that -- maybe related to github rate limiting for fetchRemoteLicense? -- but it seems likely to break Reproducible Builds.
Acclorite
commented
2 months ago I'm not sure what caused that -- maybe related to github rate limiting for fetchRemoteLicense? -- but it seems likely to break Reproducible Builds.
I know about this issue, and I seem to fix it on AboutLibraries Reproducibility Test branch. As I said, there is no much sense in trying to fix outdated code, we'll see after I merge everything(which is after successful appearing of the app in F-Droid). Then I will run tests b4 release, ensuring everything is reproducible.
You can learn about my suffering from this issue here.
P.s. Tested with intervals, first 2 clean builds - success. Compared with one more clean build after ~3-5 hours - success.
obfusk
commented
2 months ago Nice work. Good luck.
FYI: I built the tooling F-Droid depends on for Reproducible Builds. But they no longer have an expert to fix things when they break or anyone that understands the tooling they depend on. No one to build new workarounds when Google breaks something again in their toolchain. I really hope that doesn't end up causing problems for your app in the future if new Reproducible Builds problems crop up.
IzzySoft
commented
2 months ago a lot of files just do not exist.
That makes me wonder how the F-Droid RB was able to succeed…
I guess it's fine to leave it as is until next version.
Sure: other than with F-Droid, it's not stopping distribution at IzzyOnDroid. Always good to have a (safety) net and a double bottom :wink: RB even gives the option for cross-updates here then – like, if you need to push out some "emergency update" and don't want folks to wait for F-Droid's build cycle. Whoever has both repos added (many have, we have more than 180k daily visitors just on the main server, not counting the mirrors), can easily update from both places (depending on the F-Droid client used, it might mean manual steps though, e.g. with the official client).
I am waiting for app appear in F-Droid to merge all developments up
I'd say no need to wait, as their cycle is just through – and your app was already built. According to their monitor, stuff is now in "sneaker net" (Ciaran manually pulling the files from the build server to the air-gapped signing server for signing, then back for publication). That started 10 hours ago – not sure if he already has the sneakers on…
Some secret volunteer is here, huh?
When in doubt, it was the ferret. Ferrets are known to drag things behind and below the canapé… :see_no_evil: Just one thing for sure: it was not the cat who dragged it in :stuck_out_tongue_winking_eye:
I am not, just don't understand how that happened, even before F-Droid release. Although I had some plans to add it there, but someone predicted and added it before me.
Hehe, easy to explain: at IoD it takes less than a day to add an app. One of my crawlers found and suggested your app to me, so I added it :man_shrugging: F-Droid takes weeks as best from you opening an RFP and the app becoming available at F-Droid. They can't do that in less than 3 days, as that's as long as a build cycle needs there. If you're unlucky, they just manage to merge the MR a minute after the cycle started – so it's 6 days already…
Acclorite
commented
2 months ago That makes me wonder how the F-Droid RB was able to succeed…
Because it seems to happen exactly today or yesterday, what a timing. 2 days ago I tested and it was reproducible, which is a day after F-Droid merge request succeed.
I'd say no need to wait, as their cycle is just through – and your app was already built.
Well really, I better wait one more day to merge changes, and actually I actively develop new feature now, which is quite large, so it will take a while, therefore I see no minuses in waiting a day or so if it ensures no problems.
One of my crawlers found and suggested your app to me, so I added it
And still, who could suggest it? I mean this app is clearly not a synonymous of "popular", and such timing..
I will comment here when this topic moves.
obfusk
commented
2 months ago Because it seems to happen exactly today or yesterday, what a timing. 2 days ago I tested and it was reproducible, which is a day after F-Droid merge request succeed.
Yeah, timing indeed. Your release APK was published on 1 Aug. The F-Droid MR had a successful Reproducible Build at that time. And your tests succeeded just fine. F-Droid built on 4 Aug and got the same APK as well.
But then there was a new dagger/hilt release on 5 Aug. And thus today, 6 Aug, Izzy got a different APK.
Still won't believe one version can cause 100+ differences in files, but who knows...
Given that hilt/dagger is not just a regular library but one that generates code (for dependency injection) I do not find this surprising at all.
Acclorite
commented
2 months ago Given that hilt/dagger is not just a regular library but one that generates code (for dependency injection) I do not find this surprising at all.
Well you do have a point.
IzzySoft
commented
2 months ago And still, who could suggest it?
My crawler:
I will comment here when this topic moves.
:+1:
Acclorite
commented
2 months ago Hey @IzzySoft, I did merge all the required changes.
Also seems like I was finally able to get rid of AboutLibraries issues - by manually getting libraries beforehand and adding them to res/raw/ folder.
Can you please test this APK on your side? This would make things easier if there are no issues, as I plan on releasing a new version soon:
Commit hash: 69a72c59fcf1c5fc4bf0fe8d6b546acb61ce3ad2 Local build: local build.zip
If you get errors this time too, please provide the apk, so I can test it too.
IzzySoft
commented
2 months ago After fixing the line breaks in META-INF/services/* and res/M7.json (you've built this on Windows, we build on Linux), it's RB, thanks!
Your regular builds happen on Linux, too, right? So no need to keep that "linebreak fixing" on our end then?
Acclorite
commented
2 months ago Your regular builds happen on Linux, too, right? So no need to keep that "linebreak fixing" on our end then?
No, my regular builds happen on Windows.
After fixing the line breaks in META-INF/services/* and res/M7.json (you've built this on Windows, we build on Linux), it's RB, thanks!
Guys on F-Droid told me to ignore META-INF and original folders, do you not? I will write here once more just on the expected tag commit, so you can test it.
obfusk
commented
2 months ago No, my regular builds happen on Windows.
That's odd; book-story.apk from tag v1.1.0:
The APK you uploaded here:
Looks like something changed then, if both builds were on Windows.
Guys on F-Droid told me to ignore META-INF and original folders, do you not?
They don't ignore those either. They can't. Because Reproducible Builds require bit-by-bit identical APKs. Which is why F-Droid also uses my apksigcopier (to copy the signature) and reproducible-apk-tools (to fix things like newline differences).
The one thing that does make sense is ignoring the 3 v1 signature files (but not everything else) in META-INF/ when comparing the extracted contents of a signed and unsigned APK. But in the end every byte in the APK must be identical (after signature copying).
All of these F-Droid recipes use my tooling to fix newlines in META-INF/services:
It looks like they will need to do the same for your app, just like Izzy.
Acclorite
commented
2 months ago It looks like they will need to do the same for your app, just like Izzy.
Can I do it myself?
That's odd; book-story.apk from tag v1.1.0
I am not sure either, I clearly built everything on Windows machine.
Looks like something changed then, if both builds were on Windows.
But one thing caught my eye, these classes are similar to those I kept before, or played to get build reproducible, so my guess is proguard-rules changes impacted this?
The one thing that does make sense is ignoring the 3 v1 signature files (but not everything else) in META-INF/ when comparing the extracted contents of a signed and unsigned APK. But in the end every byte in the APK must be identical (after signature copying).
Well maybe that's why they told me that, after all I tested unsigned and signed apks.
So, should I be alright if I ignore those folders as I did before? I mean, do differences in those folders matter for me? Is there anything I should be aware of and fix? I already replaced latest.release versions, manually loaded libraries and set up proguard-rules to work without conflicts.
obfusk
commented
2 months ago Can I do it myself?
You could build on Linux. Or use my tooling on your end to change the line endings from CRLF to LF. But for the latter you would have to build an unsigned APK, then fix the line endings, and then manually sign it. It's probably easier to handle it on our end (and F-Droid's).
But one thing caught my eye, these classes are similar to those I kept before, or played to get build reproducible, so my guess is proguard-rules changes impacted this?
Yeah, maybe. Was also wondering if it was related to r8/minification. I reported the newline differences to Google ages ago but never got a response to the issue I filed and I don't know which part of the toolchain generates those files or why it does so differently on Windows and Linux unlike other parts.
So, should I be alright if I ignore those folders as I did before? I mean, do differences in those folders matter for me?
Well, as Izzy just demonstrated differences in those folders caused Reproducible Builds to fail. The same would apply to F-Droid. Newline differences are fixable -- if we know about them. But other differences -- like different commit hashes or versions -- might indicate something actually "wrong". You should only ignore the v1 signature files. But your APKs don't have a v1 signature (only v2), so there are no v1 signature files for you to ignore. I guess they missed that.
Is there anything I should be aware of and fix? I already replaced
latest.releaseversions, manually loaded libraries and set up proguard-rules to work without conflicts.
With those changes, Izzy got a successful Reproducible Build (and we know it won't fail because of unpinned versions changing again). F-Droid should be able to do the same with the same workarounds, which they (and we) already use for many other apps. So unless we overlooked something (or Google breaks something again) it should be fine now, apart from the workaround needed because of differences between building on Linux and Windows.
obfusk
commented
2 months ago Well maybe that's why they told me that, after all I tested unsigned and signed apks.
You can use apksigcopier compare to compare a signed and unsigned APK. Though it will only tell you if they are identical. It doesn't tell you what the differences are if not. But if it gives you an error, the build was not reproducible.
Acclorite
commented
2 months ago Okay, thanks for reply.
I will test reproducibility anyway before release with fdroid build.
Minification works really strange to me, it can give you just one letter difference in file name, or just one letter difference in class name.
Oddly enough, FDroid wiki does not have much info about R8, only that keep classes recommendation, which caused only more troubles for me as R8 works really bad if you keep classes in proguard, at least in my experience. Is that a rare issue?
Also FDroid does not seem to have much info about fixing these differences, I just feel like this reproducibility topic is quite undeveloped.
IzzySoft
commented
2 months ago Also FDroid does not seem to have much info about fixing these differences
All they know about it came from Fay. They do no longer have any expert on RB on the team. They still use Fay's tooling (as she pointed out), use the instructions from her tooling, and so on. But all what changed since we left last year, all that Fay developed since then, they could only "copy over" without really understanding it. So if things fail, they often cannot address that methodically but go mostly by try-and-error.
I won't say I'm a master of the topic. And I'd rather leave it to my sensei (Fay) to say where I stand on the topic (I'd put myself somewhere at junior level, but guess I can say I'm out of my apprentice shoes). But I have the great luck and honor that I can ask Fay when I don't understand something and cannot figure it myself. They closed that door and thus can't.
Acclorite
commented
2 months ago But I mean.. even me, who never ever heard about Reproducible Builds a week ago could already add a couple of things there.
And that's not even hard to find out, just some tests. It feels more like they do not test it well enough. There are a lot of good advices, but some are missing, like the work is only half done.
obfusk
commented
2 months ago Oddly enough, FDroid wiki does not have much info about R8, only that keep classes recommendation, which caused only more troubles for me as R8 works really bad if you keep classes in proguard, at least in my experience. Is that a rare issue?
That R8 part of their docs was written before I started working on Android RB (before anyone had figured out v2/v3 signature copying). I have never seen that issue. I'm not sure why it's in their docs when no one has seen it in years. I've only seen the R8 desugaring issue, but that was also fixed long ago.
About libraries, how to manually export libraries to some path, so you don't have to deal with random hash
I've actually never seen that issue before. At least not causing differences between builds (I don't usually look at what's in the JSON). Just the timestamps. I guess most apps don't use fetchRemoteLicense like you did. Manual exporting can be useful, yeah. Though it does mean you need to keep that up to date somehow (could use a CI job to automate it).
There are a lot of good advices, but some are missing, like the work is only half done.
I wrote most of that page, but there was still much to do when I left. And for example when Android Gradle Plugin >= 8.3 changed the default page alignment from 4k to 16k it took them months of failing RB to come up with an ugly workaround because they didn't realise that and were using my tooling wrong.
The Android ecosystem is constantly changing. Some reproducibility issues get fixed, but plenty of new ones keep appearing all the time. Even with an expert it's hard to keep up as some things can't easily be worked around and Google can be very slow to fix issues in the toolchain, if they do so at all. And as Izzy said, F-Droid no longer has an expert on their team.
Acclorite
commented
2 months ago I guess most apps don't use fetchRemoteLicense like you did.
Haha, fair. I just am a completionist, so I felt like I should include that license description if there is a way to.
Though it does mean you need to keep that up to date somehow (could use a CI job to automate it).
Actually not hard at all. Just one single command in terminal, which will automatically fetch and export libraries to desired path, then simple commit - done.
F-Droid no longer has an expert on their team.
Doesn't that mean, that Reproducible Builds topic sooner or later will backfire and they will have to deal with hundreds of packages not being updated - if only one single major issue would appear?
I see it as a good sign to encourage developers switch to Reproducible Builds, but what if you cannot ensure, that those Reproducible Builds may be achieved?.
I feel like it was a good thing for my app to be added on IzzyOnDroid xD
obfusk
commented
2 months ago I just am a completionist
Relatable :)
Actually not hard at all. Just one single command in terminal, which will automatically fetch and export libraries to desired path, then simple commit - done.
I figured it was easy. I just prefer automating things so I don't have to (remember to) do stuff manually :P
Doesn't that mean, that Reproducible Builds topic sooner or later will backfire and they will have to deal with hundreds of packages not being updated - if only one single major issue would appear?
I was already worried about that before I left, with such a low bus factor. And back then I didn't realise just how little they understood the tooling I built for them. Now even more. They seem to think trial and error will somehow work out. Or that someone -- usually me -- will fix it for them :woman_shrugging:
I feel like it was a good thing for my app to be added on IzzyOnDroid xD
We're glad to have you on board :)
IzzySoft
commented
2 months ago Doesn't that mean, that Reproducible Builds topic sooner or later will backfire
first occurrences of that already happened, and at least one app was "migrated back" from RB to F-Droid signing it – which means for whoever had installed it to uninstall and reinstall. Sure, one could have used Signing Key Rotation – but unfortunately that was knowingly broken back in January/February despite multiple warnings, so that option no longer exists at F-Droid (but is still available at IzzyOnDroid, as we used proper patches which they refused).
I see it as a good sign to encourage developers switch to Reproducible Builds
It definitely is, and we cannot agree more!
but what if you cannot ensure, that those Reproducible Builds may be achieved?.
That's one reason why at IzzyOnDroid we choose a different way to achieve this. So even if we could no longer achieve RBs, no "delivery" will be dropped or even postponed. All that would be gone would be the "RB shields" telling that an app was successfully confirmed RB.
I feel like it was a good thing for my app to be added on IzzyOnDroid xD
We certainly hope so :smiley:
They seem to think trial and error will somehow work out.
Or that the next release will magically fix the issue (one of the more frequent comments: Let's wait for the next release…)
We're glad to have you on board :)
^^ that :smiley:
Acclorite
commented
2 months ago Hey @IzzySoft, can you test this APK? If everything goes right it should be the one to be released, just want to verify.
Commit Hash: 6fdd5f1c1046cb4d7b3d8c1b35e60e864a008dc1 Local Build: local.zip
Also, is there a need to ask you to test APK if the one from F-Droid passes?
Edit: Everything is fine except META-INF, original and aboutlibraries.json(lol) - diffs in line breaking(WIndows/Linux), I guess that's fine? Edit 2: +1 reason to dump this piece of crap and say I use arch btw
IzzySoft
commented
2 months ago "upstream_signed_apk_sha256": "78dbaf313669180ced759075e6c0c065b69f95d388b1c2a6865daa0ed53aa84e",
"built_unsigned_apk_sha256": "5a26749329962732a323234244385a57ec7e9fab202666aad1e1e89dda2dd3a3",
"signature_copied_apk_sha256": "78dbaf313669180ced759075e6c0c065b69f95d388b1c2a6865daa0ed53aa84e"That's a clean RB, thanks! Btw: had you just renamed the local.apk to local.apk.zip instead of putting it inside a zip, we could have used it directly from the attachment :wink: But unpacking and feeding it to a local web server here worked as well :stuck_out_tongue_winking_eye:
diffs in line breaking(WIndows/Linux)
Our build recipe here takes care for that, thanks:
- git clone -b v0.2.8 https://github.com/obfusk/reproducible-apk-tools.git
- OUT=app/build/outputs/apk/release/app-release-unsigned.apk
- reproducible-apk-tools/inplace-fix.py --internal --page-size 16 fix-newlines "$OUT" 'META-INF/services/*' 'res/M7.json'(the --page-size 16 is for uncompressed *.so page alignment, the rest is fixing the newlines)
Edit 2: +1 reason to dump this piece of crap and say I use arch btw
Hehe, LOL, yeah. We build on Debian btw :speak_no_evil: :dash:
IzzySoft
commented
2 months ago PS: You give us a ping once the release is up, so we can "finalize" on our end as well?
Acclorite
commented
2 months ago Hey @obfusk, @IzzySoft. I have a problem.
Is there a way to test reproducibility other than through GitLab with metadata on fdroiddata fork? I ran out of compute minutes(...), and on top of that noticed serious issue caused by R8..
As R8 a serious topic for reproducibility I need a way to build(preferably locally) an APK - same as from F-Droid, to further compare it with my local build.
Before I would just update commit hash on test metadata on my fdroiddata fork, but now, when I have no more compute minutes, I need some other method, would be good without restraints.
As I feel like pinging you to test APK is inconvenient for both us.
obfusk
commented
2 months ago Is there a way to test reproducibility other than through GitLab with metadata on fdroiddata fork?
We use rbtlog, which is completely independent of F-Droid and a lot easier to setup than F-Droid's buildserver. All you need is some python packages and podman (or docker).
It does however require an upstream tag and a link to an upstream APK to download as it's meant to verify published APKs.
I guess I could add a way to build a specific commit and compare against a local APK instead for those that want to test RB before a release, though the commit would still need to be published to the repo.
Edit: I'm not sure it runs on Windows though. But you could run it in GitHub Actions.
Acclorite
commented
2 months ago I guess I could add a way to build a specific commit and compare against a local APK instead for those that want to test RB before a release, though the commit would still need to be published to the repo.
Haha that would be great, commiting to the origin is not the issue.
Edit: I'm not sure it runs on Windows though. But you could run it in GitHub Actions.
Well I can run Linux Sub-System, or switch......
So, what should I do for now?
Acclorite
commented
2 months ago @IzzySoft, can you please test this apk? I have no compute minutes, and this one is direct change to proguard-rules which is very sensitive for reproducibility..
Commit hash: 1368422175d87390557392655fa5418320341c2b Local build: local.apk.zip
Thanks in advance!
IzzySoft
commented
2 months ago Running now…
IzzySoft
commented
2 months ago Not RB:
-------------------------------
--- /dev/fd/63 2024-08-10 00:52:31.513202170 +0200
+++ /dev/fd/62 2024-08-10 00:52:31.513202170 +0200
@@ -3,11 +3,11 @@
META-INF/version-control-info.textproto
32-bit CRC value (hex): c1df57cb
assets/dexopt/baseline.prof
- 32-bit CRC value (hex): 9b21c307
+ 32-bit CRC value (hex): b35e8e9a
assets/dexopt/baseline.profm
32-bit CRC value (hex): 5732a882
classes.dex
- 32-bit CRC value (hex): 3e6565d7
+ 32-bit CRC value (hex): 94996e33
lib/arm64-v8a/libandroidx.graphics.path.so
32-bit CRC value (hex): 9734baa0
lib/arm64-v8a/libdatastore_shared_counter.soDexdiff does not give much of a clue, though:
-- /tmp/tmp.ofRVsbpAHf 2024-08-10 00:53:24.392847527 +0200
+++ /tmp/tmp.pE25kPjBPt 2024-08-10 00:53:25.104842693 +0200
@@ -1,9 +1,9 @@
-Processing '/tmp/tmp.dGXYU0cHbs'...
-Opened '/tmp/tmp.dGXYU0cHbs', DEX version '038'
+Processing '/tmp/tmp.apvOwMHuit'...
+Opened '/tmp/tmp.apvOwMHuit', DEX version '038'
DEX file header:
magic : 'dex\n038\0'
-checksum : a5219789
-signature : 9a5a...a8be
+checksum : 04359689
+signature : abdc...d8b7
file_size : 8112064
header_size : 112
link_size : 0The Prof just differs in the hashsum for the dex.
obfusk
commented
2 months ago @IzzySoft can you compare xxd output for the DEX?
Acclorite
commented
2 months ago I know the issue as I already went through it, just was hoping this time..
Well, that's keeping classes in proguard-rules.pro for you.
Gladly I have an option B. 🤫 Although will need to do some manual work by replacing .xml with .png.
Basically, to properly display and not give an exception you need to keep XmlParser in proguard, but that creates unreproducible build. As a workaround you can replace all .xml drawables with .png, which will make XmlParser unused, resulting in no exceptions, therefore no need to keep.
Kinda ugly, as you cannot use SVG for your app, but better than no minification.
And here comes one more complaint about FDroid wiki:
Why absolutely 0 information about R8's keeping classes problems? Well, I suppose it can be issue on my side, but if that happened to me, shouldn't it be possible to happen for others too?
obfusk
commented
2 months ago I have never heard of apps being unreproducible because they use XML drawables. Do you have more info about that?
I do suspect the RB failure here is related to R8 as that empty diff suggests it's the pg-map-id that differs, which happens when the proguard map file isn't identical (e.g. not deterministic somehow). Might be interesting to compare the one on your end to one from our builds.
Acclorite
commented
2 months ago Okay. I tried to be short in previous comment, but I will try to explain more as you may know something..
I have never heard of apps being unreproducible because they use XML drawables. Do you have more info about that?
The problem is not in XML Drawables nor XmlParser. The problem is in R8. Now explanation.
Basically, to properly display and not give an exception you need to keep XmlParser in proguard
This means, to use XML Drawables I need to -keep class org.xmlpull.** { *; }. But on the other hand -keep class ... itself result in not deterministic results for me. So, the problem is not in XmlParser, it's just one of the results of the problem.
Let's put down XmlParser as it has nothing to do with this problem.
I have checked almost every line of code somehow related to this and I found 2 things that matter.
isMinifyEnabled = false - fixes reproducibility issues by disabling minification at allproguard-rules.pro - very sensitive, may cure and also curse you.To begin with, proguard-rules.pro by default(without keeping any names or classes) already produces not deterministic results. Why? I don't know. My main suspect is Windows/Linux differences, as it is the only thing that I can really compare..
To fix issues with proguard, I need to -keepnames class **, which keeps the names of all classes, as the main issue is in that.
If something produces more issues, I also need to keep class method e.g. -keepnames class some.class.Class { someMethod; }
Oh, one interesting idea got in my head, hey @IzzySoft, can you please test this? I try to keep both class and keep it's name..
Commit Hash: 3dcf849d5540cd77080fb3786e2838401e1a25c3 Local Build: local.apk.zip
I always chose one option, but why not try both -keep and -keepnames? If my theory is right, this may be a solution! Haha, thanks for pointing me @obfusk.
IzzySoft
commented
2 months ago can you compare xxd output for the DEX?
pg-map-id alright, as expected. diff -Naur <( unzip -p signed.apk classes.dex | xxd ) <( unzip -p unsigned.apk classes.dex | xxd ) (on the previous APK/commit which I quoted the output from above) gives:
--- /dev/fd/63 2024-08-10 15:19:40.029876752 +0200
+++ /dev/fd/62 2024-08-10 15:19:40.029876752 +0200
@@ -1,5 +1,5 @@
-00000000: 6465 780a 3033 3800 8997 21a5 9a5a 8c8e dex.038...!..Z..
-00000010: d6ba 1562 3190 46fa 114d b9c7 5185 a8be ...b1.F..M..Q...
+00000000: 6465 780a 3033 3800 8996 3504 abdc 1802 dex.038...5.....
+00000010: b85e 1367 7dab 6737 91b3 4d2a 41e8 d8b7 .^.g}.g7..M*A...
00000020: c0c7 7b00 7000 0000 7856 3412 0000 0000 ..{.p...xV4.....
00000030: 0000 0000 e4c6 7b00 0d96 0000 7000 0000 ......{.....p...
00000040: 4433 0000 a458 0200 5c26 0000 b425 0300 D3...X..\&...%..
@@ -444193,8 +444193,8 @@
006c7200: 6522 3a22 7265 6c65 6173 6522 2c22 6861 e":"release","ha
006c7210: 732d 6368 6563 6b73 756d 7322 3a66 616c s-checksums":fal
006c7220: 7365 2c22 6d69 6e2d 6170 6922 3a32 362c se,"min-api":26,
-006c7230: 2270 672d 6d61 702d 6964 223a 2231 6465 "pg-map-id":"1de
-006c7240: 3437 6439 222c 2272 382d 6d6f 6465 223a 47d9","r8-mode":
+006c7230: 2270 672d 6d61 702d 6964 223a 2232 6139 "pg-map-id":"2a9
+006c7240: 6431 3235 222c 2272 382d 6d6f 6465 223a d125","r8-mode":
006c7250: 2266 756c 6c22 2c22 7665 7273 696f 6e22 "full","version"
006c7260: 3a22 382e 332e 3336 227d 0001 c2a0 0001 :"8.3.36"}......
006c7270: cc80 0001 cc81 0001 cc82 0001 cc83 0001 ................so that would be fixable using reproducible-apk-tools/inplace-fix.py --internal --page-size 16 --zipalign fix-pg-map-id "$OUT" 1de47d9 – though I'd rather not go that road: it would require manual intervention on each version. Tried it out of curiosity, though (btw, could I simpy have appended --zipalign fix-pg-map-id "$OUT" 1de47d9 to the existing reproducible-apk-tools/inplace-fix.py --internal --page-size 16 fix-newlines "$OUT" 'META-INF/services/*' 'res/M7.json' to run it as one command?). Funnily, all I get there is an error: fix-pg-map-id.py: error: unrecognized arguments: 1de47d9 – though you use that with org.fossify.musicplayer successfully @obfusk (I just added the --page-size 16. Error message reads:
+ reproducible-apk-tools/inplace-fix.py --internal --page-size 16 --zipalign fix-pg-map-id app/build/outputs/apk/release/app-release-unsigned.apk app/build/outputs/apk/release/app-fixed.apk 1de47d9
usage: fix-pg-map-id.py [-h] INPUT_DIR_OR_APK OUTPUT_DIR_OR_APK PG_MAP_ID
fix-pg-map-id.py: error: unrecognized arguments: 1de47d9
Error: /usr/bin/python3 command failed.
[RUN] python3 fix-pg-map-id.py app/build/outputs/apk/release/app-release-unsigned.apk /tmp/tmp_3ihvk7j/fixed.apk app/build/outputs/apk/release/app-fixed.apk 1de47d9Looks like something is adding another parameter (/tmp/tmp_3ihvk7j/fixed.apk) in between there. Your fossify example uses v0.2.5 of reproducible-apk-tools, mine v0.2.8. Did something change there?
Now going for your "hopeful build", @Acclorite ("try to keep both class and keep it's name"). Unfortunately, not RB:
"upstream_signed_apk_sha256": "d8d78661f5ff7c5fdcf2cb10071c3f554515fd0ceca4422100a52fc84d3a392d",
"built_unsigned_apk_sha256": "c09f33485c12450654a4b77863f8c8d358fd2cd736a4de1d8cce9dc9503648bd",
"signature_copied_apk_sha256": "4fa2096688e0586e2040ac11732f301f6f2a5aea96d0a340c46a487fe09dc8d2"Diff looks familiar:
-------------------------------
--- /dev/fd/63 2024-08-10 16:31:16.192805037 +0200
+++ /dev/fd/62 2024-08-10 16:31:16.192805037 +0200
@@ -3,11 +3,11 @@
META-INF/version-control-info.textproto
32-bit CRC value (hex): 1de73911
assets/dexopt/baseline.prof
- 32-bit CRC value (hex): 9b21c307
+ 32-bit CRC value (hex): b35e8e9a
assets/dexopt/baseline.profm
32-bit CRC value (hex): 5732a882
classes.dex
- 32-bit CRC value (hex): 3e6565d7
+ 32-bit CRC value (hex): 94996e33
lib/arm64-v8a/libandroidx.graphics.path.so
32-bit CRC value (hex): 9734baa0
lib/arm64-v8a/libdatastore_shared_counter.soSo does the Dex diff:
--- /tmp/tmp.XS0F3bwqRh 2024-08-10 16:32:16.664397902 +0200
+++ /tmp/tmp.3Nn1fLfAMz 2024-08-10 16:32:17.356393243 +0200
@@ -1,9 +1,9 @@
-Processing '/tmp/tmp.1bAAO5Y4oN'...
-Opened '/tmp/tmp.1bAAO5Y4oN', DEX version '038'
+Processing '/tmp/tmp.9fRLFivrs8'...
+Opened '/tmp/tmp.9fRLFivrs8', DEX version '038'
DEX file header:
magic : 'dex\n038\0'
-checksum : a5219789
-signature : 9a5a...a8be
+checksum : 04359689
+signature : abdc...d8b7
file_size : 8112064
header_size : 112
link_size : 0And yes, pg-map-id again:
--- /dev/fd/63 2024-08-10 16:33:30.327901974 +0200
+++ /dev/fd/62 2024-08-10 16:33:30.327901974 +0200
@@ -1,5 +1,5 @@
-00000000: 6465 780a 3033 3800 8997 21a5 9a5a 8c8e dex.038...!..Z..
-00000010: d6ba 1562 3190 46fa 114d b9c7 5185 a8be ...b1.F..M..Q...
+00000000: 6465 780a 3033 3800 8996 3504 abdc 1802 dex.038...5.....
+00000010: b85e 1367 7dab 6737 91b3 4d2a 41e8 d8b7 .^.g}.g7..M*A...
00000020: c0c7 7b00 7000 0000 7856 3412 0000 0000 ..{.p...xV4.....
00000030: 0000 0000 e4c6 7b00 0d96 0000 7000 0000 ......{.....p...
00000040: 4433 0000 a458 0200 5c26 0000 b425 0300 D3...X..\&...%..
@@ -444193,8 +444193,8 @@
006c7200: 6522 3a22 7265 6c65 6173 6522 2c22 6861 e":"release","ha
006c7210: 732d 6368 6563 6b73 756d 7322 3a66 616c s-checksums":fal
006c7220: 7365 2c22 6d69 6e2d 6170 6922 3a32 362c se,"min-api":26,
-006c7230: 2270 672d 6d61 702d 6964 223a 2231 6465 "pg-map-id":"1de
-006c7240: 3437 6439 222c 2272 382d 6d6f 6465 223a 47d9","r8-mode":
+006c7230: 2270 672d 6d61 702d 6964 223a 2232 6139 "pg-map-id":"2a9
+006c7240: 6431 3235 222c 2272 382d 6d6f 6465 223a d125","r8-mode":
006c7250: 2266 756c 6c22 2c22 7665 7273 696f 6e22 "full","version"
006c7260: 3a22 382e 332e 3336 227d 0001 c2a0 0001 :"8.3.36"}......
006c7270: cc80 0001 cc81 0001 cc82 0001 cc83 0001 ................Eh.. Seems like will need some manual work ;( Gladly I have saved all PNGs alongside SVGs ;) Sadly now I need to trim them all ;(
Acclorite
commented
2 months ago @IzzySoft can you provide APK? I will try to see, maybe will figure something..
obfusk
commented
2 months ago could I simpy have appended
No. inplace-fix can only run one subcommand at a time. Just like you can't combine git clone and git merge either :P. Separating which argument/option is for inplace-fix and which is for the subcommand is hard enough already, and many subcommands take a variable number of arguments.
reproducible-apk-tools/inplace-fix.py --internal --page-size 16 --zipalign fix-pg-map-id app/build/outputs/apk/release/app-release-unsigned.apk app/build/outputs/apk/release/app-fixed.apk 1de47d9
The whole point of using inplace-fix is that it fixes and zipaligns in-place, without having a separate input and output APK. Which is why you see it use a temporary output file -- /tmp/tmp_3ihvk7j/fixed.apk -- when it runs the underlying script (all of which expect an input and output APK), that it will then replace the original file with. But you're providing an output APK -- app-fixed.apk -- so it gets too many arguments.
FYI: --page-size 16 implies --zipalign as specifying a page size for alignment not performed wouldn't make sense and it makes the command a bit shorter :)
obfusk
commented
2 months ago @Acclorite thanks for the explanation. R8 causing RB issues is not that unexpected, though usually some kind of bug is involved (e.g. desugaring using a nondeterministic order). Most apps we build do not seem to have the R8 issues you run into, though I'm not sure why. For example Neo Store is RB with minification enabled and various proguard rules: https://github.com/NeoApplications/Neo-Store/blob/4f30e6b9d7d42fb6a16d547260ca49377541f967/proguard.pro
But I'm not really an Android dev, and unfamiliar with the specifics of writing proguard rules and what effects they have, just some basic knowledge.
The pg-map-id difference is odd. I think it's a bug in the toolchain. But without a mapping file to compare I can't really tell much or report it to Google. And all my builds with rbtlog always have the same id for the same source. I do know that e.g. Element X uses my tooling to use a fixed all-zeroes pg-map-id in both builds to make it RB. But no one knows why it's different (and few people even know what the pg-map-id is, F-Droid admits they just use my tooling without understanding what it does).
Acclorite
commented
2 months ago For example Neo Store is RB with minification enabled and various proguard rules: https://github.com/NeoApplications/Neo-Store/blob/4f30e6b9d7d42fb6a16d547260ca49377541f967/proguard.pro
I think the most important line there is -dontobfuscate. This probably solves issue. From the wiki
Specifies not to obfuscate the input class files. By default, ProGuard obfuscates the code: it assigns new short random names to classes and class members. It removes internal attributes that are only useful for debugging, such as source files names, variable names, and line numbers.
Basically, issues I run into are all about class naming, the classes itself in most cases are similar, just have a bit different names(like b.0.smali - B.0.smali).
I use R8 Full Mode and Obfuscation, which reduces APK size + Increases performance. The main problem is in obfuscation process I believe, when giving random short names, it generated not deterministic results. And the only cure I found is do not interfere inside and try to keep classes, only keeping names may work. Again from wiki:
-keepSpecifies classes and class members (fields and methods) to be preserved as entry points to your code. For example, in order to keep an application, you can specify the main class along with its main method. In order to process a library, you should specify all publicly accessible elements.
while
-keepnamesShort for -keep,allowshrinking class_specification Specifies classes and class members whose names are to be preserved, if they aren't removed in the shrinking phase. For example, you may want to keep all class names of classes that implement the Serializable interface, so that the processed code remains compatible with any originally serialized classes. Classes that aren't used at all can still be removed. Only applicable when obfuscating.
My guess is at shrinking/obfuscating stage that kept class gets in the way, causing not deterministic result. That's about how I see it..
Edit: I'd say, most apps do not use R8 to the fullest, resulting in less problems, I said it already, but I am a completionist, if I have a tool, I will use it to it's maximum.
I've checked your app if its build is reproducible (see: Reproducible bulds, special client support and more in our repo), but while I was able to successfully generate the APK using
./gradlew assembleRelease, the differences to the one provided at your latest release were huge. Was that APK really built from the commit the tag points to? If so, did I miss some build options? And if not, which commit was it?APK diff:
We'd appreciate if you could help making your build reproducible. We've prepared some hints on reproducible builds for that.
Looking forward to your reply!
PS: If you could insert an ampty line before the bullet point list in your fastlane's
full_description.txt, it would perfectly render as Markdown :wink: If you wonder for the background of this question, I've just added your app to the IzzyOnDroid repo, where it will show up with the next sync in about an hour. So be also welcome to pick a badge to link there e.g. from your Readme if you wish :smile:And last but not least:
This blob is easy to avoid by a simple change to your
build.gradle.kts:For some background: that BLOB is supposed to be just a binary representation of your app's dependency tree. But as it's encrypted with a public key belonging to Google, only Google can read it – and nobody else can even verify what it really contains. More details can be found e.g. here: Ramping up security: additional APK checks are in place with the IzzyOnDroid repo.
Thanks in advance!