Open IzzySoft opened 1 month ago
I believe i built the apk before posting the commit. So that could be the case. I have some changes pending. I'll commit those and build an apk for you to compare.
I believe i built the apk before posting the commit.
Which is one way to guarantee it's not RB, yeah – see the linked hints, §1 :stuck_out_tongue_winking_eye:
I have some changes pending. I'll commit those and build an apk for you to compare.
Will that be a new release then? Because only releases are relevant. I could of course test-run it first to see if it is RB – just in case there are other culprits, to ensure we cover them all and have a fine RB release then. For that just ping me with an APK and the information which commit it was built from. Thanks!
Alright, next release it is. Will let you know
Try this apk using the latest commit https://we.tl/t-KWOsKfVlOs
Could you please attach the APK here (simply rename it to *.zip
, do not put it inside another zip) and name the commit? "latest" is a "moving target" (and I cannot download from the link you provided, sorry).
Cant send it here because the max limit is 25mb. What other method do you prefer
Maybe temporarily attach it to the release (with a "test name"), I'll let you know once I've grabbed it so you can remove it again? Don't forget to name the commit hash I should check against.
ok
Build failed:
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':app:dataBindingMergeDependencyArtifactsRelease'.
> Could not resolve all files for configuration ':app:releaseCompileClasspath'.
> Could not find com.devbrackets.android:exomedia:4.3.0.
Required by:
project :app
> Could not find com.github.TeamNewPipe:NewPipeExtractor:0.24.2.
Required by:
project :app
Are those dependencies in some private repo(s)?
No
Note: this artifact is located at JCenter repository (https://jcenter.bintray.com/)
Well, JCenter closed a couple of weeks ago – and that's where com.devbrackets.android:exomedia:4.3.0
was located. If you'd switch to a newer version: 5.x is available at Maven Central.
NewPipeExtractor seems to be available at Jitpack, which you have enabled, so I wonder why that is not found. Can you check, please – and fix those dependencies which no longer resolve?
I'll have a look at it later today
I switched to exomedia 5.1.0 and that built fine but cant reproduce the newpipextractor issue.
What error is popping up?
The one shown above. If you give me a commit hash and its matching APK, I can re-check if you want.
@IzzySoft the test release was somewhat of a disaster. People still downloaded it and threw me errors 😁. Need to find another way to give you the apk. Like there isnt any other file hosting service i can use?
Guess if you name the release "THIS WILL EAT YOUR CAT!!!" they'd still try it out, huh? :see_no_evil: Are you using Github Actions to build? I see there's a workflow defined. If so, you could link me to the artifact maybe.
As for file hosting services: haven't use any in ages, so I cannot tell. The one you used last time wanted me to agree to their TOS and all, and that's a bit much for just downloading a test file…
Ppl use obtainium and that will grab new releases no matter what he writes in the release or filename
that cannot be helped then. Can Obtainium not be told to ignore pre-releases? Oof…
@IzzySoft i managed to get the github action build to succeed. I just now need to find a way to create apks right in github. I guess that would be the best course of action.
Sounds good! :crossed_fingers: Doing it via Github CI ensures builds from a clean tree. Dirty trees are the main reason of RB fails, so this sounds good :smiley:
@IzzySoft the latest action build generated the apks you can check out and see. I guess in the following releases, ill just grab the generated apks from github and use them in the release so we can have transparency.
Also about that newpipe error, somehow gradle wanted the company name to be in lowercase to work. 😁
the latest action build generated the apks you can check out and see.
Now I'll need to figure how to find that artifact and how to match it to which commit it was built from… OK, the latter is clear. The former not: there's only a 250+ MB ZIP file… Eek, and not even downloadable via wget
or curl
as the link is not direct to the zip… Guess you'll add downloading artifacts later then?
OK, managed to download the full ZIP, extract the APK, and running the RB check. But it's not RB:
-rw-r--r-- 0.0 unx 120 b- 118 defN 1981-01-01 01:01:02 16e72a9e META-INF/version-control-info.textproto
- -rw-r--r-- 0.0 unx 5125 b- 5125 stor 1981-01-01 01:01:02 e6456266 assets/dexopt/baseline.prof
+ -rw-r--r-- 0.0 unx 5125 b- 5125 stor 1981-01-01 01:01:02 0f8bf6e7 assets/dexopt/baseline.prof
-rw-r--r-- 0.0 unx 396 b- 396 stor 1981-01-01 01:01:02 48b4a4cc assets/dexopt/baseline.profm
- -rw-r--r-- 0.0 unx 10043204 b- 4354747 defN 1981-01-01 01:01:02 f3c4df5a classes.dex
+ -rw-r--r-- 0.0 unx 10042968 b- 4354666 defN 1981-01-01 01:01:02 6fd96133 classes.dex
-rw-r--r-- 0.0 unx 631400 b- 254102 defN 1981-01-01 01:01:02 52f51e23 classes2.dex
-rw-r--r-- 0.0 unx 2362568 b- 852191 defN 1981-01-01 01:01:02 2a243272 lib/arm64-v8a/libaria2c.so
-rw-r--r-- 0.0 unx 1672335 b- 1665102 defN 1981-01-01 01:01:02 648a6fac lib/arm64-v8a/libaria2c.zip.so
@@ -1667,7 +1667,7 @@
-rw---- 0.0 fat 884 b- 884 stor 1981-01-01 01:01:02 0ef1b92c res/zz.png
-rw---- 0.0 fat 532 b- 205 defN 1981-01-01 01:01:02 a50d3889 res/zz.xml
-rw---- 0.0 fat 2922860 b- 2922860 stor 1981-01-01 01:01:02 7f51ca9f resources.arsc
- -rw-r--r-- 0.0 unx 150110 b- 68236 defN 1981-01-01 01:01:02 b3cdd6b7 META-INF/CERT.SF
- -rw-r--r-- 0.0 unx 1375 b- 1102 defN 1981-01-01 01:01:02 e1a8e130 META-INF/CERT.RSA
- -rw-r--r-- 0.0 unx 150036 b- 64662 defN 1981-01-01 01:01:02 413b0dcc META-INF/MANIFEST.MF
- 1670 files, 46855332 bytes uncompressed, 37623193 bytes compressed: 19.7%
+ -rw-r--r-- 0.0 unx 150110 b- 68235 defN 1981-01-01 01:01:02 621c0082 META-INF/CERT.SF
+ -rw-r--r-- 0.0 unx 1167 b- 1021 defN 1981-01-01 01:01:02 be28c0d6 META-INF/CERT.RSA
+ -rw-r--r-- 0.0 unx 150036 b- 64657 defN 1981-01-01 01:01:02 0d450b76 META-INF/MANIFEST.MF
Next to the classes.dex
, I wonder how comes that my local build ends up to be signed?
Signer #1 certificate DN: C=US, O=Android, CN=Android Debug
Signer #1 certificate SHA-256 digest: 66bddff52ca3d64700b8f0793d79efbc5634b7668740683ce3be18b48a457f7e
Signer #1 certificate SHA-1 digest: eb702217e298990e41c2cb25815f027ab0c4723f
Signer #1 certificate MD5 digest: c2e582e8e99d644d5ef7db02d7774034
Signer #1 key algorithm: RSA
Signer #1 key size (bits): 2048
Hmpf. Guess I'll need to update my recipe and patch out some more signing stuff from build.gradle
. Looking at the dex diff it's clear what's missing:
- #5 : (in Lcom/deniscerri/ytdl/BuildConfig;)
- name : 'signingConfigkeyAlias'
- type : 'Ljava/lang/String;'
- access : 0x0019 (PUBLIC STATIC FINAL)
- value : "S4n1t1z3D"
- #6 : (in Lcom/deniscerri/ytdl/BuildConfig;)
- name : 'signingConfigkeyPassword'
- type : 'Ljava/lang/String;'
- access : 0x0019 (PUBLIC STATIC FINAL)
- value : "S4n1t1z3D"
- #7 : (in Lcom/deniscerri/ytdl/BuildConfig;)
- name : 'signingConfigstoreFile'
- type : 'Ljava/lang/String;'
- access : 0x0019 (PUBLIC STATIC FINAL)
- value : "release_keystore.jks"
- #8 : (in Lcom/deniscerri/ytdl/BuildConfig;)
- name : 'signingConfigstorePassword'
- type : 'Ljava/lang/String;'
- access : 0x0019 (PUBLIC STATIC FINAL)
- value : "S4n1t1z3D"
Your artifact's APK contains the build config's signing data? :eyes: :scream: Guess it's time for a new certificate then (in the diff, I've sanitized the values except for the keystore file). Was it that way with previous releases as well? (checking) oof, unfortunately yes – so we must consider your signing key compromised, and you need to change to a new one. You could use Signing Key Rotation, but that will only help devices running Android-9 or later (so 6-8 would need an uninstall/reinstall).
Why new certificate? How can you even figure out the signing data? Am i not supposed to sign the app?
Or does the github action configuration not build the apk properly?
Look above in the diff: there's the keystore, the alias and the passwords. What else do I need to properly sign an APK in your name? Do I miss something? Ah, maybe the keystore is just mentioned but luckily was not checked in. (checking) OK, it's generated at build time from a secret, oof. OK, then maybe no need to change the key itself. But the passwords at least, and make sure they're not ending up in the APK anymore then. Luckily at least the keystore itself didn't end up in the APK. So if you consider its storage safe enough (and thus the risk of someone obtaining it low enough), maybe that should be sufficient.
And that com/deniscerri/ytdl/BuildConfig
should not be part of the APK (else we cannot achieve RB – apart from the fact of your passwords being contained there in cleartext being a very bad idea).
The confusing part is that the passwords are also stored in secrets. But i create the local.properties file in the github actions to build the apk. Did you get the password from that? Did the file not delete after the build?
Can you guide me how to resolve this?
I think buildconfig gets data from local.properties. If i add the passwords to another .properties file ill be good. Testing that
Try this run please https://github.com/deniscerri/ytdlnis/actions/runs/10654766901
The confusing part is that the passwords are also stored in secrets. But i create the local.properties file in the github actions to build the apk. Did you get the password from that? Did the file not delete after the build?
I get the passwords when using Apktool to unpack it and generate the Smali code from it. They are stored in a class named com/deniscerri/ytdl/BuildConfig
, as shown by the above Dex diff. I've never seen such before in any of the apps I've analyzed. You can find the corresponding code references here.
I think buildconfig gets data from local.properties. If i add the passwords to another .properties file ill be good. Testing that
Not sure, but I'm no Android dev. I've seen other projects using local.properties
for keystore stuff, but it didn't end up this way.
Try this run please
OK, will do… BuildConfig is still there, but the sensitive information is gone from it. Now let's see what the verification builder say: not RB. Hm. APK diff:
-rw-r--r-- 0.0 unx 120 b- 118 defN 1981-01-01 01:01:02 1aac3f57 META-INF/version-control-info.textproto
- -rw-r--r-- 0.0 unx 5104 b- 5104 stor 1981-01-01 01:01:02 430965b1 assets/dexopt/baseline.prof
+ -rw-r--r-- 0.0 unx 5105 b- 5105 stor 1981-01-01 01:01:02 232ce725 assets/dexopt/baseline.prof
-rw-r--r-- 0.0 unx 396 b- 396 stor 1981-01-01 01:01:02 baf29ca2 assets/dexopt/baseline.profm
- -rw-r--r-- 0.0 unx 10045520 b- 4355000 defN 1981-01-01 01:01:02 a4800405 classes.dex
+ -rw-r--r-- 0.0 unx 10045520 b- 4354997 defN 1981-01-01 01:01:02 c16457ad classes.dex
-rw-r--r-- 0.0 unx 631400 b- 254102 defN 1981-01-01 01:01:02 52f51e23 classes2.dex
-rw-r--r-- 0.0 unx 2362568 b- 852191 defN 1981-01-01 01:01:02 2a243272 lib/arm64-v8a/libaria2c.so
-rw-r--r-- 0.0 unx 1672335 b- 1665102 defN 1981-01-01 01:01:02 648a6fac lib/arm64-v8a/libaria2c.zip.so
@@ -140,7 +140,7 @@
-rw---- 2.0 fat 25972 b- 7092 defN 1981-01-01 01:01:02 0eee32e3 org/mozilla/javascript/resources/Messages_fr.properties
-rw---- 2.0 fat 2579 b- 1042 defN 1981-01-01 01:01:02 1e1d6243 org/mozilla/javascript/tools/debugger/test.js
-rw---- 2.0 fat 8222 b- 2787 defN 1981-01-01 01:01:02 9bda5daa org/mozilla/javascript/tools/resources/Messages.properties
- -rw---- 0.0 fat 42792 b- 7129 defN 1981-01-01 01:01:02 1d803ecb AndroidManifest.xml
+ -rw---- 0.0 fat 42792 b- 7135 defN 1981-01-01 01:01:02 133a5067 AndroidManifest.xml
-rw---- 0.0 fat 744 b- 344 defN 1981-01-01 01:01:02 1312bbae res/-1.xml
OK, still the Dex. But also the Manifest? (Edit: removed this section as it turned out irrelevant) So let's see the dex diff:
-checksum : 9ddf47b4
-signature : 63dd...7072
+checksum : ea914968
+signature : aa7b...7cc1
file_size : 10045520
header_size : 112
link_size : 0
@@ -409077,7 +409077,7 @@
name : 'VERSION_CODE'
type : 'I'
access : 0x0019 (PUBLIC STATIC FINAL)
- value : 107090100
+ value : 107090104
#4 : (in Lcom/deniscerri/ytdl/BuildConfig;)
Much cleaner. What irritates me there… isn't that the versionCode
? I've picked the arm64 APK out of your artifacts. Then I've adjusted your build.gradle
to have it only build the arm64:
sed -r '/abi \{/,/}/ { s/include.*/include "arm64-v8a"/ ; s/universalApk true/universalApk false/ }' -i app/build.gradle
Why does that result in the arm64 APK getting the "base versionCode
"? I mean, it would be a waste of resources to build 5 APKs if I just need 1 of them, no? Probably caused by this:
def baseAbiVersionCode = project.abiCodes.get(output.getFilter(com.android.build.OutputFile.ABI))
So no fix definition, but looking for the build output. Let me try to confirm (just disabling the UniversalApk but having it build all defined ABIs)… Ah, that looks better: just the dex diff (and hence baseline.prof) remain now. Eh, and the dex diff is just … the versionCode
? Now I'm really curious. Last run, leaving the ABI definitions entirely untouched (i.e. also building the UniversalApk). If that works, maybe you can create a static mapping for the overrides, instead of relying on build output? I'm not even sure if that forEach
is deterministic (if not, this would give us a chance of 4 out of 5 to be wrong when trying to confirm RB).
Oof:
"upstream_signed_apk_sha256": "9c37a627b88d8a2c6ea62344407f826f1b0fbc7c8ecd4c6bb962ab4582e79289",
"built_unsigned_apk_sha256": "a63e4872d64844b76b1ef9a1e1c00f6588b0a0bb91000c41cf390912a94ac4e8",
"signature_copied_apk_sha256": "9c37a627b88d8a2c6ea62344407f826f1b0fbc7c8ecd4c6bb962ab4582e79289"
OK, that's RB now – congrats! Where do we want to go from here? Leave it as-is (building additional 4 APKs we then just throw away) – or try with the mapping?
Well i thought I'd build everything in github action and use those for release. So i never build in my local computer.
Yeah, and you build everything in one action, so there it totally makes sense. The verification builders for IoD as well as per-ABI builds with F-Droid.org would build each ABI separately. While the recipe in my builder would just cover the ABI present at IoD, F-Droid would have 4 build blocks (one for each ABI) and thus build all 5 variants (4x ABI plus the UniversalApk) 4 times.
I'm no Android dev, so it's hard for me to tell how much overhead that might be (as long as it's not pulling additional files for each ABI, it would most likely be some additional local processing concerning the ABI specific stuff). Which is why I raised the question with you. If you say you want to stay with the process as it currently is, I'd add the (now working) recipe to my builder (just adding a note to it for this), and the green shield for confirmed RBs will be established starting with this version – once you've released it. If you say you're willing to give the "mapping" a chance, I happily try another time having the sed
re-established (I'm using that with several other apps successfully, but I cannot tell how many of them, if any, use versionCode
override).
https://github.com/element-hq/element-x-android/pull/2911
Using BuildConfig.VERSION_CODE break the reproducible build since the value will always end with 0, and should end with a digit depending on the architecture.
@obfusk @IzzySoft is there any project i can look over and see how they implemented the build process? thanks
AFAIK Element X won't build reproducibly unless you build multiple ABIs either (because the manifest will differ otherwise). So doing that might be easiest. I suspect it's fixable but the gradle DSL stuff is pretty complex. My guess is there are no ABI filters when there is only one ABI configured and thus the lookup fails. Thus I suspect simply adding a second sed
to replace the ABI code in the lookup as well might work, at least for the manifest (Izzy would have to test to confirm that):
sed -r 's/abiCodes.get\(.*\)/abiCodes.get("arm64-v8a")/' -i app/build.gradle
AFAIK Element X won't build reproducibly unless you build multiple ABIs either
Same for com.tawhid.webcapture
btw.
Izzy would have to test to confirm that
v1.7.9.2 without touching the ABIs (i.e. building all of them) is RB. Now with this recipe (i.e. adding the first 2 lines):
build:
- sed -r '/abi \{/,/}/ { s/include.*/include "arm64-v8a"/ ; s/universalApk true/universalApk false/ }' -i app/build.gradle
- sed -r 's/abiCodes.get\(.*\)/abiCodes.get("arm64-v8a")/' -i app/build.gradle
- sed -r '/signingConfigs.debug/d' -i app/build.gradle
- chmod +x gradlew
- touch local.properties
- ./gradlew assembleRelease
is NOT RB. APK diff:
-rw-r--r-- 0.0 unx 120 b- 118 defN 1981-01-01 01:01:02 73f7a028 META-INF/version-control-info.textproto
- -rw-r--r-- 0.0 unx 5106 b- 5106 stor 1981-01-01 01:01:02 505830c3 assets/dexopt/baseline.prof
+ -rw-r--r-- 0.0 unx 5106 b- 5106 stor 1981-01-01 01:01:02 104ca8c8 assets/dexopt/baseline.prof
-rw-r--r-- 0.0 unx 396 b- 396 stor 1981-01-01 01:01:02 48b4a4cc assets/dexopt/baseline.profm
- -rw-r--r-- 0.0 unx 10043656 b- 4354933 defN 1981-01-01 01:01:02 54462bfd classes.dex
+ -rw-r--r-- 0.0 unx 10043656 b- 4354933 defN 1981-01-01 01:01:02 d87fbd41 classes.dex
-rw-r--r-- 0.0 unx 631400 b- 254102 defN 1981-01-01 01:01:02 52f51e23 classes2.dex
-rw-r--r-- 0.0 unx 2362568 b- 852191 defN 1981-01-01 01:01:02 2a243272 lib/arm64-v8a/libaria2c.so
-rw-r--r-- 0.0 unx 1672335 b- 1665102 defN 1981-01-01 01:01:02 648a6fac lib/arm64-v8a/libaria2c.zip.so
@@ -140,7 +140,7 @@
-rw---- 2.0 fat 25972 b- 7092 defN 1981-01-01 01:01:02 0eee32e3 org/mozilla/javascript/resources/Messages_fr.properties
-rw---- 2.0 fat 2579 b- 1042 defN 1981-01-01 01:01:02 1e1d6243 org/mozilla/javascript/tools/debugger/test.js
-rw---- 2.0 fat 8222 b- 2787 defN 1981-01-01 01:01:02 9bda5daa org/mozilla/javascript/tools/resources/Messages.properties
- -rw---- 0.0 fat 42792 b- 7129 defN 1981-01-01 01:01:02 11b141f1 AndroidManifest.xml
+ -rw---- 0.0 fat 42792 b- 7133 defN 1981-01-01 01:01:02 1f0b2f5d AndroidManifest.xml
-rw---- 0.0 fat 744 b- 344 defN 1981-01-01 01:01:02 9d59dbe0 res/-1.xml
Manifest:
E: activity-alias (line=68)
- A: http://schemas.android.com/apk/res/android:icon(0x01010002)=@0x7f100000
- A: http://schemas.android.com/apk/res/android:name(0x01010003)="com.deniscerri.ytdl.Default" (Raw: "com.deniscerri.ytdl.Default")
- A: http://schemas.android.com/apk/res/android:enabled(0x0101000e)=true
+ A: http://schemas.android.com/apk/res/android:icon(0x01010002)=@0x7f100002
+ A: http://schemas.android.com/apk/res/android:name(0x01010003)="com.deniscerri.ytdl.LightIcon" (Raw: "com.deniscerri.ytdl.LightIcon")
+ A: http://schemas.android.com/apk/res/android:enabled(0x0101000e)=false
A: http://schemas.android.com/apk/res/android:exported(0x01010010)=true
A: http://schemas.android.com/apk/res/android:launchMode(0x0101001d)=2
A: http://schemas.android.com/apk/res/android:configChanges(0x0101001f)=0x00000004
A: http://schemas.android.com/apk/res/android:targetActivity(0x01010202)="com.deniscerri.ytdl.MainActivity" (Raw: "com.deniscerri.ytdl.MainActivity")
A: http://schemas.android.com/apk/res/android:windowSoftInputMode(0x0101022b)=0x00000020
- A: http://schemas.android.com/apk/res/android:roundIcon(0x0101052c)=@0x7f100003
+ A: http://schemas.android.com/apk/res/android:roundIcon(0x0101052c)=@0x7f100005
E: meta-data (line=78)
A: http://schemas.android.com/apk/res/android:name(0x01010003)="android.app.shortcuts" (Raw: "android.app.shortcuts")
A: http://schemas.android.com/apk/res/android:resource(0x01010025)=@0x7f170008
@@ -127,15 +127,15 @@
E: data (line=119)
A: http://schemas.android.com/apk/res/android:mimeType(0x01010026)="application/txt" (Raw: "application/txt")
E: activity-alias (line=122)
- A: http://schemas.android.com/apk/res/android:icon(0x01010002)=@0x7f100002
- A: http://schemas.android.com/apk/res/android:name(0x01010003)="com.deniscerri.ytdl.LightIcon" (Raw: "com.deniscerri.ytdl.LightIcon")
- A: http://schemas.android.com/apk/res/android:enabled(0x0101000e)=false
+ A: http://schemas.android.com/apk/res/android:icon(0x01010002)=@0x7f100000
+ A: http://schemas.android.com/apk/res/android:name(0x01010003)="com.deniscerri.ytdl.Default" (Raw: "com.deniscerri.ytdl.Default")
+ A: http://schemas.android.com/apk/res/android:enabled(0x0101000e)=true
A: http://schemas.android.com/apk/res/android:exported(0x01010010)=true
A: http://schemas.android.com/apk/res/android:launchMode(0x0101001d)=2
A: http://schemas.android.com/apk/res/android:configChanges(0x0101001f)=0x00000004
A: http://schemas.android.com/apk/res/android:targetActivity(0x01010202)="com.deniscerri.ytdl.MainActivity" (Raw: "com.deniscerri.ytdl.MainActivity")
A: http://schemas.android.com/apk/res/android:windowSoftInputMode(0x0101022b)=0x00000020
- A: http://schemas.android.com/apk/res/android:roundIcon(0x0101052c)=@0x7f100005
+ A: http://schemas.android.com/apk/res/android:roundIcon(0x0101052c)=@0x7f100003
E: meta-data (line=132)
Dex diff:
magic : 'dex\n035\0'
-checksum : 8f9795f3
-signature : 4d8c...e13a
+checksum : ba619438
+signature : 4589...800b
file_size : 10043656
header_size : 112
link_size : 0
@@ -409077,7 +409077,7 @@
name : 'VERSION_CODE'
type : 'I'
access : 0x0019 (PUBLIC STATIC FINAL)
- value : 107090200
+ value : 107090204
#4 : (in Lcom/deniscerri/ytdl/BuildConfig;)
So do we go with building everything to have RB now – or do we wait until we can find out how to properly build a single ABI with RB later?
Let's go with how it is for now
That manifest diff is very odd. I'm thinking this is a bug in the toolchain then. So yeah, made sense to try a few things first, but let's just use what works for now :)
I've checked your app if its build is reproducible (see: Reproducible bulds, special client support and more in our repo), but while I was able to successfully generate the APK using
./gradlew assembleRelease
, the differences to the one provided at your latest release were huge. Was that APK really built from the commit the tag points to? If so, did I miss some build options? And if not, which commit was it?APK Diff:
We'd appreciate if you could help making your build reproducible. We've prepared some hints on reproducible builds for that.
Looking forward to your reply!