apksigner from build-tools >= 35.0.0-rc1 requires workaround

[licaon-kter]

Suddenly for the past week more repro APKs can't be verified.

eg. https://github.com/Futsch1/medTimer/releases/tag/v1.8.10 has unsigned and signed

They have in common a thing, they get built by the Github CI, eg. signed here with apksigner from build-tools 35: https://github.com/Futsch1/medTimer/actions/runs/9750431791/job/26909848960#step:16:5

CI recipes use ubuntu-latest eg. https://github.com/Futsch1/medTimer/blob/v1.8.10/.github/workflows/android.yml#L14 so maybe the Github image got updated to build-tools 35 and now CI's just end up using it by default

apksigcopier can't cope with whatever apksigner 35 does?

[obfusk]

Third edit

I now added an FAQ entry, copied here:

What about signatures made by apksigner from build-tools >= 35.0.0-rc1?

Since build-tools >= 35.0.0-rc1, backwards-incompatible changes to apksigner break apksigcopier as it now by default forcibly replaces existing alignment padding and changed the default page alignment from 4k to 16k (same as Android Gradle Plugin >= 8.3, so the latter is only an issue when using older AGP).

Unlike zipalign and Android Gradle Plugin, which use zero padding, apksigner uses a 0xd935 "Android ZIP Alignment Extra Field" which stores the alignment itself plus zero padding and is thus always at least 6 bytes.

It now forcibly replaces existing padding even when the file is already aligned as it should be, except when --alignment-preserved is specified, in which case it will keep existing (non)alignment and padding.

This means it will replace existing zero padding with different padding for each and every non-compressed file. This padding will not only be different but also longer for regular files aligned to 4 bytes with zero padding, but often the same size for .so shared objects aligned to 16k (unless they happened to require less than 6 bytes of zero padding before).

Unfortunately, supporting this change in apksigcopier without breaking compatibility with the signatures currently supported would require rather significant changes. Luckily, there are 3 workarounds available:

First: use apksigner from build-tools <= 34.0.0 (clearly not ideal).

Second: use apksigner sign from build-tools >= 35.0.0-rc1 with the --alignment-preserved option.

Third: use zipalign.py --page-size 16 --pad-like-apksigner --replace on the unsigned APK to replace the padding the same way apksigner now does before using apksigcopier.

---

Original reply

Click "Details" to expand.

Indeed, Google broke `apksigcopier` by making backwards-incompatible changes to `apksigner` that add garbage to the ZIP metadata for absolutely no reason (and GHA added `build-tools` 35.0.0 to the pre-installed tooling in their Ubuntu images last week). This is a known issue. Feel free to report it to Google though. Maybe they'll fix it. They did fix the `zipflinger`-added garbage eventually. However, supporting that breaking change in `apksigcopier` without breaking backwards compatibility for older (and non-`apksigner`) signatures (or requiring the user to manually specify which version (and options) of `apksigner` the copying should be compatible with -- which would essentially be the same as the last alternative below, just combined into a single tool) would require significant changes that would require either unreliable autodetection or copying a lot of metadata instead of just the signature. As `apksigcopier` should be as simple as possible, not perform any autodetection unless it is reliable, and only copy signatures, nothing else, this is considered out of scope and thus currently not planned unless it is absolutely necessary to make signature copying work (which it isn't). Luckily, there are already 3 alternatives to changing `apksigcopier` to make it somehow add the same garbage when needed: - use `apksigner` from `build-tools` <= 34.0.0 for signing (not ideal) - use `apksigner` from `build-tools` >= 35.0.0-rc1 for signing and tell it to not add the garbage - use the available tooling to add the same garbage to the unsigned APK before using `apksigcopier`

---

First edit

- "tell it to not add the garbage" means using `apksigner sign` with the `--alignment-preserved` option - "use the available tooling to add the same garbage" means using `zipalign.py --page-size 16 --pad-like-apksigner --replace` on the unsigned APK

---

Second edit

As for the "garbage": unlike `zipalign` and AGP, which use zero padding, `apksigner` uses a `0xd935` "Android ZIP Alignment Extra Field" which stores the alignment itself plus zero padding and is thus always at least 6 bytes. Since it now forcibly replaces existing padding even when the file is already aligned (unless `--alignment-preserved` is specified in which case it will still align files that aren't but keep existing padding for already aligned files) this means it will replace existing zero padding with different padding (which will be longer for regular files aligned to 4 bytes, but often the same size for `.so` shared objects aligned to 16k unless they happened to require less than 6 bytes of padding before) for each and every non-compressed file.

[obfusk]

The third alternative could be automated: if copying the signature fails, add the garbage and retry. However, this is still out of scope for apksigcopier, which only copies the signature; validation of the resulting APK is left up to the user -- except for the compare command which is provided merely for convenience, the copy and patch commands do not and cannot validate the signature (and should not require apksigner to work) -- or whatever program is using the apksigcopier API.

[obfusk]

As I pointed out elsewhere and not for the first time:

I worry that there isn't really a sustainable solution for F-Droid running a Reproducible Builds project when they no longer have an expert to fix things when they break or anyone having a proper understanding of all the tooling they depend on.

If you don't understand what changes tooling like zipalign, apksigner, repro-apk, etc. make to the APK, you will not be able to fix anything if it breaks because something changed in the toolchain like here. Except by downgrading or getting lucky with trial and error of various options you happen to know about. Hoping someone else fixes it for you seems like a bad strategy :woman_shrugging:

But I resigned from F-Droid last year so it's not my problem any more 🤷‍♀️

[licaon-kter]

This is a known issue.

Link?

Use apksigner from build-tools >= 35.0.0-rc1 for signing and tell it to not add the garbage

Ah, but 'tis a sekret, wink-wink, nudge-nudge, say no more

diff between `./34.0.0/apksigner sign --help` and `./35.0.0-rc3/apksigner sign --help` ``` > --alignment-preserved Whether the existing alignment within the APK should > be preserved; the default for this setting is false. > When this value is false, the value provided to > --lib-page-alignment will be used to page align native > library files and all other files will be aligned to 4 > bytes in the APK. > > --lib-page-alignment The number of bytes to be used to page align native > library files in the APK; the default value is 16384. ``` so let's try this: ``` $ /opt/android-sdk/build-tools/35.0.0-rc3/apksigner sign --ks keystore.p12 --ks-key-alias mine --in unsigned/com.futsch1.medtimer_57.apk --out com.futsch1.medtimer_57_35_normal.apk $ apksigcopier compare com.futsch1.medtimer_57_35_normal.apk --unsigned com.futsch1.medtimer_57_unsigned.apk && echo OK DOES NOT VERIFY ERROR: APK Signature Scheme v3 signer #1: APK integrity check failed. CHUNKED_SHA512 digest mismatch. Expected: <587fa6f810a0e009957fcf8dbb0d49d58949dbc927f14c6a2d101cafff3e9dac4253a31e9d438cfb55b87a7735e5c9e6dfbd796f7915553b962c4df17a466641>, actual: <534460755f6c5dd8282ef8895d93869a79d72bf3413e884464ae942cde96767a3e5fb2e6ae085ce5088fba858b831bb9641bf4390d76536855b9ef366ad89edd> Error: failed to verify /tmp/tmpaikxftsz/output.apk. ``` As expected :disappointed: Now: ``` $ /opt/android-sdk/build-tools/35.0.0-rc3/apksigner sign --ks keystore.p12 --ks-key-alias mine --alignment-preserved true --in unsigned/com.futsch1.medtimer_57.apk --out com.futsch1.medtimer_57_35_keepalign.apk $ apksigcopier compare com.futsch1.medtimer_57_35_keepalign.apk --unsigned com.futsch1.medtimer_57_unsigned.apk && echo OK OK ``` Sums? ``` $ sha256sum com.futsch1.medtimer_57_* 43a37d3f06bbe9a82f13d613c99cc0b3a58b410fcf8b5d3524748c22a52b6a77 com.futsch1.medtimer_57_34.apk 43a37d3f06bbe9a82f13d613c99cc0b3a58b410fcf8b5d3524748c22a52b6a77 com.futsch1.medtimer_57_35_keepalign.apk cd0707edc6db09e817c042fa285f76a4dcd7e738cfabda34f5d84572755fa3b2 com.futsch1.medtimer_57_34.apk.idsig cd0707edc6db09e817c042fa285f76a4dcd7e738cfabda34f5d84572755fa3b2 com.futsch1.medtimer_57_35_keepalign.apk.idsig 79c5c0c755d2efc70447a91c42fc705720386ec3963edf140b772231823b7ea7 com.futsch1.medtimer_57_35_normal.apk.idsig d3da4a143f007a124f0195c13149d44e157c810cf750b30f1c15cc6554451bd3 com.futsch1.medtimer_57_35_normal.apk ```

:tada:

getting lucky with trial and error of various options you happen to know about

How do you think I got so far? :cat:

[obfusk]

Link?

"Known issue" as in "I knew about it but just didn't get around to updating the apksigcopier FAQ yet". I told you to "feel free to report it to Google" if you want them to fix apksigner.

Don't forget, apksigcopier may be a vital dependency for F-Droid, but to me it's just an unpaid hobby project I work on whenever I have time and feel like it. After my resignation, Hans' behaviour got him permanently banned from all of my projects; the rest of F-Droid should expect no support from me either. The only reason you got a detailed reply here is that I was already aware of this and planning to document it. Had I needed to investigate this just for F-Droid, you would have been out of luck.

Ah, but 'tis a sekret, wink-wink, nudge-nudge, say no more [it works with --alignment-preserved true, as expected] 🎉

What? I just didn't remember the option name and didn't bother looking it up just for you, knowing you couldn't possibly fail to find it, as you clearly just demonstrated.

How do you think I got so far?

If you didn't need my help, why open this issue to ask for it?

[obfusk]

@licaon-kter @linsui I don't care about credit here. I'm just glad this can be fixed for the developers that were affected. But given that LK is now taking credit for figuring out what changed: do not ask me for help again next time. You say you don't need my help, fine. Maybe you don't. Maybe your luck won't run out. But then you don't get to ask for my help either. You're on your own from now on. My issue trackers are closed to F-Droid now. Good luck next time.

[obfusk]

Reported to Google: https://issuetracker.google.com/issues/351408623

[obfusk]

If anyone wants Google to fix this on their end and has a Google account, please consider adding a +1 to the issue.