Write apksigner

[akavel]

Write/port an apksigner tool. Ideally, should be able to reproduce the results of the build.sh script from the fractalwrench/ApkGolf repository. (ApkGolf seems to use v2 signatures; we'll stay with v1 for now as it seems easier, and supported in pre-7 Android versions.) This should let us create signed, installable .apk files without need for AndroidStudio or JRE. The .apk files seem to be regular .zip archives, with a few special files inside. Files in META-INF/ should be created by apksigner.

[akavel]

I'm porting a Java apksigner project. I'm currently writing the port in Go, to do it faster, as it appears Go has better library support/availability for crypto algorithms needed in the tool. It can be ported again to Nim at some later time (probably will need an openssl wrapper library). Current status:

• [x] META-INF/MANIFEST.MF — SHA1 digests added
• [x] META-INF/CERT.SF — signature file generated
• [x] META-INF/CERT.RSA — digital signature file (PKCS7 format) generated
  • this will be the most difficult step of this tool, and it needs a private key & cert (a "keystore") — sample keys should be taken from the ApkGolf repo
  • one of pkcs7 libraries should probably be used (there's an example in godoc of the packages that seems appropriate for what we need to do)
• [x] copy the rest of the files from the original (non-signed) .apk to the output (signed) .apk
• [x] verify using apksigner verify — passed
• [x] test if a signed .apk works on a real (or emulated) Android phone, using adb — tested on a real phone; created a "hello world" apk in Android Studio, signed it with akavel/apksigner, it installed successfully and started. :smile: :tada: zipalign was not needed; I now believe it's only needed for files which are archived with STORE algorithm, not DEFLATE.

References:

• some docs about JAR manifest files, which seem to be the same as used in .apk files

LATER: Further research shows that we might also need:

• [ ] aapt — for translating AndroidManifest.xml into a special binary format, required before archiving it in .apk file; we might be able to just copy its Apache-licensed C sources from the Android SDK
  • see also: sdklite/aapt
• [ ] zipalign — must be called on the .apk after it is signed

[richard-jansson]

You have some great ideas! Thank you for an orderly and insightful message!!

I have some reactions to it of course! Let me know if you think I'm on to something.

One drawback of having one part in go, is that we have to spend more time on defining the interface. With two languages all the shared data-structures need to be updated on both sides. Whereas if we went nim only we could make a really comfortable configuration.

You are speaking of PCKS7 which I have trouble finding anything usable on. I got more hits on PCKS11 for example in ubuntu's default package manager. So I have to points here.

I've got quite a lot of experience interfacing nim with C! So maybe it's possible to think up something clever!! I've done some go coding as well so if I think I'm dismissive of the language you are gravely mistaken :D

https://christine.website/blog/ffi-ing-golang-from-nim-for-fun-and-profit-2015-12-20

In the link above there are some great news. Apparently it's possible to compile go code to a format that is compatible with the C-calling conventions that NIM support. Actually working with C is quite a breeze actually so I think it's doable!

Probably should've enumerated my questions, but now it's to late! You'll have to keep track yourself!

So my concerns are the following, with more parts there have to be a communication protocol. So in a sense adding one component to a system of one, really yields three components as we have to deal with the interfacing/API:ing between the components. It gets really worse when you add more, according to (n)(n-1) connections between n totally interconnected systems.

Another drawback of invoking a different programming language with all that that entails is that it adds quite a lot of complexity to setting up a development environment.

So a lot of back and forth from my side, but it's hard to straighten out all the questions. My 5 cents are that we ought to rewrite the apk signer in nim.

As far as the missing openssl calls are concerned: I am quite capable and very interested in missing any missing functions. Preferably the interfacing ought to be done towards a C-library.

I sense that you'll object that this is a lot of work. And surely you are right that implementing the full API would be an nearly insurmountable problem. However since we can only implement an interface via using the subset that we need we can do it in quite a short time.

So if you can compile a requirement list of what we need as far as signing is concerned I could create nim bindings for those calls. I've developed quite a nifty method for doing quick and dirty interfacing between nim and C, so don't worry it could be done in no time!!

Hope I didn't miss any questions from your part. Sort of in a hurry so I don't mean to be hasty!! Great work as always!

P.S. I'm not inherently against using a GO backend, so if you feel like it that's fine with me as well!

[richard-jansson]

My God. I haven't had the time to properly understand the apksigner codebase! However looking at apksigner.go it hit me that I've used several of these techniques in nim. Also I've been targeting it in a platform inter-operable manner.

I've added functions like "crypto/sha1" "encoding/base64" or equivalent for a low level C - nim websocket interface! I think we could copy that code simply and be up and running quite quickly.

Are you on a linux or windows environment? The WinAPI for encryption is quite neat actually so I have a small C-nim interface written as a thin overlay over the system. So either linux or windows are easily achievable environments. My guess is that the mac approach would be very similar to the linux implementation almost to the extent that it is negligible!

When you produce an APK, what is your targeting environment. At the moment I don't have access to an android phone!! Which is quite ironic I guess. However my thought is to setup Phoenix OS on a virtual machine. Phonix OS is android compiled for x86 machines. So this is a target where the nim-JNI interface would cause problems. And also it makes testing much convenient as the instructions doesn't have to be translated back and forth between arches!

What setup are you using?

[akavel]

I'll try to reply in full later, as I can't now, but my main thought is that the apksigner tool can be a separate standalone tool IIUC, so there's no need to worry about FFI. (I totally wasn't even expecting to try doing FFI here!) That said, I don't really understand the PKCS7 thing well enough to know what I need from C... that's kinda why I'm prototyping this in Go. It's great that you know so much about interfacing Nim with C, this will be really helpful in porting the tool to Nim. Please don't worry now about this; it appears to be a really small tool, so once we find correct libraries, it should be super simple to eventually migrate to Nim. I think what we need may be one of the i2d_x509 functions from openssl, but I'm not 100% sure yet. I tried wrapping x509.h with nimgen, trying to expand the scope of the nimssl library, but it quickly got out of my control. Also I feel wrapping the i2d_X509_bio may be tricky. So, I went with Go just for the "initial hack/PoC" version, to get it as easily and quickly as possible. I would also prefer to have it in Nim eventually, so that contributing to them doesn't require two completely separate compilers & package managers.

[akavel]

To supplement my previous comment:

• I expect we will first build an non-signed.apk with Nim, then run apksigner, which will be a standalone commandline tool, like this: apksigner -i non-signed.apk -o result-signed.apk -k signing-key.pm8 -c signing-cert.x509.pem. So, no FFI needed at all!
• Anyway, once I finish the Go version of it and confirm it works correctly, we can decide if we want to keep it in Go, and port to Nim only at some later time, or try doing a Nim port/translation immediately. I like Nim's conciseness, so I'm totally for doing the port/translation sooner or later.
• sha1 and base64 are not a big problem here. The bigger one is x509, signing, PKCS7, PKCS8, DER, ASN1. There's a lot of formats to translate between here, so it's easy to make a stupid mistake, and hard to debug it later. It's easier for me to make an initial PoC in Go here.
• As to your other comment: I do have both a Windows computer and a Linux one.
• I do have an Android phone with an ARM processor, but I may still install some Android emulator on one of the PCs anyway, for easier testing.

[akavel]

I tried adding the pkcs7 code into apksigner, but I'm now a bit stuck with regards to how to proceed with testing the result. I will have to think about it a bit more to lay out some course of action for myself. I think I won't be able to postpone installing Android Studio any more now, especially because I might need adb. I also realized there's still one more element to the puzzle, a tool called zipalign, that I overlooked before. If I have to install Android Studio, I'd strongly prefer to do it on Linux using Nix, so that I could then hopefully remove it easily, without leaving a trace on my system. Though if that proves to be too much trouble, I might fall back to installing it with apt-get (if at all possible), and maybe creating a separate temporary user on my system, to be later removed together with any extra cruft that Android Studio might add in the home directory.

[akavel]

:smile: Current version of apksigner successfully passed verification with Android SDK's apksigner verify, on some .apk I had around, using the keys from the fractalwrench/ApkGolf repository:

$ ./apksigner -i unsigned.apk -o signed.apk -k key.pk8 -c key.x509.pem
$ ~/Android/Sdk/build-tools/28.0.3/apksigner verify signed.apk 
DOES NOT VERIFY
ERROR: JAR signer CERT.EC: JAR signature META-INF/CERT.EC uses digest algorithm SHA-1 and signature algorithm SHA-1 with ECDSA which is not supported on API Level(s) 8-17 for which this APK is being verified
$ ~/Android/Sdk/build-tools/28.0.3/apksigner verify -v --min-sdk-version 18 signed.apk 
DOES NOT VERIFY
ERROR: JAR signer CERT.EC: Failed to verify JAR signature META-INF/CERT.EC against META-INF/CERT.SF: java.security.SignatureException: APKs with Signed Attributes broken on platforms with API Level < 19
$ ~/Android/Sdk/build-tools/28.0.3/apksigner verify -v --min-sdk-version 19 signed.apk 
Verifies
Verified using v1 scheme (JAR signing): true
Verified using v2 scheme (APK Signature Scheme v2): false
Number of signers: 1

[akavel]

Woohooo! :D @richard-jansson I managed to build a simple hello-world app with Dali + apksigner! :D The only extra thing I needed from Android SDK was aapt binary + android.jar required by aapt. That's still relatively many extras, but a significant achievement nonetheless! :)

Steps to reproduce, I hope:

$ git clone https://github.com/akavel/apksigner
$ cd apksigner
$ go build
$ cd ..
$ git clone https://github.com/akavel/dali
$ cd dali
$ nim c czak_hello.nim
$ ./czak_hello > classes.dex
$ cp czak_manifest.xml AndroidManifest.xml
$ ~/Android/Sdk/build-tools/28.0.3/aapt p -M AndroidManifest.xml -F foo.apk -I ~/Android/Sdk/platforms/android-28/android.jar
$ zip foo.apk classes.dex
$ wget https://github.com/fractalwrench/ApkGolf/raw/master/key.pk8
$ wget https://github.com/fractalwrench/ApkGolf/raw/master/key.x509.pem
$ ../apksigner/apksigner -i foo.apk -o czak_hello.apk -k key.pk8 -c key.x509.pem

...now, take the czak_hello.apk file, install it on a phone/emulator, and run...
...should show a simple black screen with a "Hello Dali" text in top-left corner...

The files have "czak" in name as they're based on czak/minimal-android-project (the .java file is translated by hand to Dali structures).

And boy, does it compile faaaast!!!! Compared to doing it in Android Studio...