The 0.14.0 release on Google Play cannot be verified

[Giszmo]

This time it looks like you might have an extra line in some code file which I think is generated, so maybe your DAO generator is not deterministic? Anyway, please see here for all details. The diff is big but to my understanding only about debugging.

[abdrasulov]

Hi @Giszmo,

Thanks for your review. Yes, that file is generated one. That is strange since we have several other DAO files. We'll check it.

[Giszmo]

Maybe I can gain your interest to run the WalletScrutiny.com test script to see if it can reproduce your build and ideally to also verify other wallets.

PRs are welcome to make the script more compatible with other systems or to document it. If you want, I can also clean it up and document it.

I hope to get to a point where each release is checked by several re-builders. I have the problem that I am the maintainer of a wallet myself, so my verdict on my own wallet is particularly suspicious but if others approve the verdict with a signature, it's not just me claiming stuff anymore.

The script has some dependencies (apksigner, apktool, git, docker, custom docker images that I want to elimininate since a while, disorderfs for Mycelium) but else is easy to run: ./test.sh someApp.apk. It unpacks the apk, checks the appId and runs the appId specific script. If that fails, it needs adjustment. Adjustments should be submitted as PRs of course.

If I can gain your interest, I would also ask other verifiable wallet projects to do the same for unstoppable, to have at least 2 re-builders per wallet.

[abdrasulov]

Hi @Giszmo. I need to discuss it with a team.

[rafaelekol]

After debugging the issue, I found that difference was caused by automatically generated file RatesDao_Impl.java. To fix this issue we decided to wait for the next release .

[Giszmo]

0.14.1 verified. Thanks.

[rafaelekol]

To use your script you mentioned we need to have dependencies (apksigner, apktool, git, docker, custom docker images, disorderfs for Mycelium). How can we setup this environment? For example, I didn't use docker before. How I should setup docker and where I can get these custom docker images. And what is disorderfs for Mycelium?

[Giszmo]

Thank you for looking into the script :pray:

In fact I am working on making things easier as we speak. Struggling since yesterday.

Once done, the script will have way less dependencies. I created a docker image that is on docker hub, so it should "just work". No custom image needed anymore.

That image contains apktool and apksigner and I can also use git from docker if that is an issue else. Then it would be a single dependency. Not sure ... git is not a fringe dependency to have and devs should be using it anyway. Will see.

disorderfs is a bit more tricky as it has to be run as root on the host system and cannot be bundled in the docker image. It is a tool that allows to randomize the file order to find non-determinisms but it also allows to sort file order to fix non-determinisms in third party tools. The latter is what Mycelium is using it for.

Both remaining dependencies you can easily get on debian for example: sudo apt install -y docker.io disorderfs git.

---

Edit: I spent hours searching how to run disorderfs in docker and I'm constantly learning new things but I couldn't find a way. If you do, I would happily get rid of as much sudo as possible. Trust is precious and asking root access to one's machine is delicate.

[Giszmo]

Pushed the announced change regarding apktool and apksigner in docker image and docker image from docker hub. The script is not getting nicer from that and as I said I don't know how to kill the disorderfs dependency and git is what you need to get the script in the first place.

[Giszmo]

I also updated the Readme.md. Hope it's easy enough now.

[rafaelekol]

I installed Docker, and these ones: ruby-dev ruby-bundler nodejs. But when I try to run your script on Mac I got this error:

Exception in thread "main" java.io.FileNotFoundException: unstoppablePlaystore.apk (No such file or directory)
    at java.io.RandomAccessFile.open0(Native Method)
    at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
    at com.android.apksig.ApkVerifier.verify(ApkVerifier.java:118)
    at com.android.apksigner.ApkSignerTool.verify(ApkSignerTool.java:431)
    at com.android.apksigner.ApkSignerTool.main(ApkSignerTool.java:92)

[Giszmo]

Please try with absolute path. Also ruby-dev ruby-bundler nodejs are only needed to run the website locally. The test.sh should only need docker and git now if you don't want to test Mycelium which additionally needs disorderfs.

Edit: And thanks again for being the first to test my toys. Any contributions to make things smoother are highly welcome. Any feedback helps me to make it smoother, too. The relative path issue for example should be easy to fix. I created an issue for this.

[esengulov]

@Giszmo

I will to take over from @rafaelekol here. Went ahead with your instructions on README, looks like access rights have changes on repository as I am not able to clone it:

getting:

git@gitlab.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

[Giszmo]

@esengulov this looks like you don't have your ssh key configured with your gitlab account. Try to clone using https:

$ git clone https://gitlab.com/walletscrutiny/walletScrutinyCom.git

[rafaelekol]

I managed to run successfully your old script file, that I downloaded on June 15.

./test_old.sh apk/unstoppablePlay.apk 
./test_old.sh: line 28: apksigner: command not found
Extracting APK content ...
I: Using Apktool 2.4.1 on unstoppablePlay.apk
I: Loading resource table...
I: Decoding AndroidManifest.xml with resources...
I: Loading resource table from file: /Users/rafael/Library/apktool/framework/1.apk
I: Regular manifest package...
I: Decoding file-resources...
I: Decoding values */* XMLs...
I: Baksmaling classes.dex...
I: Baksmaling classes2.dex...
I: Baksmaling classes3.dex...
I: Baksmaling classes4.dex...
I: Copying assets and libs...
I: Copying unknown files...
I: Copying original files...
I: Copying META-INF/services directory

Testing "apk/unstoppablePlay.apk" (io.horizontalsystems.bankwallet version 0.15.0)

Testing io.horizontalsystems.bankwallet from https://github.com/horizontalsystems/unstoppable-wallet-android revision 0.15.0 ...
Password:
Cloning into 'app'...
remote: Enumerating objects: 412, done.
remote: Counting objects: 100% (412/412), done.
remote: Compressing objects: 100% (189/189), done.
remote: Total 56720 (delta 184), reused 292 (delta 133), pack-reused 56308
Receiving objects: 100% (56720/56720), 36.57 MiB | 4.42 MiB/s, done.
Resolving deltas: 100% (37245/37245), done.
Trying to checkout version 0.15.0 ...
Note: checking out '0.15.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 2b5c0e99... Increase build version to 29
+ ./gradlew clean :app:assembleRelease
Downloading https://services.gradle.org/distributions/gradle-6.3-all.zip
..........................................................................................................................................

Welcome to Gradle 6.3!

Here are the highlights of this release:
 - Java 14 support
 - Improved error messages for unexpected failures

For more details see https://docs.gradle.org/6.3/release-notes.html

Starting a Gradle Daemon (subsequent builds will be faster)

> Configure project :app
WARNING: BuildType(apptestnet): buildConfigField 'testMode' value is being replaced: false -> true
File /root/.android/repositories.cfg could not be loaded.
Checking the license for package Android SDK Platform 29 in /opt/android-sdk/licenses
License for package Android SDK Platform 29 accepted.
Preparing "Install Android SDK Platform 29 (revision: 4)".
"Install Android SDK Platform 29 (revision: 4)" ready.
Installing Android SDK Platform 29 in /opt/android-sdk/platforms/android-29
"Install Android SDK Platform 29 (revision: 4)" complete.
"Install Android SDK Platform 29 (revision: 4)" finished.
Checking the license for package Android SDK Build-Tools 29.0.3 in /opt/android-sdk/licenses
License for package Android SDK Build-Tools 29.0.3 accepted.
Preparing "Install Android SDK Build-Tools 29.0.3 (revision: 29.0.3)".
"Install Android SDK Build-Tools 29.0.3 (revision: 29.0.3)" ready.
Installing Android SDK Build-Tools 29.0.3 in /opt/android-sdk/build-tools/29.0.3
"Install Android SDK Build-Tools 29.0.3 (revision: 29.0.3)" complete.
"Install Android SDK Build-Tools 29.0.3 (revision: 29.0.3)" finished.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-fr/values-fr.xml:58: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-fr/values-fr.xml:61: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-fr/values-fr.xml:248: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-fr/values-fr.xml:272: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-fr/values-fr.xml:294: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.

/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-ko/values-ko.xml:58: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-ko/values-ko.xml:61: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-ko/values-ko.xml:249: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-ko/values-ko.xml:272: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-ko/values-ko.xml:294: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.

/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-fa/values-fa.xml:58: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-fa/values-fa.xml:61: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-fa/values-fa.xml:247: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-fa/values-fa.xml:270: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-fa/values-fa.xml:292: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.

/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-de/values-de.xml:58: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-de/values-de.xml:61: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-de/values-de.xml:249: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-de/values-de.xml:272: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-de/values-de.xml:294: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.

/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-zh/values-zh.xml:58: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-zh/values-zh.xml:61: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-zh/values-zh.xml:248: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-zh/values-zh.xml:271: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-zh/values-zh.xml:293: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.

/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values/values.xml:1057: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values/values.xml:1060: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values/values.xml:1262: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values/values.xml:1292: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values/values.xml:1314: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.

/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-ru/values-ru.xml:58: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-ru/values-ru.xml:61: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-ru/values-ru.xml:247: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-ru/values-ru.xml:270: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-ru/values-ru.xml:292: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.

/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-tr/values-tr.xml:58: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-tr/values-tr.xml:61: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-tr/values-tr.xml:247: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-tr/values-tr.xml:270: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-tr/values-tr.xml:292: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.

/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-es/values-es.xml:58: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-es/values-es.xml:61: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-es/values-es.xml:247: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-es/values-es.xml:270: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.
/mnt/app/build/intermediates/incremental/mergeReleaseResources/merged.dir/values-es/values-es.xml:292: warn: multiple substitutions specified in non-positional format; did you mean to add the formatted="false" attribute?.

> Task :components:chartview:compileReleaseKotlin
w: /mnt/components/chartview/src/main/java/io/horizontalsystems/chartview/helpers/IndicatorHelper.kt: (50, 25): Variable 'rStrength' initializer is redundant

> Task :components:pin:compileReleaseKotlin
w: /mnt/components/pin/src/main/java/io/horizontalsystems/pin/PinFragment.kt: (198, 41): Parameter 'cryptoObject' is never used

> Task :app:kaptReleaseKotlin
warning: Current JDK version 1.8.0_252-8u252-b09-1~19.10-b09 has a bug (https://bugs.openjdk.java.net/browse/JDK-8007720) that prevents Room from being incremental[WARN] Incremental annotation processing requested, but support is disabled because the following processors are not incremental: androidx.room.RoomProcessor (DYNAMI
> Task :app:compileReleaseKotlin
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/core/adapters/EosAdapter.kt: (106, 13): Variable 'disposable' is never used
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/core/managers/PriceAlertManager.kt: (11, 13): Variable 'disposable' is never used
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/modules/createwallet/view/CoinItemsAdapter.kt: (111, 40): 'getter for adapterPosition: Int' is deprecated. Deprecated in Java
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/modules/createwallet/view/CoinItemsAdapter.kt: (123, 28): 'getter for adapterPosition: Int' is deprecated. Deprecated in Java
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/modules/cryptonews/CryptoNewsFragment.kt: (27, 41): No cast needed
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/modules/notifications/NotificationsActivity.kt: (164, 34): 'getter for adapterPosition: Int' is deprecated. Deprecated in Java
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/modules/ratechart/RateChartView.kt: (46, 32): The corresponding parameter in the supertype 'View' is named 'enabled'. This may cause problems when calling this function with named arguments.
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/modules/ratechart/RateChartView.kt: (50, 33): The corresponding parameter in the supertype 'View' is named 'enabled'. This may cause problems when calling this function with named arguments.
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/modules/ratechart/RateChartView.kt: (54, 32): The corresponding parameter in the supertype 'View' is named 'enabled'. This may cause problems when calling this function with named arguments.
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/modules/send/submodules/confirmation/ConfirmationFragment.kt: (63, 28): Parameter 'context' is never used, could be renamed to _
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/modules/settings/managekeys/views/ManageKeysDeleteAlert.kt: (98, 43): 'getter for adapterPosition: Int' is deprecated. Deprecated in Java
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/modules/settings/security/privacy/PrivacySettingsActivity.kt: (55, 68): Parameter 'enabled' is never used, could be renamed to _
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/modules/settings/security/privacy/PrivacySettingsAdapter.kt: (80, 59): 'getter for adapterPosition: Int' is deprecated. Deprecated in Java
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/modules/settings/security/privacy/PrivacySettingsPresenter.kt: (190, 71): Parameter 'currentValue' is never used
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/modules/transactions/TransactionsFragment.kt: (163, 60): 'getter for adapterPosition: Int' is deprecated. Deprecated in Java
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/modules/transactions/TransactionsFragment.kt: (279, 57): 'getter for adapterPosition: Int' is deprecated. Deprecated in Java
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/ui/extensions/BottomSheetSelectorDialog.kt: (98, 25): 'getter for adapterPosition: Int' is deprecated. Deprecated in Java
w: /mnt/app/src/main/java/io/horizontalsystems/bankwallet/ui/extensions/SelectorDialog.kt: (98, 61): 'getter for adapterPosition: Int' is deprecated. Deprecated in Java

> Task :app:stripReleaseDebugSymbols
WARNING: Compatible side by side NDK version was not found. Default is 20.0.5594570.
Compatible side by side NDK version was not found. Default is 20.0.5594570.
Unable to strip the following libraries, packaging them as they are: libabieos-lib.so, libabieos-lib.so, libabieos-lib.so, libabieos-lib.so, libdashjbls.so, libdashjbls.so, libdashjbls.so, libdashjbls.so, libin3_jni.so, libin3_jni.so, libin3_jni.so, libin3_jni.so, libpthread.so, libpthread.so, libpthread.so, libpthread.so, libtor.so, libtor.so, libtor.so, libtor.so.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 10m 35s
226 actionable tasks: 216 executed, 10 up-to-date
I: Using Apktool 2.4.1 on app-release-unsigned.apk
I: Loading resource table...
I: Decoding AndroidManifest.xml with resources...
I: Loading resource table from file: /Users/rafael/Library/apktool/framework/1.apk
I: Regular manifest package...
I: Decoding file-resources...
I: Decoding values */* XMLs...
I: Baksmaling classes.dex...
I: Baksmaling classes2.dex...
I: Baksmaling classes3.dex...
I: Baksmaling classes4.dex...
I: Copying assets and libs...
I: Copying unknown files...
I: Copying original files...
I: Copying META-INF/services directory
Results for 
appId:          io.horizontalsystems.bankwallet
signer:         
apkVersionName: 0.15.0
apkVersionCode: 29
apkHash:        0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da

Diff:
Files /tmp/fromPlay_io.horizontalsystems.bankwallet_29/apktool.yml and /tmp/fromBuild_io.horizontalsystems.bankwallet_29/apktool.yml differ

Run a full
diff --recursive /tmp/fromPlay_io.horizontalsystems.bankwallet_29 /tmp/fromBuild_io.horizontalsystems.bankwallet_29
meld /tmp/fromPlay_io.horizontalsystems.bankwallet_29 /tmp/fromBuild_io.horizontalsystems.bankwallet_29
for more details.

[rafaelekol]

When I run latest script it is stuck.

Here is the log:

 ./test.sh apk2/unstoppablePlay.apk
+ '[' '!' -f /Users/rafael/Downloads/temp/apk2/unstoppablePlay.apk ']'
++ sha256sum /Users/rafael/Downloads/temp/apk2/unstoppablePlay.apk
++ awk '{print $1;}'
+ apkHash=0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da
+ fromPlayFolder=/tmp/fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da
+ rm -rf /tmp/fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da
++ getSigner /Users/rafael/Downloads/temp/apk2/unstoppablePlay.apk
+++ dirname /Users/rafael/Downloads/temp/apk2/unstoppablePlay.apk
++ DIR=/Users/rafael/Downloads/temp/apk2
+++ basename /Users/rafael/Downloads/temp/apk2/unstoppablePlay.apk
++ BASE=unstoppablePlay.apk
+++ docker run --rm --volume /Users/rafael/Downloads/temp/apk2:/mnt:ro --workdir /mnt walletscrutiny/android:3 apksigner verify --print-certs unstoppablePlay.apk
+++ grep 'Signer #1 certificate SHA-256'
+++ awk '{print $6}'
++ s=c1899493e440489178b8748851b72cbed50c282aaa8c03ae236a4652f8c4f27b
++ echo c1899493e440489178b8748851b72cbed50c282aaa8c03ae236a4652f8c4f27b
+ signer=c1899493e440489178b8748851b72cbed50c282aaa8c03ae236a4652f8c4f27b
+ echo 'Extracting APK content ...'
Extracting APK content ...
+ dockerApktool /tmp/fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da /Users/rafael/Downloads/temp/apk2/unstoppablePlay.apk
+ targetFolder=/tmp/fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da
+ app=/Users/rafael/Downloads/temp/apk2/unstoppablePlay.apk
++ dirname /tmp/fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da
+ targetFolderParent=/tmp
++ basename /tmp/fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da
+ targetFolderBase=fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da
++ dirname /Users/rafael/Downloads/temp/apk2/unstoppablePlay.apk
+ appFolder=/Users/rafael/Downloads/temp/apk2
++ basename /Users/rafael/Downloads/temp/apk2/unstoppablePlay.apk
+ appFile=unstoppablePlay.apk
++ id -u
++ id -g
+ docker run --rm --volume /tmp:/tfp --volume /Users/rafael/Downloads/temp/apk2:/af:ro walletscrutiny/android:3 sh -c 'apktool d -o "/tfp/fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da" "/af/unstoppablePlay.apk"; chown 501:20 -R /tfp/'
I: Using Apktool 2.4.1 on unstoppablePlay.apk
I: Loading resource table...
I: Decoding AndroidManifest.xml with resources...
I: Loading resource table from file: /root/.local/share/apktool/framework/1.apk
I: Regular manifest package...
I: Decoding file-resources...
I: Decoding values */* XMLs...
I: Baksmaling classes.dex...
I: Baksmaling classes2.dex...
I: Baksmaling classes3.dex...
I: Baksmaling classes4.dex...
I: Copying assets and libs...
I: Copying unknown files...
I: Copying original files...
I: Copying META-INF/services directory
chown: changing ownership of '/tfp/AlTest1.out': Operation not permitted
chown: changing ownership of '/tfp/powerlog': Operation not permitted
chown: changing ownership of '/tfp/AlTest1.err': Operation not permitted
+ return 1
+ exit 1

[Giszmo]

Looks like you are not root inside your docker? The crashing part is ... running docker and writing files with it by default results in files the normal user can't delete. The failing operation tries to fix that. Maybe it should just not fail if chown fails?

[rafaelekol]

I tried to fix issue with permission in dockerApktool(), by adding --privileged and there is also permission issue in Mac OS Catalina(in my PC docker has permission only in Downloads directory, where I run the script), but none has worked. Then I tried to replace dockerApktool() with command from old script apktool d -o $fromPlayFolder "$downloadedApp" || exit 1 But this time I got this error:

./test.sh apk/unstoppablePlay.apk            
+ '[' '!' -f /Users/rafael/Downloads/temp/apk/unstoppablePlay.apk ']'
++ sha256sum /Users/rafael/Downloads/temp/apk/unstoppablePlay.apk
++ awk '{print $1;}'
+ apkHash=0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da
+ fromPlayFolder=/tmp/fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da
+ rm -rf /tmp/fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da
++ getSigner /Users/rafael/Downloads/temp/apk/unstoppablePlay.apk
+++ dirname /Users/rafael/Downloads/temp/apk/unstoppablePlay.apk
++ DIR=/Users/rafael/Downloads/temp/apk
+++ basename /Users/rafael/Downloads/temp/apk/unstoppablePlay.apk
++ BASE=unstoppablePlay.apk
+++ docker run --rm --volume /Users/rafael/Downloads/temp/apk:/mnt:ro --workdir /mnt walletscrutiny/android:3 apksigner verify --print-certs unstoppablePlay.apk
+++ grep 'Signer #1 certificate SHA-256'
+++ awk '{print $6}'
++ s=c1899493e440489178b8748851b72cbed50c282aaa8c03ae236a4652f8c4f27b
++ echo c1899493e440489178b8748851b72cbed50c282aaa8c03ae236a4652f8c4f27b
+ signer=c1899493e440489178b8748851b72cbed50c282aaa8c03ae236a4652f8c4f27b
+ echo 'Extracting APK content ...'
Extracting APK content ...
+ apktool d -o /tmp/fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da /Users/rafael/Downloads/temp/apk/unstoppablePlay.apk
I: Using Apktool 2.4.1 on unstoppablePlay.apk
I: Loading resource table...
I: Decoding AndroidManifest.xml with resources...
I: Loading resource table from file: /Users/rafael/Library/apktool/framework/1.apk
I: Regular manifest package...
I: Decoding file-resources...
I: Decoding values */* XMLs...
I: Baksmaling classes.dex...
I: Baksmaling classes2.dex...
I: Baksmaling classes3.dex...
I: Baksmaling classes4.dex...
I: Copying assets and libs...
I: Copying unknown files...
I: Copying original files...
I: Copying META-INF/services directory
++ cat /tmp/fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da/AndroidManifest.xml
++ head -n 1
++ sed 's/.*package=\"//g'
++ sed 's/\".*//g'
+ appId=io.horizontalsystems.bankwallet
++ cat /tmp/fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da/apktool.yml
++ grep versionName
++ sed 's/.*\: //g'
++ sed 's/'\''//g'
+ versionName=0.15.0
++ cat /tmp/fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da/apktool.yml
++ grep versionCode
++ sed 's/.*\: //g'
++ sed 's/'\''//g'
+ versionCode=29
+ fromPlayUnpacked=/tmp/fromPlay_io.horizontalsystems.bankwallet_29
+ workDir=/tmp/testio.horizontalsystems.bankwallet
+ rm -rf /tmp/fromPlay_io.horizontalsystems.bankwallet_29
+ mv /tmp/fromPlay0dc205735c5a61d2f735a1ff7c06369a564657e2eaea73b6bbaa599da442d6da /tmp/fromPlay_io.horizontalsystems.bankwallet_29
+ '[' -z io.horizontalsystems.bankwallet ']'
+ '[' -z 0.15.0 ']'
+ '[' -z 29 ']'
+ echo

+ echo 'Testing "/Users/rafael/Downloads/temp/apk/unstoppablePlay.apk" (io.horizontalsystems.bankwallet version 0.15.0)'
Testing "/Users/rafael/Downloads/temp/apk/unstoppablePlay.apk" (io.horizontalsystems.bankwallet version 0.15.0)
+ echo

+ case "$appId" in
+ testUnstoppable
+ repo=https://github.com/horizontalsystems/unstoppable-wallet-android
+ tag=0.15.0
+ builtApk=/tmp/testio.horizontalsystems.bankwallet/app/app/build/outputs/apk/release/app-release-unsigned.apk
+ prepare
+ echo 'Testing io.horizontalsystems.bankwallet from https://github.com/horizontalsystems/unstoppable-wallet-android revision 0.15.0 ...'
Testing io.horizontalsystems.bankwallet from https://github.com/horizontalsystems/unstoppable-wallet-android revision 0.15.0 ...
+ sudo rm -rf /tmp/testio.horizontalsystems.bankwallet
+ mkdir /tmp/testio.horizontalsystems.bankwallet
+ cd /tmp/testio.horizontalsystems.bankwallet
+ git clone https://github.com/horizontalsystems/unstoppable-wallet-android app
Cloning into 'app'...
remote: Enumerating objects: 451, done.
remote: Counting objects: 100% (451/451), done.
remote: Compressing objects: 100% (201/201), done.
remote: Total 56743 (delta 195), reused 332 (delta 144), pack-reused 56292
Receiving objects: 100% (56743/56743), 36.58 MiB | 4.82 MiB/s, done.
Resolving deltas: 100% (37245/37245), done.
+ cd app
+ echo 'Trying to checkout version 0.15.0 ...'
Trying to checkout version 0.15.0 ...
+ git checkout '"0.15.0"'
error: pathspec '"0.15.0"' did not match any file(s) known to git.
+ exit 1

[Giszmo]

I am a bit confused now. The line git checkout \"$tag\" is indeed producing the wrong git checkout '"0.15.0"' but where do these extra ' come from? I vaguely remember I had added \" fighting with some wallet's creative tags containing some spaces or something but I get your error for 0.14.1, too, so I will revert this to not contain \".

[rafaelekol]

I tried to remove this part ; chown $(id -u):$(id -g) -R /tfp/ from your script and now its working. So this method looks like this.

dockerApktool() {
  targetFolder=$1
  app=$2
  targetFolderParent=$(dirname "$targetFolder")
  targetFolderBase=$(basename "$targetFolder")
  appFolder=$(dirname "$app")
  appFile=$(basename "$app")
  docker run --rm --volume $targetFolderParent:/tfp --volume $appFolder:/af:ro $wsDocker sh -c "apktool d -o \"/tfp/$targetFolderBase\" \"/af/$appFile\""
  return $?
}

I also tried other wallet, AirGap Vault. Script also worked without a problem.

[Giszmo]

@rafaelekol many thanks!

I would appreciate your comment on some brainstorming. Any thoughts on how to submit rebuild success? I hope to get to a point where several rebuilders rebuild newly found binaries in a timely manner and report back with a pgp signed message. The website would then have to report on those with a reference to who did the rebuilding and signed with which key for verification.

I have a server to collect binaries and thought that rebuilders could register there with a callback link or something so they get notified on new binaries or maybe with mail/xmpp/telegram/slack/... notifications. The rebuilders could then call an API of that server to report back the signed rebuild success. The server could allow arbitrary rebuilders but on the website, some with profiles would get emphasized. This way, rebuilders put their reputation on the line if they report a binary to be reproducible which was clearly not reproducible and educated visitors could check the pgp signature, so it wouldn't be me who said it but the rebuilder.

Ideally it would all be automated:

• app collects binaries
• server pings rebuilders
• rebuilders fetch binary
• rebuilders submit test result
• walletscrutiny serves the result, sourced from collection server.

[Giszmo]

@rafaelekol for my routine, the chown part is quite helpful. I turned it into chown $(id -u):$(id -g) -R /tfp/ || true and tested it with cccccc || true and I think it should do the trick.

[rafaelekol]

Me and my colleague discussed how we would like to see your build checker script. Ideally we think this task should be so simple, that even non developer guy can do it. For example this guy once(or twice) in a month would run the script which will download updated binaries, build their copies and compare them, then it will publish results or send them to you with account name, so you will know who sent you the report. Also on each run of the script, it can check for latest version of script, then update itself and start doing comparing job. For authentication of reports, you can provide API key for each account.

[Giszmo]

Making it as simple as possible is certainly a goal but I think recruiting people who do not understand anything about the script would not be very suited as the script itself is part of the audit process, so if that is broken, then it might need fixing, too and only a dev would be able to recognize what is going on.

Other than that, apk collection is almost automated. It should be easy to make those apks downloadable. Then it should be easy to extend the test script so that you can run ./test.sh --all-from-server --new and it would work through all of the new apks or ./test.sh --appId=io.horizontalsystems.bankwallet --latest and it would test that one.

I think the initial setup with "install docker" cannot be made easier.

Results could be written to a file to avoid re-testing the same apk with the same test script and uploaded to the server but somehow I think human interaction should be required at some point. If rebuild fails, the app might still be rebuildable. If test script signals success, it might be that it tested wrong.

[rafaelekol]

Initial setup as Docker install is very simple now, comparing to the previous setup, when we needed APKtool and APKsigner to be installed too. I think having small documentation with all existing commands to run script would be helpful. While testing script I noted that single build can take a lot of time. And if we try to check multiple builds at once, that can take hours. Is there an option to run this task on CI(like Jenkins) machine? Running on notebook while I work on it is not preferable.

[Giszmo]

The script requires some interaction: The docker is not closed, so you can poke around and find out how to fix things if a build failed, without having to slowly get to that point again. Maybe there should be a headless option? On the other hand some projects, especially when joining the ranks of reproducible apps tend to require changes to the test script. Later, when more rebuilders run it, I hope though that projects will provide the required changes to the test script ahead of others having to rebuild.

Also I could imagine massive speed-ups by retaining state. State can result in conflicting rebuild results though. Not sure ... stuff that is pinned to a hash should be cached but other stuff not.

[Giszmo]

Pull requests are welcome. I bet the test script could be improved in a hundred ways. I just changed the clone part to only clone the tagged revision which reduces downloads for Mycelium clone by 70%. Another improvement would be to allow multiple tests at once, using smart parallelization and output of results.

[rafaelekol]

Also each time when I download new script I need to change permission, to be able to run this script. This also extra task for me.

[Giszmo]

That is weird. According to this Stackoverflow permissions can be set using git update-index --chmod=+x test.sh but:

$ git ls-files --stage grep test.sh
100755 31b612fe1a01652a6796919bcf36073c2c8de9f6 0   test.sh

Looks executable to me.

$ git update-index --chmod=-x test.sh
$ git ls-files --stage grep test.sh
100644 31b612fe1a01652a6796919bcf36073c2c8de9f6 0   test.sh
$ git update-index --chmod=+x test.sh
$ git ls-files --stage grep test.sh
100755 31b612fe1a01652a6796919bcf36073c2c8de9f6 0   test.sh
$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

So I would say while test.sh is stored in git as executable, this doesn't get reflected on your system. Weird it removes the execute flag when you update or do you clone anew?

Instead of ./test.sh you can always run sh ./test.sh to avoid this issue completely.

[rafaelekol]

Since its a single file I decided just manually download it without using git. But as you said in case it will be regularly updated, then I should better use git for updating this file.

[rafaelekol]

Can we close this issue, and mode brainstorming conversation on improvement of your script on your issue tracker. In order to keep other people run your script I think you can put prerequisite for wallets to be listed on your site. Like running your script each month for top listed wallets.

[Giszmo]

Ok. If you want to help, I created this issue. Hope to see you there or some other way to keep the chat and ultimately see rebuild results from your side.

[rafaelekol]

Your issue on Gitlab is looking nice. For new comers it would be easier to understand whats going on :) I also saw some typos(like: MR welcome and re-builable), I guess you haven't checked text yet. Also how will I know when I should rebuild apks? Is this when you finish this testing system automation?

[Giszmo]

MR welcome: the guys at gitlab feel really strong about calling "merge request" what github calls "pull request" and if you are familiar with git pull and git merge, you clearly see why ;)

The other question is what you asked me to create the other issue for but if you want to continue here: I only get so much done every day. Yes, the idea is that a server collects all binaries from an app but you can also submit your latest builds before the app collects it. Then it pings rebuilders via mail/slack/irc/... to check out this new binary. Rebuilders then can download it from the server.

[rafaelekol]

I see what you mean. If I submit build directly to your server, how you can be sure that this is same build that goes to playstore?

[Giszmo]

If I submit build directly to your server, how you can be sure that this is same build that goes to playstore?

I don't. I monitor many builds and assume evil wallet devs will deliver a good version here and a bad version there as they can slice the market by country, language, device type, even device id. That is why I am also working on an app that collects updates and submits them to the server for review. Ultimately the project will track rebuildability of apps based on app hash and app version and signer and version number are only meta data.

So when you submit an app for checking and you do so without signing it, it will have a different hash than what comes later from google play. This is a pain point indeed as you will first get the feedback that your build was reproducible but that check will run again some days later and this time it might fail anyway. I'm open for ideas. One way would be to strip the signature before hashing but I'm not sure if that is possible deterministically. Another way would be to pinky-swear to not share the apk with non-rebuilders before it hits Google Play and have a list of accredited rebuilders but that's difficult to guarantee as there is plausible deniability once files leak and there will hopefully be several rebuilders.