search

akumaigorodski / wallet

Bitcoin wallet for Android
https://sbw.app
Apache License 2.0
240 stars 73 forks source link

App not reproducible #90

Closed emanuelb closed 3 years ago

emanuelb commented 3 years ago

Comparing APKs latest version 2.1.14 from github https://github.com/btcontract/wallet/releases/download/2.1.14/SBW-2.1.14.apk to APK from fdroid: https://f-droid.org/repo/com.btcontract.wallet_75.apk

Result in diff in several files: diff --recursive --brief ./SBW-2.1.14/ ./com.btcontract.wallet_75/

Only in ./SBW-2.1.14/assets: graph.snapshot-mainnet.zlib
Files ./SBW-2.1.14/classes3.dex and ./com.btcontract.wallet_75/classes3.dex differ
Files ./SBW-2.1.14/classes.dex and ./com.btcontract.wallet_75/classes.dex differ
Files ./SBW-2.1.14/lib/arm64-v8a/libsecp256k1.so and ./com.btcontract.wallet_75/lib/arm64-v8a/libsecp256k1.so differ
Files ./SBW-2.1.14/lib/arm64-v8a/libwallycore.so and ./com.btcontract.wallet_75/lib/arm64-v8a/libwallycore.so differ
Files ./SBW-2.1.14/lib/armeabi-v7a/libsecp256k1.so and ./com.btcontract.wallet_75/lib/armeabi-v7a/libsecp256k1.so differ
Files ./SBW-2.1.14/lib/armeabi-v7a/libwallycore.so and ./com.btcontract.wallet_75/lib/armeabi-v7a/libwallycore.so differ
Files ./SBW-2.1.14/lib/x86/libsecp256k1.so and ./com.btcontract.wallet_75/lib/x86/libsecp256k1.so differ
Files ./SBW-2.1.14/lib/x86/libwallycore.so and ./com.btcontract.wallet_75/lib/x86/libwallycore.so differ
Files ./SBW-2.1.14/lib/x86_64/libsecp256k1.so and ./com.btcontract.wallet_75/lib/x86_64/libsecp256k1.so differ
Files ./SBW-2.1.14/lib/x86_64/libwallycore.so and ./com.btcontract.wallet_75/lib/x86_64/libwallycore.so differ
  1. why file graph.snapshot-mainnet.zlib not exists in f-droid version? is it a bug that should be opened in their gitlab?
  2. diffoscope https://diffoscope.org/ should be run on both files to look into the diffs in the files.
  3. in what environment the github APK was build, as the diffs are probably caused by it.
akumaigorodski commented 3 years ago

why file graph.snapshot-mainnet.zlib not exists in f-droid version? is it a bug that should be opened in their gitlab?

Not a bug, having that file included is an optimization which shortens initial graph sync time from a few minutes to a few seconds, without it LN graph gets synchronized from scratch.

F-droid version does not have it because they fetch my releases from repository and compile them on their servers, and this particular snapshot file is rather big, also needs to be updated occasionally, so I've decided to not include it in commit history.

in what environment the github APK was build, as the diffs are probably caused by it.

Pop!_OS 21.04. As to binary files incompatibility, it just occurred to me I can actually remove libwallycore code and leave libsecp256k1 only since libwallycore was initially used because of Scrypt library which is not utilized currently. This should at least remove some diverging .so files, I'll do that in next release.

emanuelb commented 3 years ago

Building with ./gradlew assembleRelease by Containerfile that use ubuntu 21.04 as base and JDK 11:


FROM ubuntu:21.04

RUN set -ex; \
    mkdir -p /usr/share/man/man1/; \
    apt-get update; \
    apt-get install --yes --no-install-recommends openjdk-11-jdk git wget unzip; \
    rm -rf /var/lib/apt/lists/*; \
    useradd -ms /bin/bash appuser;

USER appuser

ENV ANDROID_SDK_ROOT="/home/appuser/app/sdk" \
    ANDROID_HOME="/home/appuser/app/sdk" \
    ANDROID_NDK_HOME="/home/appuser/app/sdk/ndk/22.1.7171670/" \
    JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF8"

RUN set -ex; \
    mkdir -p "/home/appuser/app/sdk/licenses" "/home/appuser/app/sdk/ndk" "/home/appuser/app/simplebitcoinwallet/"; \
    printf "\n24333f8a63b6825ea9c5514f83c2829b004d1fee" > "/home/appuser/app/sdk/licenses/android-sdk-license"; \
    cd /home/appuser/app/sdk/; \
    wget https://dl.google.com/android/repository/android-ndk-r22b-linux-x86_64.zip; \
    unzip android-ndk-r22b-linux-x86_64.zip; \
    rm android-ndk-r22b-linux-x86_64.zip; \
    mv android-ndk-r22b "/home/appuser/app/sdk/ndk/22.1.7171670/"; \
    cd /home/appuser/app/simplebitcoinwallet/; \
    git clone https://github.com/btcontract/wallet; \
    cd /home/appuser/app/simplebitcoinwallet/wallet/; \
    git checkout 2.1.14; \
    cd /home/appuser/app/simplebitcoinwallet/wallet/app/src/main/assets/; \
    wget https://github.com/btcontract/wallet/releases/download/2.1.14/graph.snapshot-mainnet.zlib; \
    cd /home/appuser/app/simplebitcoinwallet/wallet/;

WORKDIR /home/appuser/app/simplebitcoinwallet/wallet/

Comparing to github APK:

Files ./simplebitcoinwallet2114_lv_rebuild/classes2.dex and ./SBW-2.1.14/classes2.dex differ
Files ./simplebitcoinwallet2114_lv_rebuild/classes3.dex and ./SBW-2.1.14/classes3.dex differ
Files ./simplebitcoinwallet2114_lv_rebuild/classes4.dex and ./SBW-2.1.14/classes4.dex differ
Files ./simplebitcoinwallet2114_lv_rebuild/classes.dex and ./SBW-2.1.14/classes.dex differ
Files ./simplebitcoinwallet2114_lv_rebuild/lib/arm64-v8a/libsecp256k1.so and ./SBW-2.1.14/lib/arm64-v8a/libsecp256k1.so differ
Files ./simplebitcoinwallet2114_lv_rebuild/lib/arm64-v8a/libwallycore.so and ./SBW-2.1.14/lib/arm64-v8a/libwallycore.so differ
Files ./simplebitcoinwallet2114_lv_rebuild/lib/armeabi-v7a/libsecp256k1.so and ./SBW-2.1.14/lib/armeabi-v7a/libsecp256k1.so differ
Files ./simplebitcoinwallet2114_lv_rebuild/lib/armeabi-v7a/libwallycore.so and ./SBW-2.1.14/lib/armeabi-v7a/libwallycore.so differ
Files ./simplebitcoinwallet2114_lv_rebuild/lib/x86/libsecp256k1.so and ./SBW-2.1.14/lib/x86/libsecp256k1.so differ
Files ./simplebitcoinwallet2114_lv_rebuild/lib/x86/libwallycore.so and ./SBW-2.1.14/lib/x86/libwallycore.so differ
Files ./simplebitcoinwallet2114_lv_rebuild/lib/x86_64/libsecp256k1.so and ./SBW-2.1.14/lib/x86_64/libsecp256k1.so differ
Files ./simplebitcoinwallet2114_lv_rebuild/lib/x86_64/libwallycore.so and ./SBW-2.1.14/lib/x86_64/libwallycore.so differ
Only in ./SBW-2.1.14/META-INF: CERT.RSA
Only in ./SBW-2.1.14/META-INF: CERT.SF
Files ./simplebitcoinwallet2114_lv_rebuild/META-INF/MANIFEST.MF and ./SBW-2.1.14/META-INF/MANIFEST.MF differ

Inspecting diffs in files should be done with diffoscope, also rebuild with JDK8 instead.

akumaigorodski commented 3 years ago

libwallycore was removed in https://github.com/btcontract/wallet/releases/tag/2.2.14 and APK itself was built inside of container using this container file: https://github.com/btcontract/wallet/blob/master/Containerfile.

I've also updated build and sign instructions here: https://github.com/btcontract/wallet#building-from-source

akumaigorodski commented 3 years ago

I've identified a few advanced Scala features which seemingly do not play well with reproducibility (chunks of code get moved around non-deterministically while compiling), currently doing some refactoring to get rid of those and hopefully in next release this will be resolved.

emanuelb commented 3 years ago

Containerfile fixes/suggestions:

  1. use COPY instructions to copy current directory instead of using git clone & git checkout to get files in repo, thus work on every commit in repo and on files changes.
  2. use variables for stuff like version of NDK such as "22.1.7171670" & "r22b" as they will be upgraded sometime in future.

will test RB after you say related issues/refactoring are resolved.

akumaigorodski commented 3 years ago

Issues were fixed in 2.2.15.1. APK was built in container again and should now be reproducible. I've also updated Containerfile and build instructions to avoid rebuilding image from scratch.

emanuelb commented 3 years ago

Tested version 2.2.15.1 sha256sum for https://github.com/btcontract/wallet/releases/download/2.2.15.1/SBW-2.2.15.1.apk is: 26851863c62f4114009c1d5cab95610d9d93035714153466d89be28d4b57af4b

Running ./gradlew assembleRelease in above container:

podman build --rm -t simplebitcoinwallet_lv_build_apk -f ContainerFile
podman run --rm --name simplebitcoinwallet_lv_build_apk -ti simplebitcoinwallet_lv_build_apk
podman cp simplebitcoinwallet_lv_build_apk:/app/simplebitcoinwallet/wallet/app/build/outputs/apk/release/SBW-2.2.15.1.apk ~/simplebitcoinwallet_build_2.2.15.1.apk
FROM debian:sid-slim

RUN set -ex; \
    mkdir -p /usr/share/man/man1/ /app/; \
    useradd -ms /bin/bash appuser; \
    chown appuser /app/; \
    apt-get update; \
    DEBIAN_FRONTEND=noninteractive apt-get install --yes -o APT::Install-Suggests=false --no-install-recommends openjdk-11-jdk git wget unzip; \
    rm -rf /var/lib/apt/lists/*;

USER appuser

ENV ANDROID_SDK_ROOT="/app/sdk" \
    ANDROID_HOME="/app/sdk" \
    ANDROID_NDK_HOME="/app/sdk/ndk/22.1.7171670/" \
    JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF8"

RUN set -ex; \
    mkdir -p "/app/sdk/licenses" "/app/sdk/ndk" "/app/simplebitcoinwallet/"; \
    printf "\n24333f8a63b6825ea9c5514f83c2829b004d1fee" > "/app/sdk/licenses/android-sdk-license"; \
    cd /app/sdk/; \
    wget https://dl.google.com/android/repository/android-ndk-r22b-linux-x86_64.zip; \
    unzip android-ndk-r22b-linux-x86_64.zip; \
    rm android-ndk-r22b-linux-x86_64.zip; \
    mv android-ndk-r22b "/app/sdk/ndk/22.1.7171670/"; \
    cd /app/simplebitcoinwallet/; \
    git clone https://github.com/btcontract/wallet; \
    cd /app/simplebitcoinwallet/wallet/; \
    git checkout 2.2.15.1; \
    cd /app/simplebitcoinwallet/wallet/app/src/main/assets/; \
    wget https://github.com/btcontract/wallet/releases/download/2.2.15.1/graph.snapshot-mainnet.zlib; \
    cd /app/simplebitcoinwallet/wallet/;

WORKDIR /app/simplebitcoinwallet/wallet/

app is reproducible as brief diff is: 

diff --recursive --brief ./SBW-2.2.15.1/ ./simplebitcoinwallet_build_2.2.15.1/
Only in ./SBW-2.2.15.1/META-INF: BITCOINS.RSA
Only in ./SBW-2.2.15.1/META-INF: BITCOINS.SF
Files ./SBW-2.2.15.1/META-INF/MANIFEST.MF and ./simplebitcoinwallet_build_2.2.15.1/META-INF/MANIFEST.MF differ