search

Blockstream / green_android

Blockstream Green Wallet for Android
GNU General Public License v3.0
196 stars 80 forks source link

F-Droid build failed #173

Closed licaon-kter closed 2 months ago

licaon-kter commented 1 year ago 
* What went wrong:
Execution failed for task ':lightning:compileReleaseJavaWithJavac'.
> Could not resolve all files for configuration ':lightning:releaseCompileClasspath'.
   > Could not find breez_sdk:bindings-android:0.1.0.
     Required by:
         project :lightning

No other maven except their own? https://sdk-doc.breez.technology/

Can we build this from source?

/LE: fyi https://gitlab.com/fdroid/fdroiddata/-/commit/554ea14f0cbc450dc5de881cb7ca077e3ac90f2d

roeierez commented 1 year ago

We are going to push to another public maven repo.

licaon-kter commented 1 year ago

@roeierez any timeframe? I see that 4.0.10 was tagged

roeierez commented 1 year ago

@licaon-kter we are working on integrating with https://jitpack.io/ we hope to resolve these and publish in a week or so (cc @cnixbtc) Can you confirm this is going to work with F-Droid?

Also the the breez sdk dependency is an open source project, you can find the build instructions for the android package here: https://github.com/breez/breez-sdk/tree/main/libs/sdk-bindings

yaslama commented 1 year ago 
* What went wrong:
Execution failed for task ':lightning:compileReleaseJavaWithJavac'.
> Could not resolve all files for configuration ':lightning:releaseCompileClasspath'.
   > Could not find breez_sdk:bindings-android:0.1.0.
     Required by:
         project :lightning

How can we reproduce?

No other maven except their own? https://sdk-doc.breez.technology/

Is this a problem for fdroid building?

Can we build this from source?

Yes. See https://github.com/breez/breez-sdk/tree/main/libs/sdk-bindings

licaon-kter commented 1 year ago

How can we reproduce?

Remove breez.technology maven repo from code

Is this a problem for fdroid building?

Yes, see https://f-droid.org/2022/07/22/maven-central.html and https://f-droid.org/en/docs/Inclusion_Policy/ from "Trusted maven repos"

licaon-kter commented 1 year ago

Ok, so... this srclib/breez-sdk.yml was added:

RepoType: git
Repo: https://github.com/breez/breez-sdk.git
Subdir: libs/sdk-bindings

and the recipe adapted:

  - versionName: 4.0.11
    versionCode: 22000411
    commit: 27bdf31e92b3d07acb2c912e6bcc141147b6eccf
    timeout: 25200
    subdir: green
    sudo:
      - apt-get update
      - apt-get install -y automake clang cmake jq libffi-dev libtool make pkg-config
        swig virtualenv protobuf-compiler
      - apt-get install -y openjdk-17-jdk-headless
      - update-java-alternatives -a
    gradle:
      - production
    srclibs:
      - breez-sdk@0.1.4
      - rustup@1.25.2
    rm:
      - bump_gdk.sh
    prebuild:
      - sed -i -e '/mvn.breez/d' ../build.gradle.kts
      - sed -i -e '/signingConfigs {/,+8d' -e '/signingConfigs.getByName/,+3d' -e
        '/versionNameSuffix/d' build.gradle.kts
      - sed -i -e '/dependencies {/aimplementation(files(\"libs/breez-sdk.aar\"))'
        build.gradle.kts
    build:
      - $$rustup$$/rustup-init.sh -y
      - source $HOME/.cargo/env
      - rustup install 1.68.0
      - export ANDROID_NDK_HOME=$$NDK$$
      - pushd $$breez-sdk$$
      - sed -i -e '/apple/d' makefile
      - make init
      - make bindings-android
      - popd
      - mkdir -p libs
      - cp $$breez-sdk$$/bindings-android/lib/build/outputs/aar/lib-release.aar libs/breez-sdk.aar
      - JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::") PATH=$HOME/.local/bin:$PATH
        ../gdk/prepare_gdk_clang.sh arm64-v8a
    ndk: r25c

But fails:

...
> Task :lightning:compileReleaseJavaWithJavac FAILED
> Task :gdk:mergeReleaseJniLibFolders
> Task :no-gms:javaPreCompileRelease
> Task :green:dataBindingMergeDependencyArtifactsProductionRelease FAILED

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':lightning:compileReleaseJavaWithJavac'.
> Could not resolve all files for configuration ':lightning:releaseCompileClasspath'.
   > Could not find breez_sdk:bindings-android:0.1.4.
     Required by:
         project :lightning

full log: com.greenaddress.greenbits_android_wallet_22000411.log.gz

Surely this is an easy fix, right? Maybe I've not setup the kotlin thing to pickup the .aar correctly? Maybe easier?

yaslama commented 1 year ago

How can we reproduce?

Remove breez.technology maven repo from code

I want to run fdroid build in a vagrant vm to reproduce the problem but I cannot find the config.yml that you used to build.

licaon-kter commented 1 year ago

Save this as metadata/com.greenaddress.greenbits_android_wallet.yml: https://gist.github.com/licaon-kter/c86fd8a8de5b5536b5fea968f4d2f15b

Don't forget to create srclib/breez-sdk.yml:

RepoType: git
Repo: https://github.com/breez/breez-sdk.git
Subdir: libs/sdk-bindings

run as: fdroid build --verbose --scan-binary --server com.greenaddress.greenbits_android_wallet

angelix commented 1 year ago

@licaon-kter is breez maven repo still an issue?

licaon-kter commented 1 year ago

@angelix yes, no progress here

angelix commented 1 year ago

Please check a260b4a46631da205784131c5671917fa35f065c for a temp solution.

In common/build.gradle.kts you have to make the following changes(comment/uncomment)

If it works i can add the change to prepare_fdroid.sh if it's used

angelix commented 1 year ago

@licaon-kter if you have any other ideas on how to solve this issue, please let me know.

licaon-kter commented 1 year ago

So updated recipe, afaik

  - versionName: 4.0.18
    versionCode: 22000418
    commit: 57dc1781c143d79dac5dc0b68fc5ec1d25aadbb1
    timeout: 25200
    subdir: green
    sudo:
      - apt-get update
      - apt-get install -y automake clang cmake jq libffi-dev libtool make pkg-config
        swig virtualenv xz-utils
      - apt-get install -y openjdk-17-jdk-headless
      - update-java-alternatives -a
    gradle:
      - production
    srclibs:
      - rustup@1.25.2
    prebuild:
      - sed -i -e '/mvn.breez/d' -e '/zendesk/d' ../build.gradle.kts
      - sed -i -e '/libs.breez.sdk.kmp/d' -e 's!// api(libs.breez.sdk.android)!api(libs.breez.sdk.android)!' ../common/build.gradle.kts
      - sed -i -e '/signingConfigs {/,+8d' -e '/signingConfigs.getByName/,+3d' -e
        '/versionNameSuffix/d' build.gradle.kts
    build:
      - $$rustup$$/rustup-init.sh -y
      - source $HOME/.cargo/env
      - rustup install 1.68.0
      - rustup target add aarch64-linux-android armv7-linux-androideabi
      - JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::") PATH=$HOME/.local/bin:$PATH
        ../gdk/prepare_gdk_clang.sh arm64-v8a
      - JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::") PATH=$HOME/.local/bin:$PATH
        ../gdk/prepare_gdk_clang.sh armeabi-v7a
    ndk: r25c

but fails somewhere: com.greenaddress.greenbits_android_wallet_22000418.log.gz

Ideas?

angelix commented 1 year ago

So updated recipe, afaik

  - versionName: 4.0.18
    versionCode: 22000418
    commit: 57dc1781c143d79dac5dc0b68fc5ec1d25aadbb1
    timeout: 25200
    subdir: green
    sudo:
      - apt-get update
      - apt-get install -y automake clang cmake jq libffi-dev libtool make pkg-config
        swig virtualenv xz-utils
      - apt-get install -y openjdk-17-jdk-headless
      - update-java-alternatives -a
    gradle:
      - production
    srclibs:
      - rustup@1.25.2
    prebuild:
      - sed -i -e '/mvn.breez/d' -e '/zendesk/d' ../build.gradle.kts
      - sed -i -e '/libs.breez.sdk.kmp/d' -e 's!// api(libs.breez.sdk.android)!api(libs.breez.sdk.android)!' ../common/build.gradle.kts
      - sed -i -e '/signingConfigs {/,+8d' -e '/signingConfigs.getByName/,+3d' -e
        '/versionNameSuffix/d' build.gradle.kts
    build:
      - $$rustup$$/rustup-init.sh -y
      - source $HOME/.cargo/env
      - rustup install 1.68.0
      - rustup target add aarch64-linux-android armv7-linux-androideabi
      - JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::") PATH=$HOME/.local/bin:$PATH
        ../gdk/prepare_gdk_clang.sh arm64-v8a
      - JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::") PATH=$HOME/.local/bin:$PATH
        ../gdk/prepare_gdk_clang.sh armeabi-v7a
    ndk: r25c

but fails somewhere: com.greenaddress.greenbits_android_wallet_22000418.log.gz

Ideas?

Seems for some reason gradle tries to download gdk binaries.

Can you comment out the following line in /gdk/build.gradle.kts?

commandLine("./fetch_android_binaries.sh")
licaon-kter commented 1 year ago

If I comment/remove that line if fails with

Execution failed for task ':gdk:fetchAndroidBinaries'.
> execCommand == null!

If I remove https://github.com/Blockstream/green_android/blob/release_4.0.18/gdk/build.gradle.kts#L25-L43 it fails later for some other reason :(

angelix commented 11 months ago

@licaon-kter Can you try with 47c14fd9d69c9c54b9a37d20846b43808315b94c?

licaon-kter commented 11 months ago 
> Task :green:mergeProductionReleaseJavaResource

> Task :green:minifyProductionReleaseWithR8 FAILED
ERROR: /home/vagrant/build/com.greenaddress.greenbits_android_wallet/gdk/build/intermediates/runtime_library_classes_jar/release/classes.jar: R8: Type com.blockstream.libgreenaddress.GDKJNI$NotificationHandler is defined multiple times: /home/vagrant/build/com.greenaddress.greenbits_android_wallet/gdk/build/intermediates/runtime_library_classes_jar/release/classes.jar:com/blockstream/libgreenaddress/GDKJNI$NotificationHandler.class, /home/vagrant/build/com.greenaddress.greenbits_android_wallet/green/build/intermediates/javac/productionRelease/classes/com/blockstream/libgreenaddress/GDKJNI$NotificationHandler.class

> Task :green:lintVitalAnalyzeProductionRelease

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':green:minifyProductionReleaseWithR8'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.R8Task$R8Runnable
   > Compilation failed to complete, origin: /home/vagrant/build/com.greenaddress.greenbits_android_wallet/gdk/build/intermediates/runtime_library_classes_jar/release/classes.jar:com/blockstream/libgreenaddress/GDKJNI$NotificationHandler.class

full log: @angelix
com.greenaddress.greenbits_android_wallet_22000418.log.gz

angelix commented 11 months ago 
> Task :green:mergeProductionReleaseJavaResource

> Task :green:minifyProductionReleaseWithR8 FAILED
ERROR: /home/vagrant/build/com.greenaddress.greenbits_android_wallet/gdk/build/intermediates/runtime_library_classes_jar/release/classes.jar: R8: Type com.blockstream.libgreenaddress.GDKJNI$NotificationHandler is defined multiple times: /home/vagrant/build/com.greenaddress.greenbits_android_wallet/gdk/build/intermediates/runtime_library_classes_jar/release/classes.jar:com/blockstream/libgreenaddress/GDKJNI$NotificationHandler.class, /home/vagrant/build/com.greenaddress.greenbits_android_wallet/green/build/intermediates/javac/productionRelease/classes/com/blockstream/libgreenaddress/GDKJNI$NotificationHandler.class

> Task :green:lintVitalAnalyzeProductionRelease

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':green:minifyProductionReleaseWithR8'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.R8Task$R8Runnable
   > Compilation failed to complete, origin: /home/vagrant/build/com.greenaddress.greenbits_android_wallet/gdk/build/intermediates/runtime_library_classes_jar/release/classes.jar:com/blockstream/libgreenaddress/GDKJNI$NotificationHandler.class

full log: @angelix com.greenaddress.greenbits_android_wallet_22000418.log.gz

Great, those logs gave me full understanding on what's going on.

Seems that you both build and download gdk binaries.

If you still want to build gdk binaries from source, i'll get you a solution in a bit.

licaon-kter commented 11 months ago

Also

> Configure project :common

Please wait while Kotlin/Native compiler 1.9.20 is being installed.
Download https://download.jetbrains.com/kotlin/native/builds/releases/1.9.20/linux-x86_64/kotlin-native-prebuilt-linux-x86_64-1.9.20.tar.gz (198.16 MB)
Download kotlin-native-prebuilt-linux-x86_64-1.9.20.tar.gz finished, took 18 s 452 ms
Unpack Kotlin/Native compiler to /home/vagrant/.konan/kotlin-native-prebuilt-linux-x86_64-1.9.20
Unpack Kotlin/Native compiler to /home/vagrant/.konan/kotlin-native-prebuilt-linux-x86_64-1.9.20 finished, took 16 s 138 ms

...is a big no-no, ref: https://gitlab.com/fdroid/fdroiddata/-/issues/2989

Alternatively, you could try downloading the K/N distribution from maven central: https://central.sonatype.com/artifact/org.jetbrains.kotlin/kotlin-native-prebuilt

Would this be possible?

angelix commented 11 months ago

Alternatively, you could try downloading the K/N distribution from maven central: https://central.sonatype.com/artifact/org.jetbrains.kotlin/kotlin-native-prebuilt

Would this be possible?

Using kotlin.native.distribution.downloadFromMaven=true i got the following. So i guess we are ok on that.

Please wait while Kotlin/Native compiler 1.9.20 is being installed.
Downloading https://repo.maven.apache.org/maven2/org/jetbrains/kotlin/kotlin-native-prebuilt/1.9.20/kotlin-native-prebuilt-1.9.20-macos-aarch64.tar.gz to /Users/angelos/.gradle/.tmp/gradle_download11606821103669116292bin
<==-----------> 20% CONFIGURING [15s]
angelix commented 11 months ago 
> Task :green:mergeProductionReleaseJavaResource

> Task :green:minifyProductionReleaseWithR8 FAILED
ERROR: /home/vagrant/build/com.greenaddress.greenbits_android_wallet/gdk/build/intermediates/runtime_library_classes_jar/release/classes.jar: R8: Type com.blockstream.libgreenaddress.GDKJNI$NotificationHandler is defined multiple times: /home/vagrant/build/com.greenaddress.greenbits_android_wallet/gdk/build/intermediates/runtime_library_classes_jar/release/classes.jar:com/blockstream/libgreenaddress/GDKJNI$NotificationHandler.class, /home/vagrant/build/com.greenaddress.greenbits_android_wallet/green/build/intermediates/javac/productionRelease/classes/com/blockstream/libgreenaddress/GDKJNI$NotificationHandler.class

> Task :green:lintVitalAnalyzeProductionRelease

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':green:minifyProductionReleaseWithR8'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.R8Task$R8Runnable
   > Compilation failed to complete, origin: /home/vagrant/build/com.greenaddress.greenbits_android_wallet/gdk/build/intermediates/runtime_library_classes_jar/release/classes.jar:com/blockstream/libgreenaddress/GDKJNI$NotificationHandler.class

I think the issue is in the build recipe file.

prepare_gdk_clang.sh should be executed inside gdk folder.

Please try with this:

build:
      - $$rustup$$/rustup-init.sh -y
      - source $HOME/.cargo/env
      - rustup install 1.68.0
      - rustup target add aarch64-linux-android armv7-linux-androideabi
      - cd ../gdk
      - JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::") PATH=$HOME/.local/bin:$PATH
        ./prepare_gdk_clang.sh arm64-v8a
      - JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::") PATH=$HOME/.local/bin:$PATH
        ./prepare_gdk_clang.sh armeabi-v7a
angelix commented 11 months ago

@licaon-kter

With kotlin.native.distribution.downloadFromMaven=true if you want to test. https://github.com/Blockstream/green_android/tree/ave-fdroid

licaon-kter commented 11 months ago

Thanks https://gitlab.com/fdroid/fdroiddata/-/commit/6fb621107e948d13d0c6c482a66daf7dc4e5879f

angelix commented 11 months ago

@licaon-kter Thank you for efforts and time.

licaon-kter commented 10 months ago

new issue: https://gitlab.com/fdroid/fdroiddata/-/jobs/5807939772#L4517

/LE: fyi https://gitlab.com/fdroid/fdroiddata/-/commit/c686bd664fc83b6943f5768ff26b47b203e7d404

licaon-kter commented 10 months ago

Another issue: https://gitlab.com/fdroid/fdroiddata/-/jobs/5898148283#L3553

/LE: fyi https://gitlab.com/fdroid/fdroiddata/-/commit/4c604c6c31d74c9e40f77af551c76ddb90cd8d0f

licaon-kter commented 9 months ago

Another issue: https://gitlab.com/fdroid/fdroiddata/-/jobs/6123519265#L785 as https://github.com/Blockstream/green_android/blob/release_4.0.24/compose/build.gradle.kts#L76 is not-FOSS

/LE: fyi https://gitlab.com/fdroid/fdroiddata/-/commit/cd48f23c912f66b6298b57dd3e1108a925cb94d2

angelix commented 9 months ago

Another issue: https://gitlab.com/fdroid/fdroiddata/-/jobs/6123519265#L785 as https://github.com/Blockstream/green_android/blob/release_4.0.24/compose/build.gradle.kts#L76 is not-FOSS

/LE: fyi https://gitlab.com/fdroid/fdroiddata/-/commit/cd48f23c912f66b6298b57dd3e1108a925cb94d2

Will be removed on next version.

licaon-kter commented 9 months ago

Ping us when ready

angelix commented 8 months ago

@licaon-kter I added some metadata in #197. Can you confirm if the structure is ok? Thanks

angelix commented 8 months ago

@licaon-kter please check latest release

licaon-kter commented 7 months ago

I added some metadata in https://github.com/Blockstream/green_android/pull/197. Can you confirm if the structure is ok?

looks ok

please check latest release

will do asap

licaon-kter commented 7 months ago

Ummm

Please wait while Kotlin/Native compiler 1.9.22 is being installed.
Download kotlin-native-prebuilt-linux-x86_64-1.9.22.tar.gz finished, took 5 s 189 ms

better switch to Maven pls: https://central.sonatype.com/artifact/org.jetbrains.kotlin/kotlin-native-prebuilt ?

ref: https://gitlab.com/fdroid/fdroiddata/-/issues/2989

licaon-kter commented 7 months ago

I've added libc6-dev but still complains about pthreads, what am I missing? com.greenaddress.greenbits_android_wallet_22000426.log.gz

angelix commented 7 months ago 
Download kotlin-native-prebuilt-linux-x86_64-1.9.22.tar.gz finished, took 5 s 189 ms

Already using kotlin.native.distribution.downloadFromMaven=true in gradle.properties

licaon-kter commented 7 months ago

aaaand? :smile_cat:

angelix commented 7 months ago

aaaand? 😸

Still investigating the pthreads issue. We use bullseye (deps script for docker), maybe bookworm needs also another package.

licaon-kter commented 7 months ago

I've just started the build on my bullseye VM, brb

licaon-kter commented 7 months ago

Indeed, builds fine on Bullseye, still downloads kotlin binaries

angelix commented 7 months ago

Indeed, builds fine on Bullseye, still downloads kotlin binaries

Kotlin multiplatform binaries will always be downloaded as are not pre-packageed with Kotlin, but instead of being downloaded from jetbrains.com, are downloaded from public maven repository.

Also please be sure to use productionFDroid instead of production. In the future i want to enable certain features for production flavor, still safe to use it btw.

Lastly, NonFreeNet antifeature flags can be removed as the issues mentioned in https://github.com/Blockstream/green_android/issues/164 are fixed.

licaon-kter commented 7 months ago

Kotlin multiplatform binaries will always be downloaded as are not pre-packageed with Kotlin, but instead of being downloaded from jetbrains.com, are downloaded from public maven repository.

Ok, but... make it stop "Downloading from jetbrains"

angelix commented 7 months ago

From your logs

Please wait while Kotlin/Native compiler 1.9.22 is being installed.
Download kotlin-native-prebuilt-linux-x86_64-1.9.22.tar.gz finished, took 5 s 189 ms
Unpack Kotlin/Native compiler to /home/vagrant/.konan/kotlin-native-prebuilt-linux-x86_64-1.9.22
Unpack Kotlin/Native compiler to /home/vagrant/.konan/kotlin-native-prebuilt-linux-x86_64-1.9.22 finished, took 4 s 788 ms

From my machine with --info

Please wait while Kotlin/Native compiler 1.9.22 is being installed.
Downloading https://repo.maven.apache.org/maven2/org/jetbrains/kotlin/kotlin-native-prebuilt/1.9.22/kotlin-native-prebuilt-1.9.22-macos-aarch64.tar.gz to /Users/angelos/.gradle/.tmp/gradle_download1055476643572659025bin

Does it says jetbrains.com anywhere?

I think the kotlin.native.distribution.downloadFromMaven=true in gradle.properties solved that.

licaon-kter commented 7 months ago

Indeed, it might not, I don't have an actual URL in the logs.

Let me try something else, brb.

Any news on Bookworm?

angelix commented 7 months ago

Any news on Bookworm?

I have some issues trying to make it run in my M2 processor.

Did you try installing libevent-dev ?

licaon-kter commented 7 months ago

I'm not sure which dep is needed (what hinted you about that one?), but I'll test asap

angelix commented 7 months ago

I'm not sure which dep is needed (what hinted you about that one?), but I'll test asap

A message from gdk dev:

libevent has some reported issues with android build, it has been fixed, but fixes have not been released in a new version yet. Though, I wonder why on our CI works and in theirs doesn't

licaon-kter commented 7 months ago

Who is "our" and who is "their"? :)

angelix commented 7 months ago

Though, I wonder why on our (Blockstream's) CI works and in theirs (F-Droid) doesn't.

licaon-kter commented 7 months ago

So Blockstream (whatever that is?!) already builds in a Docker/VM with Bookworm?

Can you link to their setup?

angelix commented 7 months ago

I'll gather more info about that.