F-Droid build failed - Githubissues

linsui commented 2 years ago

"Install Android SDK Platform 31 (revision: 1)" complete.
"Install Android SDK Platform 31 (revision: 1)" finished.
Build-tool 31.0.0 is missing DX at /opt/android-sdk/build-tools/31.0.0/dx

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine the dependencies of task ':aTalk:compileFdroidReleaseJavaWithJavac'.
> Installed Build Tools revision 31.0.0 is corrupted. Remove and install again using the SDK Manager.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 18s

Could you please take a look? Thanks!

cmeng-git commented 2 years ago

Please try the following solution i.e. Renaming d8.jar & d8.bat to dx.jar and dx.bat respectively

The problem seems to come from AS

https://stackoverflow.com/questions/68390830/android-studio-4-2-2-installed-build-tools-revision-31-0-0-is-corrupted-remov

linsui commented 2 years ago

We builds apk with gradle and sdkmanager directly. We got the same issue when building telegram foss. See https://gitlab.com/fdroid/fdroiddata/-/merge_requests/9687#note_667008552 . Maybe we need to use a newer AGP?

cmeng-git commented 2 years ago

I was facing the same problem when migrate to use build-tools = 31.0.0. I renamed d8.jar to dx.jar to resolve the problem.

The d8.jar is resided in the /opt/android-sdk/build-tools/31.0.0/lib directory in my case. I think you need to find the build-tools/31.0.0/lib in your SDE to change the file d8.jar to dx.jar.

linsui commented 2 years ago

That should work but a little hack. Would you consider fixing it properly in the next version?

cmeng-git commented 2 years ago

Had tried the following based on the given links; https://developer.android.com/studio/releases/gradle-plugin?hl=hi&skip_cache=true https://android-developers.googleblog.com/2017/08/next-generation-dex-compiler-now-in.html https://stackoverflow.com/questions/45648215/how-to-enable-dex-compiler-d8next-generation-dex-compiler-in-android-studio

However it still does not resolve the problem. Seems that the problem lied in android SDE i.e. AS/SDKManager configuration. Rename seems to be the only patch that works currently.

linsui commented 2 years ago

Got it. Thanks!

cmeng-git commented 2 years ago

Today I just perform a build for another project, found that the latest android studio needs both the dx.jar and d8.jar to be present; else failed with the following message. I propose you keep the original d8.jar file and copy the dx.jar from an earlier sdk releases e.g. ./android-sdk/build-tools/30.0.3/dx.jar to ./android-sdk/build-tools/31.0.0/dx.jar

I am not sure whether reinstall of ./android-sdk/build-tools/31.0.0 will also fix the problem.

AGPBI: {"kind":"warning","text":"An API level of 31 is not supported by this compiler. Please use an API level of 30 or earlier","sources":[{}],"tool":"D8"}

linsui commented 2 years ago

I copy both dx and dx.jar. It builds fine, thanks!

licaon-kter commented 1 year ago

@cmeng-git 2.9.6 fails with https://gitlab.com/fdroid/fdroiddata/-/jobs/2732928417#L1354

cmeng-git commented 1 year ago

Did you change the built environment of on a new linux machine. Seems like the machine has missing command wget.

./init_libs_ffmpeg_x264.sh: line 41: wget: command not found

Please install wget and try again.

### Building ffmpeg-x264 jni library ###
========== init libraries for: ffmpeg (4.4.2) and x264 (163) ==========
========== Updating ffmpeg source: ffmpeg (4.4.2) ==========
./init_libs_ffmpeg_x264.sh: line 41: wget: command not found
bzip2: Compressed file ends unexpectedly;
    perhaps it is corrupted?  *Possible* reason follows.
bzip2: Inappropriate ioctl for device
    Input file = (stdin), output file = (stdout)
It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.
You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.
tar: Child returned status 2
tar: Error is not recoverable: exiting now

linsui commented 1 year ago

Ah, wget is installed on the buildserver somehow. @licaon-kter Could you please check on your buildserver? Thanks!

cmeng-git commented 1 year ago

Please pull in the new aTalk v2.9.6 repository again before build. I have updated some files for quick release but keeping the version.

licaon-kter commented 1 year ago

@linsui there's a slight difference between the CI (that failed above) and main buildserver, and not the first time when wget appears missing. Afaik the buildserver has it. Will test locally.

licaon-kter commented 1 year ago

@cmeng-git for the future, wget: command not found should be a fail case, the build should error out at this stage.

Please pull in the new aTalk v2.9.6 repository again before build. I have updated some files for quick release but keeping the version.

9c10c6d68fbc41392099072f6b22f1826563c5d2 was already tried, same error: https://gitlab.com/fdroid/fdroiddata/-/jobs/2738358851

licaon-kter commented 1 year ago

@linsui https://gitlab.com/fdroid/fdroidserver/-/blob/master/buildserver/provision-apt-get-install#L58

Tested locally and builds fine, reinstated the update and added wget as a dep since once the upgrade to Bullseye is complete this will be needed.

ref: https://gitlab.com/fdroid/fdroiddata/-/commit/218fd7517a00c381e7bc7f8828ac5c97b5e82c9c

linsui commented 1 year ago

wget is not there but it is used in https://gitlab.com/fdroid/fdroidserver/-/blob/master/fdroidserver/mirror.py#L39. I guess that's why it's not installed on the CI.

cmeng-git commented 1 year ago

Closed, problem is due to build environment setup errors.

licaon-kter commented 1 year ago

@cmeng-git one small request, can you make https://github.com/cmeng-git/atalk-android/blob/master/aTalk/jni/static_library_built/build-jnilibs4atalk.sh exit with a hard error on any build fail?

Eg. on the new builseye environment make is not installed by default any more, so openssl failed to build.... but the script carried on only to fail at the end when suddenly libcrypt.a was missing.

/LE: Another example: https://gitlab.com/fdroid/fdroiddata/-/jobs/3204203487#L624

cmeng-git commented 1 year ago

I just made a simulation of missing 'make' command; found that the build script does not throw any failure but just report missing make command and proceed.

To ensure aTalk can build on any user system, aTalk needs to perform a pre-check for the system build environment setup for all commands used in aTalk scripts. This is no practical. I would expect the developer should ensure the system use to build any app must ensure it has the proper environment setup before proceed.

Please refer to aTalk apk build for android I am sorry I have to decline your request.

**********************************************************************
***                                                                ***
***   OpenSSL has been successfully configured                     ***
***                                                                ***
***   If you encounter a problem while building, please open an    ***
***   issue on GitHub <https://github.com/openssl/openssl/issues>  ***
***   and include the output from the following command:           ***
***                                                                ***
***       perl configdata.pm --dump                                ***
***                                                                ***
***   (If you are new to OpenSSL, you might want to consult the    ***
***   'Troubleshooting' section in the INSTALL file first)         ***
***                                                                ***
**********************************************************************
+ make -j16
./build-openssl4android.sh: line 60: make: command not found
+ popd
~/workspace/android/atalk-android/aTalk/jni/static_library_built/openssl
+ echo -e '** BUILD COMPLETED: openssl (1.1.1q) for armeabi-v7a **\n\n'
** BUILD COMPLETED: openssl (1.1.1q) for armeabi-v7a **

linsui commented 1 year ago

https://monitor.f-droid.org/builds/log/org.atalk.android/301040#site-footer It failed due to a network issue but continue building... Could you please make this a hard error? It's difficult to find the real error.

cmeng-git commented 1 year ago

It is not appropriate for aTalk to create hard failure on your environment setup/behaviour. Fdroid py scripts can check for these type of errors e.g. the source must be downloaded and exist before proceed with the build.

All aTalk build scripts are designed to be manually executed offline to generate the native libraries. Fdroid needs to take care of any fallout if it wants to automate this process. In all aTalk releases, the following three native libraries are released together with the source i.e. openssl, vpx and ffmpeg-x264. Fdroid can use them as it without rebuild these libraries. Many of the android app just use the prebuilt libraries without a local build e.g. libwebrtc-m104.aar is used by Conversations as it.

Currently you need only to run the below task to download libOpus and libSpeex libraries.

task getJniLibs(type: Exec) {
    workingDir "${projectDir}/jni/"
    executable './fetch-jnilibs4atalk.sh'
}

These hopefully should create less problem for Fdroid to auto build aTalk.

linsui commented 1 year ago

We can't use prebuilt binaries. But maybe we can insert set -e into these scripts.

licaon-kter commented 1 year ago

@cmeng-git

e.g. libwebrtc-m104.aar is used by Conversations as it.

Never was... https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/eu.siacs.conversations.yml#L2027-L2029

it's not that simple, see https://gitlab.com/fdroid/fdroiddata/-/issues/2064

Newer ones use the artefacts from the dev indeed: https://codeberg.org/iNPUTmice/Conversations/src/tag/2.12.2/build.gradle#L79 for the reasons above

Also https://f-droid.org/en/2022/07/22/maven-central.html

And starting from "Though we tried to build everyting from source..." https://f-droid.org/en/docs/Inclusion_Policy/

licaon-kter commented 1 year ago

And another fail: org.atalk.android_301051.log.gz

/LE: fyi https://gitlab.com/fdroid/fdroiddata/-/commit/a5bd7b5e8ff8ffa068a3bd944a5659a9e5512449

linsui commented 1 year ago

** BUILD STARTED: x264-v164 for armeabi-v7a **
Makefile:3: config.mak: No such file or directory
cat: config.h: No such file or directory
./configure
No working C compiler found.
make: *** [config.mak] Error 1

licaon-kter commented 1 year ago

Which one is needed here, gcc? Other stuff built fine, eg. openssl.

/LE: gcc does it, thanks

licaon-kter commented 1 year ago

But maybe we can insert set -e into these scripts.

It makes it exit too soon for this harmless errors: make: *** No rule to make target 'clean'. Stop.

:(

cmeng-git commented 1 year ago

With reference to org.atalk.android_301051.log.gz:

** BUILD STARTED: x264-v164 for armeabi-v7a **
Makefile:3: config.mak: No such file or directory
cat: config.h: No such file or directory
./configure
No working C compiler found.
make: *** [config.mak] Error 1

You may ignore the above error. It seems the config.h is generated during the build process. If you refer to the log file near the statement below:

BUILD COMPLETED: x264-v164 for armeabi-v7a

The static library file libx264.a is successfully get generated. I have no idea why android ndk shows this error.

Towards the end of the log file: FAILURE: Build failed with an exception.

What went wrong:

Execution failed for task ':aTalk:generateJsonModelFdroidRelease'.
> error when building with ndkBuild using /home/vagrant/build/org.atalk.android/aTalk/jni/Android.mk: Build command failed.
Error while executing process /opt/android-sdk/ndk/20.0.5594570/ndk-build with arguments {NDK_PROJECT_PATH=null APP_BUILD_SCRIPT=/home/vagrant/build/org.atalk.android/aTalk/jni/Android.mk NDK_APPLICATION_MK=/home/vagrant/build/org.atalk.android/aTalk/jni/Application.mk APP_ABI=armeabi-v7a NDK_ALL_ABIS=armeabi-v7a NDK_DEBUG=0 APP_PLATFORM=android-21 NDK_OUT=/home/vagrant/build/org.atalk.android/aTalk/build/intermediates/ndkBuild/fdroidRelease/obj NDK_LIBS_OUT=/home/vagrant/build/org.atalk.android/aTalk/build/intermediates/ndkBuild/fdroidRelease/lib -Cjni APP_SHORT_COMMANDS=false LOCAL_SHORT_COMMANDS=false -B -n}
Android NDK: ERROR:opensles/Android.mk:avcodec: LOCAL_SRC_FILES points to a missing file    
Android NDK: Check that ffmpeg/android/armeabi-v7a/lib/libavcodec.a exists  or that its path is correct   
make: Entering directory `/home/vagrant/build/org.atalk.android/aTalk/jni'
make: Leaving directory `/home/vagrant/build/org.atalk.android/aTalk/jni'

Cannot under the above error message: opensles/Android.mk does not specify nor use the source

ffmpeg/android/armeabi-v7a/lib/libavcodec.a

Which one is needed here, gcc? Other stuff built fine, eg. openssl. /LE: gcc does it, thanks

If you refer to statitic_library_built/x264/_setting.sh, it is mentioned there i.e.:

if [[ ${STANDALONE_TOOLCHAINS} == 1 ]]; then
  TOOLCHAIN_PREFIX=${BASEDIR}/toolchain-android
  NDK_SYSROOT=${TOOLCHAIN_PREFIX}/sysroot
  CC_=clang
  CXX_=clang++

  if [[ ! -e ${TOOLCHAIN_PREFIX}/${NDK_ABIARCH} ]]; then
    rm -rf ${TOOLCHAIN_PREFIX}

  # Create standalone toolchains for the specified architecture - use .py instead of the old .sh
  # However for ndk--r19b => Instead use:
  #    $ ${NDK}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android21-clang++ src.cpp
  # cmeng: must ensure AS JNI uses the same STL library or "system" if specified
    [[ -d ${TOOLCHAIN_PREFIX} ]] || python3 ${NDK}/build/tools/make_standalone_toolchain.py \
      --arch ${NDK_ARCH} \
      --api ${ANDROID_API} \
      --stl libc++ \
      --install-dir=${TOOLCHAIN_PREFIX}
  fi
else
  TOOLCHAIN_PREFIX=${ANDROID_NDK}/toolchains/${NDK_ABIARCH}-49/prebuilt/linux-x86_64
  NDK_SYSROOT=${ANDROID_NDK}/platforms/android-${ANDROID_API}/arch-${NDK_ARCH}
  CC_=gcc
  CXX_=g++
fi

Not quite understand your comment below. I see that this message in the log, but the build processs continue without stopping?

Which one is needed here, gcc? Other stuff built fine, eg. openssl.

/LE: gcc does it, thanks

licaon-kter commented 1 year ago

No, after installing gcc it built fine.

If you refer to statitic_library_built/x264/_setting.sh, it is mentioned there i.e.:

Reading that I understand that it should need GCC only IF clang is not found? So there's an issue with the toolchain?

licaon-kter commented 1 year ago

...
[armeabi-v7a] Compile thumb  : jnbcg729 <= JNIDecoder.c

make: Leaving directory `/home/vagrant/build/org.atalk.android/aTalk/jni'

In file included from g729/JNIDecoder.c:17:

g729/JNIDecoder.h:3:10: fatal error: '/opt/android/android-sdk/ndk/20.0.5594570/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/jni.h' file not found

#include "/opt/android/android-sdk/ndk/20.0.5594570/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/jni.h"

^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1 error generated.

make: *** [/home/vagrant/build/org.atalk.android/aTalk/build/intermediates/ndkBuild/fdroidRelease/obj/local/armeabi-v7a/objs/jnbcg729/JNIDecoder.o] Error 1

from https://monitor.f-droid.org/builds/log/org.atalk.android/302010#site-footer

@cmeng-git do we need to bump ndk or?

/LE: fyi https://gitlab.com/fdroid/fdroiddata/-/commit/02a684439c5a05876c42525b144ab9be97aa4074

linsui commented 1 year ago

Fixed in https://gitlab.com/fdroid/fdroiddata/-/merge_requests/13340.

licaon-kter commented 1 year ago

wow, odd place to hardcode that path https://github.com/cmeng-git/atalk-android/blob/3.2.1/aTalk/jni/g729/JNIDecoder.h#L3

cmeng-git commented 1 year ago

My apology, the hard coded link was added in by accident. I have re-release 3.2.1 with the link removed.

linsui commented 1 year ago

Updated. https://gitlab.com/fdroid/fdroiddata/-/merge_requests/13344 Thanks!

cmeng-git / atalk-android

F-Droid build failed #185