F-Droid can't build - unknown maven, .AAR deps

licaon-kter commented 1 year ago

ref: https://gitlab.com/fdroid/fdroiddata/-/jobs/4238418967#L1990 ref: https://github.com/haiwen/seadroid/pull/971

These can't be found

Execution failed for task ':app:checkReleaseAarMetadata'.
> Could not resolve all files for configuration ':app:releaseRuntimeClasspath'.
   > Could not find com.blankj:utilcode:1.30.0.
     Searched in the following locations:
       - file:/home/vagrant/.m2/repository/com/blankj/utilcode/1.30.0/utilcode-1.30.0.pom
       - https://dl.google.com/dl/android/maven2/com/blankj/utilcode/1.30.0/utilcode-1.30.0.pom
       - https://repo.maven.apache.org/maven2/com/blankj/utilcode/1.30.0/utilcode-1.30.0.pom
       - https://jitpack.io/com/blankj/utilcode/1.30.0/utilcode-1.30.0.pom
       - https://oss.sonatype.org/content/repositories/snapshots/com/blankj/utilcode/1.30.0/utilcode-1.30.0.pom
     Required by:
         project :app
   > Could not find com.jcodecraeer:xrecyclerview:1.6.0.
     Searched in the following locations:
       - file:/home/vagrant/.m2/repository/com/jcodecraeer/xrecyclerview/1.6.0/xrecyclerview-1.6.0.pom
       - https://dl.google.com/dl/android/maven2/com/jcodecraeer/xrecyclerview/1.6.0/xrecyclerview-1.6.0.pom
       - https://repo.maven.apache.org/maven2/com/jcodecraeer/xrecyclerview/1.6.0/xrecyclerview-1.6.0.pom
       - https://jitpack.io/com/jcodecraeer/xrecyclerview/1.6.0/xrecyclerview-1.6.0.pom
       - https://oss.sonatype.org/content/repositories/snapshots/com/jcodecraeer/xrecyclerview/1.6.0/xrecyclerview-1.6.0.pom
     Required by:
         project :app
   > Could not find me.relex:circleindicator:1.3.2.
     Searched in the following locations:
       - file:/home/vagrant/.m2/repository/me/relex/circleindicator/1.3.2/circleindicator-1.3.2.pom
       - https://dl.google.com/dl/android/maven2/me/relex/circleindicator/1.3.2/circleindicator-1.3.2.pom
       - https://repo.maven.apache.org/maven2/me/relex/circleindicator/1.3.2/circleindicator-1.3.2.pom
       - https://jitpack.io/me/relex/circleindicator/1.3.2/circleindicator-1.3.2.pom
       - https://oss.sonatype.org/content/repositories/snapshots/me/relex/circleindicator/1.3.2/circleindicator-1.3.2.pom
     Required by:
         project :app
   > Could not find com.commit451:PhotoView:1.2.4.
     Searched in the following locations:
       - file:/home/vagrant/.m2/repository/com/commit451/PhotoView/1.2.4/PhotoView-1.2.4.pom
       - https://dl.google.com/dl/android/maven2/com/commit451/PhotoView/1.2.4/PhotoView-1.2.4.pom
       - https://repo.maven.apache.org/maven2/com/commit451/PhotoView/1.2.4/PhotoView-1.2.4.pom
       - https://jitpack.io/com/commit451/PhotoView/1.2.4/PhotoView-1.2.4.pom
       - https://oss.sonatype.org/content/repositories/snapshots/com/commit451/PhotoView/1.2.4/PhotoView-1.2.4.pom
     Required by:
         project :app
   > Could not find com.shuyu:gsyVideoPlayer-java:3.0.0.
     Searched in the following locations:
       - file:/home/vagrant/.m2/repository/com/shuyu/gsyVideoPlayer-java/3.0.0/gsyVideoPlayer-java-3.0.0.pom
       - https://dl.google.com/dl/android/maven2/com/shuyu/gsyVideoPlayer-java/3.0.0/gsyVideoPlayer-java-3.0.0.pom
       - https://repo.maven.apache.org/maven2/com/shuyu/gsyVideoPlayer-java/3.0.0/gsyVideoPlayer-java-3.0.0.pom
       - https://jitpack.io/com/shuyu/gsyVideoPlayer-java/3.0.0/gsyVideoPlayer-java-3.0.0.pom
       - https://oss.sonatype.org/content/repositories/snapshots/com/shuyu/gsyVideoPlayer-java/3.0.0/gsyVideoPlayer-java-3.0.0.pom
     Required by:
         project :app
   > Could not find com.shuyu:gsyVideoPlayer-ex_so:3.0.0.
     Searched in the following locations:
       - file:/home/vagrant/.m2/repository/com/shuyu/gsyVideoPlayer-ex_so/3.0.0/gsyVideoPlayer-ex_so-3.0.0.pom
       - https://dl.google.com/dl/android/maven2/com/shuyu/gsyVideoPlayer-ex_so/3.0.0/gsyVideoPlayer-ex_so-3.0.0.pom
       - https://repo.maven.apache.org/maven2/com/shuyu/gsyVideoPlayer-ex_so/3.0.0/gsyVideoPlayer-ex_so-3.0.0.pom
       - https://jitpack.io/com/shuyu/gsyVideoPlayer-ex_so/3.0.0/gsyVideoPlayer-ex_so-3.0.0.pom
       - https://oss.sonatype.org/content/repositories/snapshots/com/shuyu/gsyVideoPlayer-ex_so/3.0.0/gsyVideoPlayer-ex_so-3.0.0.pom
     Required by:
         project :app
   > Could not find com.yydcdut:markdown-processor:0.1.3.
     Searched in the following locations:
       - file:/home/vagrant/.m2/repository/com/yydcdut/markdown-processor/0.1.3/markdown-processor-0.1.3.pom
       - https://dl.google.com/dl/android/maven2/com/yydcdut/markdown-processor/0.1.3/markdown-processor-0.1.3.pom
       - https://repo.maven.apache.org/maven2/com/yydcdut/markdown-processor/0.1.3/markdown-processor-0.1.3.pom
       - https://jitpack.io/com/yydcdut/markdown-processor/0.1.3/markdown-processor-0.1.3.pom
       - https://oss.sonatype.org/content/repositories/snapshots/com/yydcdut/markdown-processor/0.1.3/markdown-processor-0.1.3.pom
     Required by:
         project :app
   > Could not find ren.qinc.edit:lib:0.0.5.
     Searched in the following locations:
       - file:/home/vagrant/.m2/repository/ren/qinc/edit/lib/0.0.5/lib-0.0.5.pom
       - https://dl.google.com/dl/android/maven2/ren/qinc/edit/lib/0.0.5/lib-0.0.5.pom
       - https://repo.maven.apache.org/maven2/ren/qinc/edit/lib/0.0.5/lib-0.0.5.pom
       - https://jitpack.io/ren/qinc/edit/lib/0.0.5/lib-0.0.5.pom
       - https://oss.sonatype.org/content/repositories/snapshots/ren/qinc/edit/lib/0.0.5/lib-0.0.5.pom
     Required by:
         project :app

...aren't they on these (trusted) mavens: https://gitlab.com/fdroid/fdroidserver/-/blob/6a239cbde46ae5443efb6c24a13e41e9910a015d/fdroidserver/scanner.py#L439-L458 ?

These libs can't be used like this: https://github.com/haiwen/seadroid/tree/master/app/libs where is their source code?

/LE: fyi https://gitlab.com/fdroid/fdroiddata/-/commit/6a1587ae6593165f027561feb49dc2e6e2f9822c

zhwanng commented 1 year ago

They are in Mavens, or you can add another Maven address: https://maven.aliyun.com/repository/public

Or check your network environment.

licaon-kter commented 1 year ago

That's not a trusted, open source hosting maven repo, as linked above.

licaon-kter commented 1 year ago

But, as said, since those are open source, we can just build them, right? Care to link to their soure code repos?

zhwanng commented 1 year ago

For some reason, we didn't do a massive refactoring, and in order to accommodate later versions, some third-party libraries were integrated into the project as local files (AARs) for continued use, and the resource bundles were located in the ./app/libs/ directory.

licaon-kter commented 1 year ago

Right, so, we should put the app on manual update mode until all deps are rebuilt from open source or available in one of the linked maven repos?

zhwanng commented 1 year ago

Some third-party libraries have not been updated for a long time, so we modified the source code of some third-party libraries and repackaged them (AAR), such as the markdown library. According to the current plan, this open source project is constantly being updated until it adapts to the latest version of the system.

zhwanng commented 1 year ago

For now, at least, you'll need to modify your automated build configuration.

licaon-kter commented 1 year ago

Disabled autoupdates for now: https://gitlab.com/fdroid/fdroiddata/-/commit/57eb615b17c92c56a29cc5a03dcc97ce1879a97f

Ping us when ready to update the recipe and build the needed deps.

woj-tek commented 1 year ago

Some third-party libraries have not been updated for a long time, so we modified the source code of some third-party libraries and repackaged them (AAR), such as the markdown library. According to the current plan, this open source project is constantly being updated until it adapts to the latest version of the system.

@zhwanng but would it be possible to publish the modified sources and not only pre-build AAR files? If those were opensource project, and you modified them then it shouldn't be a problem to publish them. That's the gist of the issue, at least to my understanding...

If those sources are not published then f-droid can't modify the build configuration to make it work (there is a requirement that dependency is either from trusted maven repo or sources are available, providing pre-build binary blob doesn't fullfil this requirement)

freeplant commented 1 year ago

We will look into the issue.

zhwanng commented 1 year ago

Starting from v2.3.5, the MarkdownView library is no longer imported into the project as AAR, and has been modified to integrate the open source library.

Open Source Projects: https://github.com/haiwen/markdownview

Open Source Libraries: https://jitpack.io/#haiwen/MarkdownView

freeplant commented 1 year ago

@licaon-kter Can you give it a check?

licaon-kter commented 1 year ago

I'm building https://github.com/zhwanng/seadroid/commit/d94176b18b95e09e4ff47c5b8a96063723b994bd now, this is the one, yes?

licaon-kter commented 1 year ago

recipe:

  - versionName: 2.3.5
    versionCode: 138
    commit: d94176b18b95e09e4ff47c5b8a96063723b994bd
    subdir: app
    sudo:
      - apt-get update
      - apt-get install -y maven
    gradle:
      - yes
    srclibs:
      - android-vcard@android-vcard-1.4
    prebuild:
      - echo -e "\norg.gradle.jvmargs=-Xms4g -Xmx16g -XX:MaxPermSize=16g -XX:-UseGCOverheadLimit"
        >> ../gradle.properties
      - find $$android-vcard$$ -name '*.jar' -delete
      - sed -i -e '/repositories/a\        mavenLocal()' -e '/aliyun/d' ../build.gradle
      - sed -i -e 's/implementation fileTree.*/implementation "com.googlecode.android-vcard:android-vcard:1.4"/'
        build.gradle
    build: mvn install -f $$android-vcard$$/pom.xml -Dmaven.test.skip=true

error log looks the same imho: https://gist.github.com/licaon-kter/ecb9d1ef3fefc53526b6b02e75e2cc4d

woj-tek commented 1 year ago

I quickly checked some of those libraries, and they seem to be available but not in those exact versions - quite often it's just a minor/bugfix version difference (which should be relatively easy adjustment without API changes)

zhwanng commented 1 year ago

"I checked the packages that caused errors during the build and found that they were not included in the default central repository of the project. However, when I used the Alibaba Cloud central repository, they were found and the project was built successfully. You can try adding 'maven { url "https://maven.aliyun.com/repository/public" }'."

licaon-kter commented 1 year ago

As my recipe above shows and the OP says, the current issue is about libs being on that untrusted maven repo only.

Are we lost in translation?

zhwanng commented 1 year ago

Maybe a translation issue is creating a gap, so maybe these libs versions are too low and the default repository is not synced？the following are trusted？https://maven.aliyun.com not trusted?


    allowed_repos = [re.compile(r'^https://' + re.escape(repo) + r'/*') for repo in [
        'repo1.maven.org/maven2',  # mavenCentral()
        'jcenter.bintray.com',     # jcenter()
        'jitpack.io',
        'www.jitpack.io',
        'repo.maven.apache.org/maven2',
        'oss.jfrog.org/artifactory/oss-snapshot-local',
        'oss.sonatype.org/content/repositories/snapshots',
        'oss.sonatype.org/content/repositories/releases',
        'oss.sonatype.org/content/groups/public',
        'oss.sonatype.org/service/local/staging/deploy/maven2',
        's01.oss.sonatype.org/content/repositories/snapshots',
        's01.oss.sonatype.org/content/repositories/releases',
        's01.oss.sonatype.org/content/groups/public',
        's01.oss.sonatype.org/service/local/staging/deploy/maven2',
        'clojars.org/repo',  # Clojure free software libs
        'repo.clojars.org',  # Clojure free software libs
        's3.amazonaws.com/repo.commonsware.com',  # CommonsWare
        'plugins.gradle.org/m2',  # Gradle plugin repo
        'maven.google.com',  # Google Maven Repo, https://developer.android.com/studio/build/dependencies.html#google-maven
        ]
    ] + [re.compile(r'^file://' + re.escape(repo) + r'/*') for repo in [
        '/usr/share/maven-repo',  # local repo on Debian installs
        ]
    ]

licaon-kter commented 1 year ago

Yes, so why aren't the libs in those maven repos updated? That's the issue?

woj-tek commented 1 year ago

Maybe a translation issue is creating a gap, so maybe these libs versions are too low and the default repository is not synced？the following are trusted？https://maven.aliyun.com not trusted?

No, this maven repository is not trusted. See https://f-droid.org/docs/Inclusion_Policy/ or https://f-droid.org/2022/07/22/maven-central.html. In general:

Trusted maven repository. While there is no guarantee that those binaries are free and correspondent to the source code, F-Droid allows the following known repositories currently:

Maven Central - the original repo, hardcoded in Maven and Gradle.

Google Maven Repo - hardcoded in Gradle, this repo hosts Google’s own libs.

JCenter - hardcoded in Gradle, this repo by Bintray tries to provide easier handling. It’s synced with Maven Central, and include some extra libs. It’s shutting down so please avoid this repo.

OSS Sonatype - maintained by the people behind mavenCentral, this repository focuses on hosting services for open source project binaries. It’s synced with Maven Central, and include some extra libs

OSS JFrog - maintained by the people behind jCenter, this repository focuses on hosting services for open source project binaries.

JitPack.io - builds directly from GitHub repositories. However, they do not provide any option to reproduce or verify the resulting binaries. Builds pre-release versions in some cases.

Clojars - Clojure libraries repo.

CommonsWare - repo holding a collection of open-source libs.

Gradle plugin repo - hardcoded in Gradle, this repo hosts Gradle plugins.

Thus you should make sure that any dependency that you use (artifact-id and it's version) is available in any of the above repositories. Alternatively you can point to the sources of the library that can be build and f-droid will build those themselves.

--

Having that in mind, I checked some of the dependndencies and you could try using versions from central, for example:

Could not find com.blankj:utilcode:1.30.0. -> https://mvnrepository.com/artifact/com.blankj/utilcode/1.30.7 (switch from 1.30.0 to 1.30.7)
Could not find com.commit451:PhotoView:1.2.4 -> https://mvnrepository.com/artifact/com.github.chrisbanes/PhotoView/1.2.5 (1.2.4 to 1.2.5)
Could not find com.shuyu:gsyVideoPlayer-java:3.0.0 -> https://mvnrepository.com/artifact/com.github.CarGuo.GSYVideoPlayer/gsyVideoPlayer-java
Could not find com.shuyu:gsyVideoPlayer-ex_so:3.0.0 -> https://mvnrepository.com/artifact/com.shuyu/gsyVideoPlayer-ex_so

etc…

zhwanng commented 1 year ago

Yes, I am considering using Trusted Maven. Going through the logs it turns out that these libraries need to be updated

com.blankj:utilcode:1.30.0
com.jcodecraeer:xrecyclerview:1.6.0
com.google.android.exoplayer:exoplayer-core:2.8.1
com.google.android.exoplayer:exoplayer-ui:2.8.1
me.relex:circleindicator:1.3.2
com.commit451:PhotoView:1.2.4
com.yydcdut:markdown-processor:0.1.3
ren.qinc.edit:lib:0.0.5

zhwanng commented 1 year ago

New PR : https://github.com/haiwen/seadroid/pull/981

please try build again

licaon-kter commented 1 year ago

First try... it failed because it wanted android.useAndroidX=true in gradle.properties

Ok, added that, now it fails with

> Task :app:processReleaseManifestForPackage

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:checkReleaseDuplicateClasses'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.CheckDuplicatesRunnable
   > Duplicate class android.support.v4.accessibilityservice.AccessibilityServiceInfoCompat found in modules support-compat-27.0.1-runtime (com.android.support:support-compat:27.0.1) and support-v4-22.2.1-runtime (com.android.support:support-v4:22.2.1)
....
many
many
more
of the same

woj-tek commented 1 year ago

@licaon-kter maybe provide simple steps to build it (to mimic how it's build for F-droid) so seafile team (@zhwanng) could build it on their own an iron out all kinks? :-)

licaon-kter commented 1 year ago

The recipe is known: https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/com.seafile.seadroid2.yml#L1282-L1301 :)

But here's how I tried:

  - versionName: 2.3.5
    versionCode: 138
    commit: 425f461a373a8ab79dedc34945f0ade801e88403
    subdir: app
    sudo:
      - apt-get update
      - apt-get install -y maven
    gradle:
      - yes
    srclibs:
      - android-vcard@android-vcard-1.4
    prebuild:
      - echo -e "android.useAndroidX=true" > ../gradle.properties
      - echo -e "\norg.gradle.jvmargs=-Xms4g -Xmx16g -XX:MaxPermSize=16g -XX:-UseGCOverheadLimit"
        >> ../gradle.properties
      - find $$android-vcard$$ -name '*.jar' -delete
      - sed -i -e '/repositories/a\        mavenLocal()' ../build.gradle
      - sed -i -e 's/implementation fileTree.*/implementation "com.googlecode.android-vcard:android-vcard:1.4"/'
        build.gradle
    build: mvn install -f $$android-vcard$$/pom.xml -Dmaven.test.skip=true

Here's more context of the error:

> Task :app:checkReleaseAarMetadata
WARNING:Your project has set `android.useAndroidX=true`, but configuration `:app:releaseRuntimeClasspath` still contains legacy support libraries, which may cause runtime issues.
This behavior will not be allowed in Android Gradle plugin 8.0.
Please use only AndroidX dependencies or set `android.enableJetifier=true` in the `gradle.properties` file to migrate your project to AndroidX (see https://developer.android.com/jetpack/androidx/migrate for more info).
The following legacy support libraries are detected:
:app:releaseRuntimeClasspath -> com.github.chrisbanes:PhotoView:2.0.0 -> com.android.support:support-core-utils:27.0.1
:app:releaseRuntimeClasspath -> com.github.haiwen:MarkdownView:0.19.1 -> com.android.support:appcompat-v7:27.0.1 -> com.android.support:support-annotations:27.0.1
:app:releaseRuntimeClasspath -> com.github.getActivity:XXPermissions:16.2 -> com.android.support:support-fragment:27.0.1 -> com.android.support:support-compat:27.0.1
:app:releaseRuntimeClasspath -> com.github.getActivity:XXPermissions:16.2 -> com.android.support:support-fragment:27.0.1 -> com.android.support:support-compat:27.0.1 -> android.arch.lifecycle:runtime:1.0.0
:app:releaseRuntimeClasspath -> com.github.getActivity:XXPermissions:16.2 -> com.android.support:support-fragment:27.0.1 -> com.android.support:support-compat:27.0.1 -> android.arch.lifecycle:runtime:1.0.0 -> android.arch.lifecycle:common:1.0.0
:app:releaseRuntimeClasspath -> com.github.getActivity:XXPermissions:16.2 -> com.android.support:support-fragment:27.0.1 -> com.android.support:support-compat:27.0.1 -> android.arch.lifecycle:runtime:1.0.0 -> android.arch.core:common:1.0.0
:app:releaseRuntimeClasspath -> com.joanzapata.iconify:android-iconify-material-community:2.2.1 -> com.joanzapata.iconify:android-iconify:2.2.1 -> com.android.support:support-v4:22.2.1
:app:releaseRuntimeClasspath -> com.github.getActivity:XXPermissions:16.2 -> com.android.support:support-fragment:27.0.1
:app:releaseRuntimeClasspath -> com.github.getActivity:XXPermissions:16.2 -> com.android.support:support-fragment:27.0.1 -> com.android.support:support-core-ui:27.0.1
:app:releaseRuntimeClasspath -> com.github.haiwen:MarkdownView:0.19.1 -> com.android.support:appcompat-v7:27.0.1
:app:releaseRuntimeClasspath -> com.github.haiwen:MarkdownView:0.19.1 -> com.android.support:appcompat-v7:27.0.1 -> com.android.support:support-vector-drawable:27.0.1
:app:releaseRuntimeClasspath -> com.github.haiwen:MarkdownView:0.19.1 -> com.android.support:appcompat-v7:27.0.1 -> com.android.support:animated-vector-drawable:27.0.1

...I guess

/LE: full log: com.seafile.seadroid2_138.log.gz

zhwanng commented 1 year ago

🆗 👀

zhwanng commented 1 year ago

https://github.com/haiwen/seadroid/pull/983

please try again

licaon-kter commented 1 year ago

Yay success, can anyone test the APK? com.seafile.seadroid2_138.apk.zip

(remove the .zip in name)

zhwanng commented 1 year ago

@licaon-kter I tested it and it worked fine😃

licaon-kter commented 1 year ago

Great, so release as usual and it will be picked up ;)

woj-tek commented 1 year ago

Yay! Awesome! Fingers crossed for new release 😇 :D

woj-tek commented 1 year ago

@licaon-kter will f-droid server pick up new release or because it's disabled it won't and requires explicit PR to remove disabled metadata?

licaon-kter commented 1 year ago

That version is disabled, yes, scroll down to AutoUpdateMode, if it's not None... it will or better said it should if no other issue arises :)

woj-tek commented 1 year ago

Reference: AutoUpdateMode.

So, currently it's None so it won't be picked up automatically thus either PR with new version and/or PR which would change AutoUpdateMode to Version? :-)

(I assumed it would be OK considering yesterday's "Yay success" :-)

licaon-kter commented 1 year ago

Yup, I've looked at an older version of the file by mistake and that was Version :facepalm:

/LE: fixed https://gitlab.com/fdroid/fdroiddata/-/commit/0a8b822d9bfb302dd525d11acfa7653bcf82281b

/LE2: autoupdate works: https://gitlab.com/fdroid/fdroiddata/-/jobs/4804397321#L2237 so I added it manually already: https://gitlab.com/fdroid/fdroiddata/-/commit/87e2ae1a9b63c5eb5063e748540090cb909e7285

woj-tek commented 1 year ago

autoupdate works: https://gitlab.com/fdroid/fdroiddata/-/jobs/4804397321#L2237

But this build failed? :-)

I added it manually already: https://gitlab.com/fdroid/fdroiddata/-/commit/87e2ae1a9b63c5eb5063e748540090cb909e7285

This one seem to be build correctly (https://gitlab.com/fdroid/fdroiddata/-/pipelines/955677281) though I don't see the version updated. Is there another knob that needs to be nudged? :-)

licaon-kter commented 1 year ago

Should be in this cycle, so bookmark https://monitor.f-droid.org/builds and keep an eye on running or build ;)

licaon-kter commented 1 year ago

Built, https://monitor.f-droid.org/builds/log/com.seafile.seadroid2/138#site-footer waiting for cycle to end and be published

woj-tek commented 1 year ago

Available for update :-)

haiwen / seadroid

F-Droid can't build - unknown maven, .AAR deps #972