DD3Boh / OuterTune

A Material 3 Music Player with YouTube Music support for Android. Forked from InnerTune
GNU General Public License v3.0
352 stars 23 forks source link

F-Droid, Accrescent, and Obtainium #23

Open candroid-man opened 3 months ago

candroid-man commented 3 months ago

Checklist

Feature description

I'm sure this is on the radar, but I thought it wouldn't hurt to track the progress. Adding OuterTune to F-Droid and perhaps even Accrescent would be very useful. Also it would be cool to get a "get it on Obtainium" link here on the GitHub.

Why do you want this feature?

Cause

Additional information

No response

IzzySoft commented 2 months ago

I was asked to establish the app with the IzzyOnDroid repo, which I'm currently doing (OuterTune will become available here with the next sync around 6 pm UTC). However, during the process some things came up. Scanners at IoD reported:

SigningBlock blobs:
-------------------
0x504b4453 (DEPENDENCY_INFO_BLOCK; GOOGLE)

which can easily be avoided with a minor addition to your build.greadle:

android {
    dependenciesInfo {
        // Disables dependency metadata when building APKs.
        includeInApk = false
        // Disables dependency metadata when building Android App Bundles.
        includeInBundle = false
    }
}

For some background: that BLOB is supposed to be just a binary representation of your app's dependency tree. But as it's encrypted with a public key belonging to Google, only Google can read it – and nobody else can even verify what it really contains. More details can be found e.g. here: Ramping up security: additional APK checks are in place with the IzzyOnDroid repo.

Next, I've tried to establish Reproducible Builds – which failed as the APK was not built from the commit the release tag points to, but from a commit that's not even public: 7fbc86c9b5beeac9b9edfd667352825a40e7bfa6 (according to the META-INF/version-control-info.textproto embedded in the APK). We've created some hints on reproducible builds which you might wish to take a look at – the culprit here is what we call the "First Basic Rule", as it's the cause for 80%+ of RB failures. We gladly assist you with establishing RBs, which are an additional security feature (also see Reproducible Builds, special client support and more at IzzyOnDroid for more details); if you can provide us with an APK built from a clean tree at a public commit, we can check again if that was the only culprit here. Currently, this is what the APK diff looks like:

  -rw-r--r--  0.0 unx       56 b-       52 defN 1981-01-01 01:01:02 b89973a3 META-INF/com/android/build/gradle/app-metadata.properties
- -rw-r--r--  0.0 unx      120 b-      117 defN 1981-01-01 01:01:02 4e64fedf META-INF/version-control-info.textproto
+ -rw-r--r--  0.0 unx      120 b-      117 defN 1981-01-01 01:01:02 470ee0f1 META-INF/version-control-info.textproto
  -rw-r--r--  0.0 unx     6028 b-     6028 stor 1981-01-01 01:01:02 6c3717f4 assets/dexopt/baseline.prof
  -rw-r--r--  0.0 unx      788 b-      788 stor 1981-01-01 01:01:02 4f15aed6 assets/dexopt/baseline.profm
  -rw-r--r--  0.0 unx  6339036 b-  2969104 defN 1981-01-01 01:01:02 e69f3e79 classes.dex
@@ -12,7 +12,7 @@
  -rw-r--r--  0.0 unx   518504 b-   518504 stor 1981-01-01 01:01:02 430213ea lib/arm64-v8a/libavformat.so
  -rw-r--r--  0.0 unx   710744 b-   710744 stor 1981-01-01 01:01:02 6233337e lib/arm64-v8a/libavutil.so
  -rw-r--r--  0.0 unx     7112 b-     7112 stor 1981-01-01 01:01:02 4f56ac38 lib/arm64-v8a/libdatastore_shared_counter.so
- -rw-r--r--  0.0 unx     7864 b-     7864 stor 1981-01-01 01:01:02 9cb8ff23 lib/arm64-v8a/libffprobejni.so
+ -rw-r--r--  0.0 unx     7864 b-     7864 stor 1981-01-01 01:01:02 7cf52308 lib/arm64-v8a/libffprobejni.so
  -rw-r--r--  0.0 unx    98536 b-    98536 stor 1981-01-01 01:01:02 724fe413 lib/arm64-v8a/libswresample.so
  -rw-r--r--  0.0 unx   608920 b-   608920 stor 1981-01-01 01:01:02 60f436e6 lib/arm64-v8a/libswscale.so
  -rw----     2.0 fat     1738 b-      782 defN 1981-01-01 01:01:02 2d73be70 DebugProbesKt.bin
@@ -102,10 +102,10 @@
  -rw----     2.0 fat        5 b-        7 defN 1981-01-01 01:01:02 6a2f0318 META-INF/kotlinx_coroutines_guava.version
  -rw----     2.0 fat        5 b-        7 defN 1981-01-01 01:01:02 6a2f0318 META-INF/kotlinx_coroutines_jdk8.version
  -rw----     2.0 fat        5 b-        7 defN 1981-01-01 01:01:02 6a2f0318 META-INF/kotlinx_coroutines_slf4j.version
- -rw----     2.0 fat        6 b-        8 defN 1981-01-01 01:01:02 5de27171 META-INF/services/H4.e
- -rw----     2.0 fat       52 b-       48 defN 1981-01-01 01:01:02 195cfebe META-INF/services/o4.g
- -rw----     2.0 fat        6 b-        8 defN 1981-01-01 01:01:02 0b70b92a META-INF/services/r5.B
- -rw----     2.0 fat        6 b-        8 defN 1981-01-01 01:01:02 09360773 META-INF/services/s5.a
+ -rw----     2.0 fat        5 b-        7 defN 1981-01-01 01:01:02 1b649675 META-INF/services/H4.e
+ -rw----     2.0 fat       51 b-       47 defN 1981-01-01 01:01:02 a1c62a19 META-INF/services/o4.g
+ -rw----     2.0 fat        5 b-        7 defN 1981-01-01 01:01:02 3a68ee40 META-INF/services/r5.B
+ -rw----     2.0 fat        5 b-        7 defN 1981-01-01 01:01:02 1145bd83 META-INF/services/s5.a
  -rw----     2.0 fat      625 b-      282 defN 1981-01-01 01:01:02 eb65be30 kotlin-tooling-metadata.json

Looking forward to your reply there – and as I wrote, gladly assisting you establish Reproducible Builds!

obfusk commented 2 months ago

@IzzySoft it seems the only difference between the two libffprobejni.so files is the embedded ELF build ID. So adding arguments += listOf("-DCMAKE_SHARED_LINKER_FLAGS=-Wl,--build-id=none") to the externalNativeBuild.cmake block in ffMetadataEx/build.gradle.kts should fix that.

IzzySoft commented 2 months ago

I totally forgot about those – thanks Fay! That should then be added here in the app's repo – @DD3Boh could you please take care for that?

mikooomich commented 2 months ago

Hi. I made the listed changes, could you see if that helped? Apk is built off of dev branch: app-universal-release.zip

I see the download only serves the arm64 version(?), the universal variant may be more preferred (it supports x86, x86_64, armeabi-v7a, arm64-v8a)

lib/arm64-v8a/libffprobejni.so: might be due to the different commit, or due to a different NDK version being used (please name yours for the retry then) – or by something else we cannot yet tell. A build from a clean tree (so no local changes, no caching) to compare against would help finding out.

We include the FFMpeg libraries (FFMpeg binaries + JNI) in a prebuilt arr library stored on this repo, and by default it builds with that, however I opt to drop the arr and build the JNI wrapper part alongside the rest of the app for releases. The full process how these libraries are built is documented here: https://github.com/DD3Boh/OuterTune/blob/dev/CONTRIBUTING.md#building-with-ffmpeg-binaries .The app I attached above is built with the prebuilt.

I'm not sure how you'd want to handle this, but this is how its currently setup.

@IzzySoft @obfusk

IzzySoft commented 2 months ago

Apk is built off of dev branch

Thanks! We'd need the exact commit (for our build).

I see the download only serves the arm64 version

Yes. The IzzyOnDroid repo has a per-app size limit of 30 MB, so we needed to pick one ABI to allow for multiple versions staying available. We could see to make an exception here as long as the added size does not go too much beyond: currently it would mean 33 MB for 2 versions, which would be acceptable. But we'd then drop to 1 version should it grow too much.

I opt to drop the arr and build the JNI wrapper part alongside the rest of the app

Building from source is always preferred, yes :wink: For the details on this part, I'd rather wait for Fay's opinion – she's the specialist there.

obfusk commented 2 months ago

@IzzySoft looks like you should be able to build from source with these additions to the recipe:

sed -r '/implementation.*\.aar/d' -i app/build.gradle.kts
sed -r '/(include|project).*:ffMetadataEx/ s!^//!!' -i settings.gradle.kts app/build.gradle.kts

We'd need the exact commit (for our build).

APK says 8bc443ca1ea996783bd8ca60e76df78b2e204a62 (which looks correct to me since that commit introduced the changes and there is no later one yet).

IzzySoft commented 2 months ago

Build fails with that:

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':ffMetadataEx:buildCMakeRelWithDebInfo[arm64-v8a]'.
> com.android.ide.common.process.ProcessException: ninja: Entering directory `/build/repo/ffMetadataEx/.cxx/RelWithDebInfo/186w6z2g/arm64-v8a'

  C++ build system [build] failed while executing:
      /opt/sdk/cmake/3.22.1/bin/ninja \
        -C \
        /build/repo/ffMetadataEx/.cxx/RelWithDebInfo/186w6z2g/arm64-v8a \
        ffmetaexjni
    from /build/repo/ffMetadataEx
  ninja: error: '/build/repo/ffMetadataEx/src/main/cpp/ffmpeg-android-maker/output/lib/arm64-v8a/libavformat.so', needed by '/build/repo/ffMetadataEx/src/main/cpp/src/main/jniLibs/arm64-v8a/libffmetaexjni.so', missing and no known rule to make it

Same for the other 3 archs.

I must admit I don't understand what that second sed command does, is that maybe related?

IzzySoft commented 2 months ago

OK, without that it's RB:

        build:
          - git checkout 8bc443ca1ea996783bd8ca60e76df78b2e204a62
          - chmod +x gradlew
          - ./gradlew assembleUniversalRelease
          - git clone -b v0.2.8 https://github.com/obfusk/reproducible-apk-tools.git
          - OUT=app/build/outputs/apk/universal/release/app-universal-release-unsigned.apk
          - reproducible-apk-tools/inplace-fix.py --internal --page-size 16 fix-newlines "$OUT" 'META-INF/services/*'
mikooomich commented 2 months ago

@IzzySoft it's failing because ffmpeg binaries are missing. Resolve them as stated here: https://github.com/DD3Boh/OuterTune/blob/dev/CONTRIBUTING.md#building-with-ffmpeg-binaries

But as a tl;dr: if you want to build if fully from source

  1. Switch to self built instead of prebuilt (I assume you've done this already on your end, ex. this commit https://github.com/mikooomich/OuterTune/commit/897a306ec62f00927c62b7f6acbebf02d8dbf29c)
  2. clone https://github.com/mikooomich/ffmpeg-android-maker into /ffMetadataEx/src/main/cpp/ffmpeg-android-maker
  3. build the libs (ffmpeg-android-maker.sh)
  4. now you can proceed building outertune

The file structure should look something like this

in directory ffMetadataEx
|   
\---src
    \---main
        +---cpp
        |   |   CMakeLists.txt
        |   |   ffMetaExJni.c
        |   |   
        |   +---ffmpeg-android-maker
        |   |   |   README.md
        |   |   |   
        |   |   +---output
        |   |   |   +---include
        |   |   |   |   +---arm64-v8a
        |   |   |   |   |   +---libavcodec
        |   |   |   |   |   |       ac3_parser.h
        |   |   |   |   |   |       adts_parser.h
        |   |   |   |   |   |       ...
        |   |   |   |   |   |       
        |   |   |   |   |   +---libavdevice
        |   |   |   |   |   |       avdevice.h
                    ...
        |   |   |   |   |   ...
        |   |   |   |   |   |       
        |   |   |   |   |   \---libswscale
        |   |   |   |   |           swscale.h
        |   |   |   |   |           version.h
        |   |   |   |   |           version_major.h
        |   |   |   |   |           
        |   |   |   |   +---armeabi-v7a
        |   |   |   |   |   ...
        |   |   |   |   |           
        |   |   |   |   +--- etc...
        |   |   |   |               
        |   |   |   \---lib
        |   |   |       +---arm64-v8a
        |   |   |       |       libavcodec.so
        |   |   |       |       libavdevice.so
        |   |   |       |       ...
        |   |   |       |       
        |   |   |       +---armeabi-v7a
        |   |   |       |      ...
        |   |   |       |
        |   |   |       +---x86
        |   |   |       |       ...
        |   |   |       |       
        |   |   |       \---x86_64
        |   |   |               ...
        |   |   |               
        |   |   \---scripts
        |   |       \---ffmpeg
        |   |               build.sh
        |   |               
        \---java
            \---com
                \---dd3boh
                    \---ffMetadataEx
                            FFMpegWrapper.kt

Sorry if this is a bit complicated, I'm not familiar with making gradle handle this automatically. I could look into packaging this as a standalone lib though

mikooomich commented 2 months ago

Yes. The IzzyOnDroid repo has a per-app size limit of 30 MB

universal release variant is ~17mb and supports x86, x86_64, armeabi-v7a, arm64-v8a. You can use this one. I'll keep size in mind for the future

DarkCrypt commented 2 months ago

Also it would be cool to get a "get it on Obtainium" link here on the GitHub.

PR #46.

IzzySoft commented 1 month ago

universal release variant is ~17mb and supports x86, x86_64, armeabi-v7a, arm64-v8a. You can use this one. I'll keep size in mind for the future

That would mean to only have a single version at IoD (as 17 x 2 is already beyond 30 MB) – while with the arm64 we can keep 3 versions. An "ARM only" variant without the x86 ABIs would at least allow for 2 versions and could be a compromise.

Apart from that: v0.6.2 is RB now, just added it to my builder. Thanks!

mikooomich commented 1 month ago

@IzzySoft I see. We can probably just keep just arm64 then, probably fine for 99% of users anyway

IzzySoft commented 2 weeks ago

I've followed the instructions concerning ffmpeg to build v0.6.3 as RB, and succeeded. But a question rises:

"Clone ffmpeg-android-maker", "Clone prebuilt ffmpeg-android-maker" – neither specifies a tag or commit. How can we ensure a later verification will still succeed when those repositories moved on? Options coming to mind:

Example ffMetadataEx/ffMetaSource.json (just brainstorming):

{
  "prebuild": {
    "repo": "https://github.com/mikooomich/ffmpeg-android-maker-prebuilt",
    "commit": "cec82814122d52c4f4e4de92c78d1a21c9c66079"
  },
  "source": {
    "repo": "https://github.com/Javernaut/ffmpeg-android-maker",
    "commit": "a64992647e06d8b2c480f65dd2a0b3df0f27834f"
  }
}

Build recipes could then extract the corresponding data, e.g.

- ffmpeg_source="$(cat ffMetadataEx/ffMetaSource.json | jq -r '.source.repo')"
- ffmpeg_commit="$(cat ffMetadataEx/ffMetaSource.json | jq -r '.source.commit')"
- git clone $ffmpeg_source ffMetadataEx/src/main/cpp/ffmpeg-android-maker
- pushd ffMetadataEx/src/main/cpp/ffmpeg-android-maker
- git checkout $ffmpeg_commit
- ...

What do you think, @mikooomich & @DarkCrypt?

mikooomich commented 2 weeks ago

@IzzySoft I do plan on migrating to [taglib] (https://github.com/Kyant0/taglib) (if feasible) and should avoid all of this mess all together, and ffmpeg could be a seperate optional plugin.

I'll let you know when that comes to be

DarkCrypt commented 2 weeks ago

@IzzySoft I only chimed in because of the request to add the Obtainium button to the README. The rest of the conversation is irrelevant for my purpose.