Open candroid-man opened 3 months ago
I was asked to establish the app with the IzzyOnDroid repo, which I'm currently doing (OuterTune will become available here with the next sync around 6 pm UTC). However, during the process some things came up. Scanners at IoD reported:
SigningBlock blobs:
-------------------
0x504b4453 (DEPENDENCY_INFO_BLOCK; GOOGLE)
which can easily be avoided with a minor addition to your build.greadle
:
android {
dependenciesInfo {
// Disables dependency metadata when building APKs.
includeInApk = false
// Disables dependency metadata when building Android App Bundles.
includeInBundle = false
}
}
For some background: that BLOB is supposed to be just a binary representation of your app's dependency tree. But as it's encrypted with a public key belonging to Google, only Google can read it – and nobody else can even verify what it really contains. More details can be found e.g. here: Ramping up security: additional APK checks are in place with the IzzyOnDroid repo.
Next, I've tried to establish Reproducible Builds – which failed as the APK was not built from the commit the release tag points to, but from a commit that's not even public: 7fbc86c9b5beeac9b9edfd667352825a40e7bfa6 (according to the META-INF/version-control-info.textproto
embedded in the APK). We've created some hints on reproducible builds which you might wish to take a look at – the culprit here is what we call the "First Basic Rule", as it's the cause for 80%+ of RB failures. We gladly assist you with establishing RBs, which are an additional security feature (also see Reproducible Builds, special client support and more at IzzyOnDroid for more details); if you can provide us with an APK built from a clean tree at a public commit, we can check again if that was the only culprit here. Currently, this is what the APK diff looks like:
-rw-r--r-- 0.0 unx 56 b- 52 defN 1981-01-01 01:01:02 b89973a3 META-INF/com/android/build/gradle/app-metadata.properties
- -rw-r--r-- 0.0 unx 120 b- 117 defN 1981-01-01 01:01:02 4e64fedf META-INF/version-control-info.textproto
+ -rw-r--r-- 0.0 unx 120 b- 117 defN 1981-01-01 01:01:02 470ee0f1 META-INF/version-control-info.textproto
-rw-r--r-- 0.0 unx 6028 b- 6028 stor 1981-01-01 01:01:02 6c3717f4 assets/dexopt/baseline.prof
-rw-r--r-- 0.0 unx 788 b- 788 stor 1981-01-01 01:01:02 4f15aed6 assets/dexopt/baseline.profm
-rw-r--r-- 0.0 unx 6339036 b- 2969104 defN 1981-01-01 01:01:02 e69f3e79 classes.dex
@@ -12,7 +12,7 @@
-rw-r--r-- 0.0 unx 518504 b- 518504 stor 1981-01-01 01:01:02 430213ea lib/arm64-v8a/libavformat.so
-rw-r--r-- 0.0 unx 710744 b- 710744 stor 1981-01-01 01:01:02 6233337e lib/arm64-v8a/libavutil.so
-rw-r--r-- 0.0 unx 7112 b- 7112 stor 1981-01-01 01:01:02 4f56ac38 lib/arm64-v8a/libdatastore_shared_counter.so
- -rw-r--r-- 0.0 unx 7864 b- 7864 stor 1981-01-01 01:01:02 9cb8ff23 lib/arm64-v8a/libffprobejni.so
+ -rw-r--r-- 0.0 unx 7864 b- 7864 stor 1981-01-01 01:01:02 7cf52308 lib/arm64-v8a/libffprobejni.so
-rw-r--r-- 0.0 unx 98536 b- 98536 stor 1981-01-01 01:01:02 724fe413 lib/arm64-v8a/libswresample.so
-rw-r--r-- 0.0 unx 608920 b- 608920 stor 1981-01-01 01:01:02 60f436e6 lib/arm64-v8a/libswscale.so
-rw---- 2.0 fat 1738 b- 782 defN 1981-01-01 01:01:02 2d73be70 DebugProbesKt.bin
@@ -102,10 +102,10 @@
-rw---- 2.0 fat 5 b- 7 defN 1981-01-01 01:01:02 6a2f0318 META-INF/kotlinx_coroutines_guava.version
-rw---- 2.0 fat 5 b- 7 defN 1981-01-01 01:01:02 6a2f0318 META-INF/kotlinx_coroutines_jdk8.version
-rw---- 2.0 fat 5 b- 7 defN 1981-01-01 01:01:02 6a2f0318 META-INF/kotlinx_coroutines_slf4j.version
- -rw---- 2.0 fat 6 b- 8 defN 1981-01-01 01:01:02 5de27171 META-INF/services/H4.e
- -rw---- 2.0 fat 52 b- 48 defN 1981-01-01 01:01:02 195cfebe META-INF/services/o4.g
- -rw---- 2.0 fat 6 b- 8 defN 1981-01-01 01:01:02 0b70b92a META-INF/services/r5.B
- -rw---- 2.0 fat 6 b- 8 defN 1981-01-01 01:01:02 09360773 META-INF/services/s5.a
+ -rw---- 2.0 fat 5 b- 7 defN 1981-01-01 01:01:02 1b649675 META-INF/services/H4.e
+ -rw---- 2.0 fat 51 b- 47 defN 1981-01-01 01:01:02 a1c62a19 META-INF/services/o4.g
+ -rw---- 2.0 fat 5 b- 7 defN 1981-01-01 01:01:02 3a68ee40 META-INF/services/r5.B
+ -rw---- 2.0 fat 5 b- 7 defN 1981-01-01 01:01:02 1145bd83 META-INF/services/s5.a
-rw---- 2.0 fat 625 b- 282 defN 1981-01-01 01:01:02 eb65be30 kotlin-tooling-metadata.json
META-INF/version-control-info.textproto
: as described above, caused by the APK being built from a different commit and thus easy to fix from your side.META-INF/services/*
: you obviously build on Windows, as those files have Windows line endings (\r\n
) – while we build on Linux and thus those files have Linux/Unix line endings (\n
) here. Easy to fix on our end, so nothing you need to be concerned by.lib/arm64-v8a/libffprobejni.so
: might be due to the different commit, or due to a different NDK version being used (please name yours for the retry then) – or by something else we cannot yet tell. A build from a clean tree (so no local changes, no caching) to compare against would help finding out.Looking forward to your reply there – and as I wrote, gladly assisting you establish Reproducible Builds!
@IzzySoft it seems the only difference between the two libffprobejni.so
files is the embedded ELF build ID. So adding arguments += listOf("-DCMAKE_SHARED_LINKER_FLAGS=-Wl,--build-id=none")
to the externalNativeBuild.cmake
block in ffMetadataEx/build.gradle.kts
should fix that.
I totally forgot about those – thanks Fay! That should then be added here in the app's repo – @DD3Boh could you please take care for that?
Hi. I made the listed changes, could you see if that helped? Apk is built off of dev branch: app-universal-release.zip
I see the download only serves the arm64 version(?), the universal variant may be more preferred (it supports x86, x86_64, armeabi-v7a, arm64-v8a)
lib/arm64-v8a/libffprobejni.so: might be due to the different commit, or due to a different NDK version being used (please name yours for the retry then) – or by something else we cannot yet tell. A build from a clean tree (so no local changes, no caching) to compare against would help finding out.
We include the FFMpeg libraries (FFMpeg binaries + JNI) in a prebuilt arr library stored on this repo, and by default it builds with that, however I opt to drop the arr and build the JNI wrapper part alongside the rest of the app for releases. The full process how these libraries are built is documented here: https://github.com/DD3Boh/OuterTune/blob/dev/CONTRIBUTING.md#building-with-ffmpeg-binaries .The app I attached above is built with the prebuilt.
I'm not sure how you'd want to handle this, but this is how its currently setup.
@IzzySoft @obfusk
Apk is built off of dev branch
Thanks! We'd need the exact commit (for our build).
I see the download only serves the arm64 version
Yes. The IzzyOnDroid repo has a per-app size limit of 30 MB, so we needed to pick one ABI to allow for multiple versions staying available. We could see to make an exception here as long as the added size does not go too much beyond: currently it would mean 33 MB for 2 versions, which would be acceptable. But we'd then drop to 1 version should it grow too much.
I opt to drop the arr and build the JNI wrapper part alongside the rest of the app
Building from source is always preferred, yes :wink: For the details on this part, I'd rather wait for Fay's opinion – she's the specialist there.
@IzzySoft looks like you should be able to build from source with these additions to the recipe:
sed -r '/implementation.*\.aar/d' -i app/build.gradle.kts
sed -r '/(include|project).*:ffMetadataEx/ s!^//!!' -i settings.gradle.kts app/build.gradle.kts
We'd need the exact commit (for our build).
APK says 8bc443ca1ea996783bd8ca60e76df78b2e204a62 (which looks correct to me since that commit introduced the changes and there is no later one yet).
Build fails with that:
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':ffMetadataEx:buildCMakeRelWithDebInfo[arm64-v8a]'.
> com.android.ide.common.process.ProcessException: ninja: Entering directory `/build/repo/ffMetadataEx/.cxx/RelWithDebInfo/186w6z2g/arm64-v8a'
C++ build system [build] failed while executing:
/opt/sdk/cmake/3.22.1/bin/ninja \
-C \
/build/repo/ffMetadataEx/.cxx/RelWithDebInfo/186w6z2g/arm64-v8a \
ffmetaexjni
from /build/repo/ffMetadataEx
ninja: error: '/build/repo/ffMetadataEx/src/main/cpp/ffmpeg-android-maker/output/lib/arm64-v8a/libavformat.so', needed by '/build/repo/ffMetadataEx/src/main/cpp/src/main/jniLibs/arm64-v8a/libffmetaexjni.so', missing and no known rule to make it
Same for the other 3 archs.
I must admit I don't understand what that second sed
command does, is that maybe related?
OK, without that it's RB:
build:
- git checkout 8bc443ca1ea996783bd8ca60e76df78b2e204a62
- chmod +x gradlew
- ./gradlew assembleUniversalRelease
- git clone -b v0.2.8 https://github.com/obfusk/reproducible-apk-tools.git
- OUT=app/build/outputs/apk/universal/release/app-universal-release-unsigned.apk
- reproducible-apk-tools/inplace-fix.py --internal --page-size 16 fix-newlines "$OUT" 'META-INF/services/*'
@IzzySoft it's failing because ffmpeg binaries are missing. Resolve them as stated here: https://github.com/DD3Boh/OuterTune/blob/dev/CONTRIBUTING.md#building-with-ffmpeg-binaries
But as a tl;dr: if you want to build if fully from source
/ffMetadataEx/src/main/cpp/ffmpeg-android-maker
The file structure should look something like this
in directory ffMetadataEx
|
\---src
\---main
+---cpp
| | CMakeLists.txt
| | ffMetaExJni.c
| |
| +---ffmpeg-android-maker
| | | README.md
| | |
| | +---output
| | | +---include
| | | | +---arm64-v8a
| | | | | +---libavcodec
| | | | | | ac3_parser.h
| | | | | | adts_parser.h
| | | | | | ...
| | | | | |
| | | | | +---libavdevice
| | | | | | avdevice.h
...
| | | | | ...
| | | | | |
| | | | | \---libswscale
| | | | | swscale.h
| | | | | version.h
| | | | | version_major.h
| | | | |
| | | | +---armeabi-v7a
| | | | | ...
| | | | |
| | | | +--- etc...
| | | |
| | | \---lib
| | | +---arm64-v8a
| | | | libavcodec.so
| | | | libavdevice.so
| | | | ...
| | | |
| | | +---armeabi-v7a
| | | | ...
| | | |
| | | +---x86
| | | | ...
| | | |
| | | \---x86_64
| | | ...
| | |
| | \---scripts
| | \---ffmpeg
| | build.sh
| |
\---java
\---com
\---dd3boh
\---ffMetadataEx
FFMpegWrapper.kt
Sorry if this is a bit complicated, I'm not familiar with making gradle handle this automatically. I could look into packaging this as a standalone lib though
Yes. The IzzyOnDroid repo has a per-app size limit of 30 MB
universal release variant is ~17mb and supports x86, x86_64, armeabi-v7a, arm64-v8a. You can use this one. I'll keep size in mind for the future
Also it would be cool to get a "get it on Obtainium" link here on the GitHub.
PR #46.
universal release variant is ~17mb and supports x86, x86_64, armeabi-v7a, arm64-v8a. You can use this one. I'll keep size in mind for the future
That would mean to only have a single version at IoD (as 17 x 2 is already beyond 30 MB) – while with the arm64 we can keep 3 versions. An "ARM only" variant without the x86 ABIs would at least allow for 2 versions and could be a compromise.
Apart from that: v0.6.2 is RB now, just added it to my builder. Thanks!
@IzzySoft I see. We can probably just keep just arm64 then, probably fine for 99% of users anyway
I've followed the instructions concerning ffmpeg to build v0.6.3 as RB, and succeeded. But a question rises:
"Clone ffmpeg-android-maker", "Clone prebuilt ffmpeg-android-maker" – neither specifies a tag or commit. How can we ensure a later verification will still succeed when those repositories moved on? Options coming to mind:
Example ffMetadataEx/ffMetaSource.json
(just brainstorming):
{
"prebuild": {
"repo": "https://github.com/mikooomich/ffmpeg-android-maker-prebuilt",
"commit": "cec82814122d52c4f4e4de92c78d1a21c9c66079"
},
"source": {
"repo": "https://github.com/Javernaut/ffmpeg-android-maker",
"commit": "a64992647e06d8b2c480f65dd2a0b3df0f27834f"
}
}
Build recipes could then extract the corresponding data, e.g.
- ffmpeg_source="$(cat ffMetadataEx/ffMetaSource.json | jq -r '.source.repo')"
- ffmpeg_commit="$(cat ffMetadataEx/ffMetaSource.json | jq -r '.source.commit')"
- git clone $ffmpeg_source ffMetadataEx/src/main/cpp/ffmpeg-android-maker
- pushd ffMetadataEx/src/main/cpp/ffmpeg-android-maker
- git checkout $ffmpeg_commit
- ...
What do you think, @mikooomich & @DarkCrypt?
@IzzySoft I do plan on migrating to [taglib] (https://github.com/Kyant0/taglib) (if feasible) and should avoid all of this mess all together, and ffmpeg could be a seperate optional plugin.
I'll let you know when that comes to be
@IzzySoft I only chimed in because of the request to add the Obtainium button to the README. The rest of the conversation is irrelevant for my purpose.
Checklist
Feature description
I'm sure this is on the radar, but I thought it wouldn't hurt to track the progress. Adding OuterTune to F-Droid and perhaps even Accrescent would be very useful. Also it would be cool to get a "get it on Obtainium" link here on the GitHub.
Why do you want this feature?
Cause
Additional information
No response