[Android] [F-Droid] Current Status of LibRetro F-Doird Repository

ghost commented 3 years ago

Not a bug report, just asking if LibRetro F-Droid Repo is deprecated. The last update dates back to last November while IIRC it should be nightly release like the one in buildbot.

BTW, I notice there is RetroArch Plus on Play Store as mentioned in LibRetro news. If F-Droid repo is not reprecated, would there be two versions of RetroArch in it?

I see there are #4247, #6446 and #7940 so I assmue this is the right place to ask the question.

licaon-kter commented 1 year ago

Thanks for testing, I'm not aware on how this should work (can the assets be bundled in the APK or is their license problematic so we resort to downloading?).

hizzlekizzle commented 1 year ago

licenses on assets should be fine/compatible. Can your recipe fetch them from a URL at bundle-time?

licaon-kter commented 1 year ago

Maybe, where are they and why aren't they in the repo here?

hizzlekizzle commented 1 year ago

They're all in here: http://buildbot.libretro.com/assets/frontend/ We don't include them in the main RetroArch repo for a few reasons, including that they're frequently their own thing managed by individuals outside of RetroArch itself (e.g., I manage the shaders, which are used by a few different programs outside of RA), and adding a few hundred MBs of images to every repo clone would be a real drag, among other reasons.

Or, if you'd rather collect them all separately from the various github repos instead of our collections on the buildbot, you can see my snap recipe, which does just that: https://github.com/libretro/retroarch-snap/blob/master/snapcraft.yaml#L219

IzzySoft commented 1 year ago

adding a few hundred MBs of images to every repo clone would be a real drag

Then maybe git submodules could work for this?

licaon-kter commented 1 year ago

Well, it's not like the APK can't be huge... but does it need to? :)

Vinfall commented 1 year ago

is making a f-droid repo auto update the stable really that complicated ? ~~sorry if i sound rude <3

As many people wondering why it takes so much time waiting, I tried to set up a self-hosted RetroArch nightly repo and it's surprising easy. Basically it's just a few commands to init the repo & write a script w/ crobjob to automate the update.

For anyone interested:

Sample repo: you can scan https://github.com/Vinfall/fdroid-server/blob/main/repo/index.png or add https://raw.githubusercontent.com/Vinfall/fdroid-server/main/repo?fingerprint=C3A7B8145FB872B67C48E7BA9E6A079E582DDCF5AAFD00F6143E91B8F45187F6 to F-Droid client to see the result. Please note there is actually no app hosted as it will abuse GitHub & I don't plan to maintain it.
My blog with some details about setting it up

For RetroArch team, if you are interested in setting up a nightly repo (hopefully it can be integrated into your CI!), I'm willing to help🤟🏼

licaon-kter commented 1 year ago

Any news here? I mean... if the tested APK is okay we can merge already, fyi ;)

Vinfall commented 1 year ago

I think the test apk is fine, see https://github.com/libretro/RetroArch/issues/11922#issuecomment-1274470419.

There are only two minor issues dangling:

Changelog (located in https://github.com/libretro/RetroArch/blob/master/CHANGES.md but it's huge, my suggestion: https://github.com/libretro/RetroArch/issues/11922#issuecomment-1274317270)
Bundle assets or not (located in https://buildbot.libretro.com/assets/frontend/assets.zip but it's huge as well (100M+), relevant snap recipe. I don't have a preference as I'll update assets after installation anyway)

Personally I want to see RetroArch on F-Droid sooner, but it's still up to the developers.

hizzlekizzle commented 1 year ago

for the assets, if it works okay with just the 'glui' assets (i.e., the ones for the default android menu), we can save a lot of space by just shipping with it. That leaves Android TV / Shield users without assets, but whatever.

For the changelog, can the recipe just do something like 'head 25 CHANGES.md > $release_hash.txt' or whatever?

Vinfall commented 1 year ago

For the changelog, can the recipe just do something like 'head 25 CHANGES.md > $release_hash.txt' or whatever?

I get this command to output the latest changelog:

cat CHANGES.md | sed --silent "/\# ${new_version}/,/\# /p" | head --lines -2

\# ${new_version} matches the latest version, which can be replaced by something like versionName in the recipe?)
\# matches the last version heading (Markdown H1)
head removes the last two lines (last version heading, as I don't know how to do non-greedy match in sed...)

@hizzlekizzle But it also output changelog on other platforms, is that acceptable?

There are too many tags to exclude and I have no idea where to start. Simply grep ANDROID would exclude other improvements like PLAYLISTS, INPUT, LOCALIZATION etc.

hizzlekizzle commented 1 year ago

I think it's fine to include everything, yeah.

Vinfall commented 1 year ago

For the assets, if it works okay with just the 'glui' assets (i.e., the ones for the default android menu), we can save a lot of space by just shipping with it. That leaves Android TV / Shield users without assets, but whatever.

From RetroArch settings it seems that assets are located in private app directory (/data/user). So I can't help much about this without rebuilding the app with glui assets myself.

@licaon-kter Is it possible to run another build with changelog and glui assets bundled (it's right inside assets.zip, just unarchive it and there is a glui folder)? As there is already a new release, I guess the recipe needs to change anyway.

denilsonsa commented 1 year ago

cat CHANGES.md | sed --silent "/\# ${new_version}/,/\# /p" | head --lines -2
[…]

head removes the last two lines (last version heading, as I don't know how to do non-greedy match in sed...)

You can try something like this (untested):

sed --silent "
  /^# ${new_version}/,/\# / {
    /^# ${new_version}/p;  # ← optional
    /^# /d;
    p
  }" CHANGES.md

The optional line will include the "new version" header in the output. Then, the following line will delete any header in that range, then it will print any remaining lines. You can also eliminate the newlines and keep everything in one line if needed (but then it's less readable).

Vinfall commented 1 year ago

For the assets, if it works okay with just the 'glui' assets (i.e., the ones for the default android menu), we can save a lot of space by just shipping with it. That leaves Android TV / Shield users without assets, but whatever.

From RetroArch settings it seems that assets are located in private app directory (/data/user). So I can't help much about this without rebuilding the app with glui assets myself.

I managed to get it working by changing the Assets directory in the setting to external storage like /storage/emulated/0/fdroidtest/assets and placed extracted glui assets in the folder. All icons seem to work this time. So I assume bundling glui assets would be enough for F-Droid packaging (it's just 1.19MB zipped!).

licaon-kter commented 1 year ago

Regarding the changelog, this just got merged so maybe it's easier to setup? https://gitlab.com/fdroid/fdroidserver/-/merge_requests/1240

licaon-kter commented 1 year ago

@Vinfall which commit I should build for glui ?

Vinfall commented 1 year ago

Not sure since I'm not a contributor and know little about Android packaging💔. Also I don't know how does recipe fetch them from a URL at bundle-time would work as was mentioned in https://github.com/libretro/RetroArch/issues/11922#issuecomment-1275068095. You'd better ask hizzlekizzle or LibretroAdmin instead.

AFAIK from release tag https://github.com/libretro/RetroArch/tree/v1.12.0, it should be https://github.com/libretro/RetroArch/commit/840c4481ab7cb0720025c30ea609d16e60927a02.

hizzlekizzle commented 1 year ago

Yes, that's the commit. For the assets, if you can just fetch the whole bundle with the recipe and then rm -f !(assets/glui) or whatever to nuke everything but the GLUI icons, we should be good. And it sounds like the recipe can just grab those first few lines from the CHANGES.md and dump them into default.txt now, right?

licaon-kter commented 1 year ago

Re: default.txt, not really, it expects them to be there on the publish phase, so somehow maybe your release workflow can create it before tagging.

hizzlekizzle commented 1 year ago

oh ok. gotcha. Yeah, I would think that's probably doable. @LibretroAdmin what do you think? Can you run a script to dump (or just manually copy/paste) the applicable changelog lines into a file named "default.txt" for the fdroid changelog? That will avoid needing to know the release hash/ID.

Vinfall commented 1 year ago

Can you run a script to dump (or just manually copy/paste) the applicable changelog lines into a file named "default.txt" for the fdroid changelog? That will avoid needing to know the release hash/ID.

What needs to change for this? I can raise a PR if the team is busy. That is, if I know where to start.

I can achieve this via https://github.com/libretro/RetroArch/issues/11922#issuecomment-1292866779, but that's Linux-specific and I don't know how the building process work (along with the right location for the script, or default.txt).

hizzlekizzle commented 1 year ago

It could probably be added to the regular Android build process, if you have any familiarity with that.

Vinfall commented 1 year ago

Not so much, sadly. How about GitHub Action? I notice that there are quite a few CI in your workflows, can I add one more to automatically dump the changelog when CHANGES.md is changed or there is a new tagged release? This can be easily done for me and doesn't require manual check once set up, so everyone should be happy!

hizzlekizzle commented 1 year ago

Yes, as long as fdroid is pulling from an archive to which that could be attached rather than just doing a 'git pull' somewhere, I think a github action should work just fine.

Vinfall commented 1 year ago

There are only two minor issues dangling:

Changelog (located in https://github.com/libretro/RetroArch/blob/master/CHANGES.md but it's huge, my suggestion: [Android] [F-Droid] Current Status of LibRetro F-Doird Repository #11922 (comment))

Bundle assets or not (located in https://buildbot.libretro.com/assets/frontend/assets.zip but it's huge as well (100M+), relevant snap recipe. I don't have a preference as I'll update assets after installation anyway)

Personally I want to see RetroArch on F-Droid sooner, but it's still up to the developers.

Hi, now that v1.14.0 is out with default.txt in place, we can proceed. Building the APK with glui bundled should be enough, see quoted comment for the right place and remember to minimize the size by sth. like rm -f !(assets/glui) in the recipe.

licaon-kter commented 1 year ago

Will update recipe asap

licaon-kter commented 1 year ago

Updated recipe and APK to test, load some cores, play some games, report back, if it's okay I'll open the MR: https://gitlab.com/fdroid/rfp/-/issues/1933#note_1209330350

Vinfall commented 1 year ago

I tested the APK and it seems that the glui assets are missing. Is it fetched upon building? If not, you can try to add the assets into the recipe like the prebuild lines (not sure how the recipe works):

Builds:
  - versionName: 1.14.0
    versionCode: 1597175255
    commit: ad89b0c655fc1d25adfcdf40268e95c5d0391111
    ...
    prebuild:
      # ↓ These two lines
      - curl -S https://buildbot.libretro.com/assets/frontend/assets.zip --output assets.zip
      - unzip assets.zip 'glui/*' -d 'assets' && rm assets.zip
      # ↑ These two lines
      - sed -i -e '/versionCode System/d' -e '/versionName "${/d' build.gradle
    ndk: r22

This would fetch the assets and place it under ./assets/glui, is this the right place? @hizzlekizzle

hizzlekizzle commented 1 year ago

I'm not super-familiar with the Android build process, but if anything in the working directory (./) gets bundled, yeah, assets/glui is the proper structure.

licaon-kter commented 1 year ago

I'll have a look at the license/contents of the assets next ond bundle them if ok.

licaon-kter commented 1 year ago

https://buildbot.libretro.com/assets/frontend/assets.zip

Umm, unversioned file is odd... so basically if I build today I get a set of assets, if I build tomorrow I might get another?

licaon-kter commented 1 year ago

@Vinfall

- unzip assets.zip 'glui/*' -d 'assets' && rm assets.zip

So we download 110Mb, extract ONLY 2Mb (glui) and delete the rest? why?

Vinfall commented 1 year ago

Umm, unversioned file is odd... so basically if I build today I get a set of assets, if I build tomorrow I might get another?

Yes, this is because assets are somewhat like rolling-release I guess? They should come from https://github.com/libretro/retroarch-assets and RetroArch clients on other platforms simply dump the archive as well (see this snap recipe). You can change the url to https://github.com/libretro/retroarch-assets/archive/master.tar.gz but that doesn't have regular tagged versions either.

So we download 110Mb, extract ONLY 2Mb (glui) and delete the rest? why?

Yes. I think it comes down to two points:

Glui is required for basic icon rendering, which means it's essential to make the app behave normally. The downloading step is required due to the fact that the assets do not come with https://github.com/libretro/RetroArch but locates at https://github.com/libretro/retroarch-assets.
The whole assets are large and bundling them all does not offer much benefit. Most importantly, users can update the assets inside the app themselves. Deleting the rest makes sure the app size is reasonable and saving F-Droid bandwidth.

Actually it's brought up in https://github.com/libretro/RetroArch/issues/11922#issuecomment-1275261274, if F-Droid is satisfied with a huge apk and can handle the traffic, I think not deleting the other assets is fine.

Vinfall commented 1 year ago

I'll have a look at the license/contents of the assets next ond bundle them if ok.

From https://github.com/libretro/retroarch-assets/blob/master/COPYING, it should be CC-BY-4.0 license.

licaon-kter commented 1 year ago

https://github.com/libretro/retroarch-assets/archive/master.tar.gz but that doesn't have regular tagged versions either.

But maybe we/you can specify the exact commit that the project is using exactly? eg https://buildbot.libretro.com/stable/1.14.0/android/RetroArch_aarch64.apk is using what?

Glui is required for basic icon rendering, which means it's essential to make the app behave normally.

There were no reports that my test APKs are broken, so not sure how this works.

The whole assets are large and bundling them all does not offer much benefit.

The project site APKs are >180Mb: https://retroarch.com/?page=platforms so why not slim those down too then?

Speaking of those, F-Droid likes to have APKs reproducible if possible so having the APK contents match your own is a goal.

licaon-kter commented 1 year ago

From https://github.com/libretro/retroarch-assets/blob/master/COPYING, it should be CC-BY-4.0 license.

Yup, saw it later.

if I build tomorrow I might get another?

wow, one hour difference and it changed :( I wanted to put the hash sum of the file so we can check it before built, that's not feasible now

licaon-kter commented 1 year ago

@Vinfall

unzip assets.zip 'glui/*' -d 'assets'

I forgot to ask, where (relative to the repo root) does the assets folder needs to be? media/assets/ ? media/ ?

Vinfall commented 1 year ago

But maybe we/you can specify the exact commit that the project is using exactly? eg https://buildbot.libretro.com/stable/1.14.0/android/RetroArch_aarch64.apk is using what? Speaking of those, F-Droid likes to have APKs reproducible if possible so having the APK contents match your own is a goal. I wanted to put the hash sum of the file so we can check it before built, that's not feasible now

I agree with the reproducibility issue. I checked https://github.com/libretro/libretro-super but could not find the recipe. @LibretroAdmin can you tell how the assets at https://buildbot.libretro.com/assets/frontend/assets.zip are updated? If the asset update is triggered by commits, it should be easy. If not, the process should generate a hash to make APKs reproducible.

I forgot to ask, where (relative to the repo root) does the assets folder needs to be? media/assets/ ? media/ ? There were no reports that my test APKs are broken, so not sure how this works.

Honestly speaking idk about these questions, hopefully some RetroArch dev working on Android part can answer this. But I think the command itself should extract it to the right place, i.e. <repo-root>/assets.

licaon-kter commented 1 year ago

Been reading https://docs.libretro.com/development/retroarch/compilation/android/#prep-work but appears not to be up to date.

eg. https://github.com/libretro/RetroArch/blob/master/fetch-submodules.sh is not mentioned and that clones assets too

/LE: even with fetch-submodules run the end APK does not have assets inside :(

WingofaGriffin commented 1 year ago

It looks like the Samsung Galaxy Store version is up to date (though I am still unsure the difference between 64 and non-bit-named versions), which is nice, but f-droid and play being behind is a bummer for non galaxy devices.

denilsonsa commented 1 year ago

(though I am still unsure the difference between 64 and non-bit-named versions)

I'm also confused about the different versions. The official download page lists three direct download links for Android .apk files. One is clearly 32-bit (_ra32), another is clearly 64-bit (_aarch64), and another has no suffix (is it a combined 32+64 version?). There is no explanation to their differences in the download page. I know the Play Store versions were split into two, but people wouldn't know it from the download page. I can also understand the bigger version has more built-in cores, but I found the core list nowhere.

This comment is a bit off-topic, we could create a new issue about properly explaining/documenting the different versions.

Vinfall commented 1 year ago

wow, one hour difference and it changed :( I wanted to put the hash sum of the file so we can check it before built, that's not feasible now

I looked at how GNU/Linux distros packages assets, but it did not help much :(

retroarch-assets in Debian Package Tracker shows the package has reproducibility, but that's because Debian hosts its own fork here.
RetroArch template in void-packages has checksum of the main app (not assets), but that's because xgensum from xtools generates it beforehand.

I still have no idea how to make assets reproducible.

@LibretroAdmin can you answer my question at https://github.com/libretro/RetroArch/issues/11922#issuecomment-1353007066, i.e. how the assets at https://buildbot.libretro.com/assets/frontend/assets.zip are updated?

Is the assets update triggered by commits? If not, where can I find the build recipe?

gknutson1 commented 1 year ago

@denilsonsa @WingofaGriffin Download (64 bit) (RetroArch_aarch64.apk) is 64-bit. Download (32 bit) (RetroArch_ra32.apk) is 32 bit. Download (RetroArch.apk) is both.

I agree that it's a bit confusing, and should be documented better - maybe rename the one marked just as "Download" to "Download (32/64bit)", or "Download (Both)"?

FYI, APK files are actually just zip files, and all supported architectures are listed as folders in the /lib directory of the APK.

Here is a nice little bash script that uses unzip to determine an APK's supported architectures:

#!/bin/bash

# This script checks what arcitecture an apk is by examining the contents of the /lib directory of the apk using unzip

[[ ! $(command -v unzip) ]] && echo "unzip is not installed" 1>&2 && exit 1

for file in "$@"; do
    [[ ! -f "$file" ]] && echo "\"$file\" is not a file" 1>&2 && continue
    # Use unzip to list all the files in the /lib folder of the zip file
    libfiles=$(unzip -l "$file" 'lib/*' 2> /dev/null)
    [[ $? != 0 ]] && echo "unzip could not process \"$file\", file does not contain a /lib directory, or no folders exist in /lib" 1>&2 && continue
    # extract the names of the folders in /lib, which are the supported architectures of the apk
    archlist=$(echo "$libfiles" | grep -oP 'lib/\K[^/]+(?=/)')
    echo -e "$file:\n$archlist\n"
done

Here is the result on running this script on the three RetroArch APKs:

❯ ./check_apk_arch.sh RetroArch*
RetroArch.apk:
arm64-v8a
x86_64
x86
armeabi-v7a

RetroArch_aarch64.apk:
arm64-v8a
x86_64

RetroArch_ra32.apk:
x86
armeabi-v7a

Slavko-P commented 1 year ago

What's the current status on this issue? On a Galaxy S8+, a release from two years ago is still the latest version.

Vinfall commented 1 year ago

We are stuck at the reproducibility issue of RetroArch assets which update too fast and therefore cannot generate a checksum to satisfy F-Droid requirement. Without it users need to manually download assets using built-in updater to make the app behave normally. If the dev decides that is acceptable (and maybe F-Droid gives an exemption I guess), the app can get on F-Droid right away.

hizzlekizzle commented 1 year ago

I think we're fine with pretty much whatever needs to happen.

We identified a reduced number of assets that would provide a basic usability with the default mobile menu, right? Can we just put that static archive somewhere? Will that satisfy the checksum issue?

licaon-kter commented 1 year ago

It would be acceptable without repro too, but... I did not see any conclusion being drawn here and upstream being okay with the workflow.

Repro is nicer indeed.

Vinfall commented 1 year ago

We identified a reduced number of assets that would provide a basic usability with the default mobile menu, right? Can we just put that static archive somewhere? Will that satisfy the checksum issue?

All the answers are yes. You can put it at somewhere like https://buildbot.libretro.com/assets/frontend/assets-minimal.zip and leave it alone. As long as it's not changed quite frequently, everything should be fine.

hizzlekizzle commented 1 year ago

Ok, great. Here's a link: http://buildbot.libretro.com/assets/frontend/glui_minimal_assets.zip

I checked to make sure it worked well enough with GLUI to get someone started, and they can always fetch the rest if they need it. That archive includes an 'assets' directory that can drop directly into the RetroArch bundle.

libretro / RetroArch

[Android] [F-Droid] Current Status of LibRetro F-Doird Repository #11922