Include pre-releases in the F-Droid repo

codenyte commented 8 months ago

Is Your Feature Request Related to a Problem?

No response

Feature Description

Publish the builds, which are marked as pre-releases on GitHub Releases, on F-Droid. I've seen some apps do this before, but I'm not exactly sure how to set this up. I looked at the F-Droid documentation, and apparently you just need to put the unstable release in F-Droid and set the latest stable version as the CurrentVersion field in F-Droid build metadata. That way, F-Droid should only install stable releases by default, but should also allow installing beta builds, if the user enables the "Allow Beta Updates" toggle. Maybe @IzzySoft can help with this?

Alternative Solutions

Currently I use Obtainium to download the nightly builds from GitHub releases

Additional Context

No response

IzzySoft commented 8 months ago

The app is currently in my repo, but pre-releases are disabled. Due to the APK size (already slightly exceeding the per-app-size-limit of 30 MB), only the latest version is available there – so enabling pre-releases would mean that, whenever a pre-release was pulled, no stable release would be available – so I don't think this is a good idea.

Your description points to the build strategy at F-Droid.org; to my knowledge Thunder isn't available there (or it must be using a different applicationId for the F-Droid build). For my repo, that's a simple toggle; by default, pre-releases are skipped. If the authors want them to be pulled as well, that can be done – but due to the outlined issue I'd recommend not to.

If you were asking about F-Droid.org, I cannot help you with that as I'm currently not active at F-Droid.

codenyte commented 8 months ago

Your description points to the build strategy at F-Droid.org; to my knowledge Thunder isn't available there (or it must be using a different applicationId for the F-Droid build). For my repo, that's a simple toggle; by default, pre-releases are skipped. If the authors want them to be pulled as well, that can be done – but due to the outlined issue I'd recommend not to.

Sorry, I didn't realize that there were differences between the official repo and yours in regards to building applications. I thought it was the same procedure, thanks for clarification.

@hjiangsu We could consider submitting Thunder to the official F-Droid repo or maybe host a custom repo on GitHub.

hjiangsu commented 8 months ago

Due to the APK size (already slightly exceeding the per-app-size-limit of 30 MB), only the latest version is available there

Ahh I didn't know about this - thanks for mentioning it! If I can reduce the app size below 15MB, would that allow the possibility to have nightly/pre-releases on the repo? This might be do-able if I split the APK per ABI.

If this is the case, which version would be hosted on the repo (armeabi-v7a, arm64-v8a, or x86_64)? Unfortunately, I'm not too familiar with Android practices so any insight would be helpful. I'll still most likely include all three APKs separately within a given GitHub release.

We could consider submitting Thunder to the official F-Droid repo or maybe host a custom repo on GitHub.

This is already in the works as mentioned in this PR: https://github.com/thunder-app/thunder/issues/347. I believe the main issues are solved already, but I'm just waiting until the stable release of 0.3.0 before I continue with the F-Droid requirements/validation.

codenyte commented 8 months ago

I would avoid split APKs, as they are kinda annoying to deal with. We could continue just shipping the stable release on IzzyOnDroid and once Thunder gets released on F-Droid.org, we could also add the unstable builds using the procedures I outlined in my original comment (if that works, because I'm not exactly sure about it).

IzzySoft commented 7 months ago

Sorry, I didn't realize that there were differences between the official repo and yours in regards to building applications.

F-Droid builds from source. My repo retrieves the APKs the developers provide.

If I can reduce the app size below 15MB, would that allow the possibility to have nightly/pre-releases on the repo?

Depends. 15 MB or less would allow for 2 releases to be kept. But if e.g. a "stable release" is followed by multiple "pre-releases", we'd have the same situation again: no stable releases available. There'd be other approaches to this, like

using different applicationIds/packageNames for stable and pre-release (e.g. give the latter ones a suffix like .nightly). That would make it separate apps which could co-exist
apply for listing at F-Droid.org and have e.g. them shipping the stable releases while keeping the pre-releases with my repo. Combined with establishing reproducible builds there, folks could cross-update when needed.

which version would be hosted on the repo

Per your preference, one of the ARM builds. x86 devices are rather rare.

I would avoid split APKs, as they are kinda annoying to deal with.

First time I hear this. The only disadvantage is that, for my repo, we'd need to decide for one ABI then – and depending on whom you ask it will always be the wrong one :see_no_evil: While theoretically 64bit devices can run armeabi builds as well, there are very few exceptions (I know only of a single exception, some Google Pixel 7 pro, where Google disabled 32bit support) – while the other way around, no 32bit device could run arm64.

we could also add the unstable builds using the procedures I outlined in my original comment

Which would require updates at F-Droid to be set to manual IIRC, as otherwise (same as here) stable releases would be moved to archive leaving only pre-releases in main, and also pre-releases would be marked as "recommended version".

codenyte commented 7 months ago

Thanks for the explanation.

I would avoid split APKs, as they are kinda annoying to deal with.

First time I hear this.

I mean this in the way, that AABs would suck for users who manually download the releases from GitHub, instead of using F-Droid or Google Play.

using different applicationIds/packageNames for stable and pre-release (e.g. give the latter ones a suffix like .nightly). That would make it separate apps which could co-exist

I'm not a big fan of this approach, because it doesn't allow for smooth updates from stable to nightly/pre-release.

apply for listing at F-Droid.org and have e.g. them shipping the stable releases while keeping the pre-releases with my repo. Combined with establishing reproducible builds there, folks could cross-update when needed.

I think this might be the best solution.

IzzySoft commented 7 months ago

I mean this in the way, that AABs would suck

Ah, different term. You refer to app bundles (AAB), I refer to "split APKs" (separate *.apk files).

because it doesn't allow for smooth updates from stable to nightly/pre-release.

No. But it allows you to test if a certain "fix" is working with a planned "stable release" before that stable release is being made, by temporarily installing the pre-release without uninstalling/altering the stable one.

I just outlined the possibilities I see – and will accept the approach the app team prefers. Which probably is for @hjiangsu to say then :wink:

codenyte commented 7 months ago

I mean this in the way, that AABs would suck

Ah, different term. You refer to app bundles (AAB), I refer to "split APKs" (separate *.apk files).

Oh, you mean seperate APKs for each architecture? Ok sure, that makes sense. I was a little confused there.

No. But it allows you to test if a certain "fix" is working with a planned "stable release" before that stable release is being made, by temporarily installing the pre-release without uninstalling/altering the stable one.

Yeah I also thought about that, and I'm genuinely surprised that Android doesn't allow the user to downgrade an app to an older version. But considering that Thunder pre-releases tend to be rather stable (at least all the ones I tried), and the user isn't forced to download pre-releases, I don't think this would pose a big issue. But that's up to @hjiangsu to decide.

IzzySoft commented 7 months ago

you mean seperate APKs for each architecture?

Exactly that, yes.

I'm genuinely surprised that Android doesn't allow the user to downgrade an app

Oh, there are good reasons for that (e.g. database upgrades rarely have a downgrade procedure in the older APK, because how should that work?). But there would be possibilities. Like having a flag like minCompatibleVercode, pointing to the oldest version one could safely downgrade to, and allowing downgrades that far. But well, it's not done and so we have this situation…

codenyte commented 7 months ago

you mean seperate APKs for each architecture?

Exactly that, yes

Ok, then I take back my previous statement, where I was confused about AABs and seperate APKs for each architecture. AFAIK those shouldn't pose an issue to the user. I just hate AABs (I know that they make sense in certain cases, but not in this one).

Oh, there are good reasons for that (e.g. database upgrades rarely have a downgrade procedure in the older APK, because how should that work?).

That really makes sense, I should have thought about that. Thanks again, I learned so much from this thread.

micahmo commented 5 months ago

Just chiming in here to say that another possible idea would be to have a separate app listing in @IzzySoft's repo for a prerelease channel. (We already have precedent for generating an app with a separate identifier for development purposes.)

That would get around IzzySoft's per-app size limit (as long as he's ok with that solution; we're not trying to do anything shady!).

The downside is that you wouldn't be able to upgrade directly from the prerelease channel to the mainline channel, because the device would see them as two separate apps. However, to counteract that, we could also publish the mainline releases to the prerelease channel. So for users who always want the very latest, they could just use that channel in perpetuity.

Thoughts on this approach?

IzzySoft commented 5 months ago

That would get around IzzySoft's per-app size limit

He, gaming the rules? :rofl: Yeah, that would effectively be a different app then.

The downside is

Forgetting the upside here, are you @micahmo? If there's some issue with the main channel release which you think having fixed in the prerelease channel, one can install the pre alongside the main to fid out – without risking to break the main one on-device.

If you read between the lines of this comment: I'm not opposed to this approach. Especially as you did put it so nicely :wink:

micahmo commented 5 months ago

one can install the pre alongside the main to fid out

Yes, this is exactly why we have the ability to build independent flavors internally. 😊

https://github.com/thunder-app/thunder/blob/ef5e8f75d4e93195251bca3c839dd2174a930ea2/android/app/build.gradle#L83-L93

Obviously the final decision is up to @hjiangsu! But I can see a lot of people (who like to be on the bleeding edge) taking advantage of this. 😊

I'm not opposed to this approach.

Awesome! Thanks so much for commenting and being so active with so many app developers.

IzzySoft commented 5 months ago

Thanks so much for commenting and being so active with so many app developers.

It's very important that all sides are on the same side – err, pulling the rope together at the same end: folks who develop the apps, those running/maintaining the repos, and the people using both. That's the only way making the FOSS world a better place. Standing together brings us forward to a bright future :yum: So I just try to do my part there as good as I can :bow:

IzzySoft commented 5 months ago

Eh, while I'm here, I see I forgot to report one thing to you: for v0.4.0, my scanner yielded an alert:

com.hjiangsu.thunder: needs NonFreeComp, needs NonFreeNet!!!

Offending libs:
---------------
* Firebase Data Transport (/com/google/android/datatransport): NonFreeNet
* Google Mobile Services (/com/google/android/gms): NonFreeComp
* Firebase (/com/google/firebase): NonFreeNet,NonFreeComp
* Firebase Installations (/com/google/firebase/installations): NonFreeNet

4 offenders.

Not sure if those were added intentionally, or whether some dependency dragged them in – but they are proprietary and should not be there. Can you please take a look? Thanks in advance!

(for now, those 2 anti-features have been added – but we'd better get rid of them, right? I hope you have an idea what changed around that time to narrow it down. Vague guess: when implementing UnifiedPush, you did not integrate the FCM distributor with your app, did you? That would perfectly match those libraries. Oh noz, that indeed seems to be the case – as I see Thunder now requests the c2dm.permission.RECEIVE permission :see_no_evil:

hjiangsu commented 5 months ago

Yes, this is exactly why we have the ability to build independent flavors internally. 😊

I'm actually running both the production and development flavours myself just to compare between them easily and test out fixes 😅

Obviously the final decision is up to @hjiangsu! But I can see a lot of people (who like to be on the bleeding edge) taking advantage of this. 😊

I have no reservations regarding this, and like you mentioned, I'm sure some people would like to have this option (assuming the process of setting up the nightly app listing is fairly straight forward)

Not sure if those were added intentionally, or whether some dependency dragged them in – but they are proprietary and should not be there. Can you please take a look? Thanks in advance!

Ahh, this was definitely accidental. I realized this myself when I was creating an F-droid build and it gave me errors with regards to one of the packages. I believe I've resolved this in the latest nightly (v0.5.0-1). For reference, this is the PR that addresses the issue, and removes firebase on one of our dependencies: https://github.com/thunder-app/thunder/pull/1412 with this relevant commit: https://github.com/thunder-app/push/commit/80fae1a6c3d506d1b7c59234a8235502b6759d37

@IzzySoft are you able to confirm if this is the case?

As an additional note of clarity: the dependency (push) which adds in the firebase transitive dependency is not actually used at all in the Android build. It's only used to allow push notifications on the iOS side. However, Flutter doesn't allow us to selectively include dependencies based on the platform (hence the removal of the Android-specific portions in the PR linked above).

Additionally, with UnifiedPush, we do have some experimental push notification logic in place but that is fully disabled within the app in v0.4.0 unless you explicitly run a debug build of the app:

https://github.com/thunder-app/thunder/blob/9e93f217acf78a5126d0809b0b3df61aeb51f812/lib/settings/pages/general_settings_page.dart#L812-L839

Thanks for letting us know about this and I hope my explanation clarifies things!

IzzySoft commented 5 months ago

are you able to confirm if this is the case?

If you provide me the APK, I can run it through my scanner. If you wish, you can use that scanner yourself, it's FOSS. It's described in my article Identify modules in apps where you also find the link to the source.

Btw, UnifiedPush has some example code for FCM that's totally FOSS (and thus would only raise NonFreeNet), similar to what Mastodon and its forks use. Just in case you're looking for something like that.

hjiangsu commented 3 months ago

@IzzySoft Not too sure where else to ask this, but is v0.5.0 still detecting the same anti-features? I noticed that v0.5.0 is still reporting anti-features on the repo and just wanted to double check!

IzzySoft commented 3 months ago

Nope, they have been removed here (checks timestamp) around 7 pm yesterday, right after the new release was pulled and the scanner reported them gone. Takes to the next sync to be reflected online, sorry – so visibly they should be gone at around 6 pm UTC then (regular sync is every 24h).

thunder-app / thunder