search

element-hq / element-x-android

Android Matrix messenger application using the Matrix Rust Sdk and Jetpack Compose
GNU Affero General Public License v3.0
1.07k stars 155 forks source link

FOSS Version #1418

Closed Sir-Photch closed 4 months ago

Sir-Photch commented 1 year ago

Hi, I wanted to raise some awareness for some objectives that the Android-FOSS community would enjoy seeing. Those are:

  1. Support for ntfy-push. This should count as the greatest FOSS-priority.
  2. Non-Fdroid FOSS releases, preferrably on GitHub
  3. Split APKs

About Nr. 2: While Element-Android has github releases that one can use, they only provide the playstore version of the app. Furthermore, the version available on fdroid often lags behind playstore by a significant timespan. You might make the argument that fdroid only receives stable versions of the app after being tested by playstore users; However for pre-releases, github already provides the necessary facilities.

I hope this could gain some traction in the early development of Element-X.

bmarty commented 1 year ago

Hello,

This is the tasks to achieve to be able to publish on F-Droid:

Already done:

In the long term: Element X Android codebase will be used to upgrade Element Android (the existing app). So at some point this work has to be done, to be able to release the new Element Android to F-Droid

licaon-kter commented 1 year ago

Current status is rather clean ( ahem https://gitlab.com/fdroid/rfp/-/issues/2556#note_1593343486 ), I mean one rm and 3 seds, but yeah, a flavor would be nicer so we don't need to hunt down future changes :wink:

Mynacol commented 9 months ago

I'd like to encourage you to try to realize reproducible builds. It eliminates trust in F-Droid and ensures you are in control of the published software. It avoids issues when end users are using multiple stores for updates, but the same app is signed with different keys (assuming you'd/can use the same key for the play store and F-Droid variants). Publishing to F-Droid first non-reproducible and later turning on reproducible builds has its drawbacks, a cleaner approach would be to publish reproducibly from the start.

For that you have to add the required code changes (new flavor, removed unfree dependencies, only/additionally building split APKs) and publishing those APKs somewhere (GitHub releases?). And maybe some compilation adjustments to make it reproducible.

licaon-kter commented 9 months ago

@Mynacol think you are missing some context.

If this app is repro, great.

But this app will die after a while and current app (non X) will be replaced with this one.

Yet, to have that reproducible then, all the current users will need to uninstall & reinstall.

There's no UI that informs as such and guide the users, at least not yet.

The TWIF (this week in fdroid) blog posts are used to inform about such changes but not sure everybody is RSS subscribed to that :)

K4LCIFER commented 9 months ago

It looks like Element X is to be published to F-Droid, with support for UnifiedPush, for the release of v0.4.2:

Next release 0.4.2 should also be published on F-Droid with push support using UnifiedPush.

Related PR: https://github.com/element-hq/element-x-android/pull/2248

Guanran928 commented 9 months ago

@linsui told me they (fdroid) are still waiting for Element team to publish FOSS flavor's apk, so they can do reproducible build.

https://matrix.to/#/!JqpjuzzVfwdybKQWQg:f-droid.org/$Z2CIcmMm-uvxvnp1keBP7DADRsuPAStDG5eWEC8Y3E4?via=mozilla.org&via=t2bot.io&via=matrix.org

licaon-kter commented 9 months ago

@K4LCIFER read this thread at least https://gitlab.com/fdroid/fdroiddata/-/merge_requests/13829#note_1595843143 maybe https://gitlab.com/fdroid/rfp/-/issues/2556 too

Mek101 commented 7 months ago

Only another 20 thousand years...

SamyDjemai commented 7 months ago

Hi @bmarty, from what I see in LoggedInPresenter.kt and in my tests, the UnifiedPush provider seems to be properly working now. Are there any other steps we can help you with before pinging the F-Droid team? :)

licaon-kter commented 7 months ago

@SamyDjemai ping who about what? https://gitlab.com/fdroid/fdroiddata/-/merge_requests/13829

SamyDjemai commented 7 months ago

@licaon-kter I was referencing this message which seems to be outdated: https://github.com/element-hq/element-x-android/issues/1418#issuecomment-1752465669 I've only now noticed that builds were already in progress on the F-Droid side :sweat_smile: Any blockers there? I'm noticing that the checkupdates GitLab CI job is failing, would we need to change something in this repo to comply?

licaon-kter commented 7 months ago

Expand the threads and read F-Droid feedback and ara4n's answers, that's the status I guess...

would we need to change something in this repo to comply?

yes, a place for F-Droid to grep the versionName and versionCode

SamyDjemai commented 7 months ago

I think that instead of hardcoding the versionName and versionCode in this repo, we could use UpdateCheckData in HTTP mode and scrape the GitHub API for latest releases. As an example, we could have:

UpdateCheckMode: HTTP
UpdateCheckData: 'https://api.github.com/repos/element-hq/element-x-android/releases?per_page=1|"name": "(\d+)|.|"tag_name": "v((?:[0-9]\d*)\.(?:[0-9]\d*)\.(?:[0-9]\d*))"'

This way, the release process stays unchanged, and we can dynamically fetch versionCode and versionName from the latest GitHub release

licaon-kter commented 7 months ago

@SamyDjemai that line yields ruamel.yaml.scanner.ScannerError: mapping values are not allowed in this context so it might need some tweaking :)

jspricke commented 7 months ago

This works for me:

UpdateCheckData: 'https://api.github.com/repos/element-hq/element-x-android/releases?per_page=1|"name":"(\d+)|.|"tag_name":"v((?:[0-9]\d*)\.(?:[0-9]\d*)\.(?:[0-9]\d*))"'
SamyDjemai commented 7 months ago

I've also reworked mine with simpler matching :)

UpdateCheckMode: HTTP
UpdateCheckData: https://api.github.com/repos/element-hq/element-x-android/releases/latest|"name":"(\d+)-|.|"tag_name":"v(\d+\.\d+\.\d+)"

I'd suggest using https://api.github.com/repos/element-hq/element-x-android/releases/latest instead of using the ?per_page=1.

I'm getting this error:

2024-03-22 12:32:41,036 ERROR: ...checkupdate failed for io.element.android.x : current version is newer: old vercode=40004064, new vercode=40004060

@licaon-kter would we track specific commits of the repo, or only releases? If we do the latter, we should clean up the Builds section of the manifest file

licaon-kter commented 7 months ago

I'm getting this error:

I've fixed that, let me test the rest

would we track specific commits of the repo, or only releases?

we usually get a Tag on checkupdate and use the commit hash

but, with HTTP we can't get commit...

SamyDjemai commented 7 months ago

What if we use AutoUpdateMode: Version v%v? According to this documentation, we can directly use the version tag, and we don't need the commit hash

licaon-kter commented 7 months ago

there's no v in versionName: https://github.com/element-hq/element-x-android/blob/v0.4.6/plugins/src/main/kotlin/Versions.kt#L63

this works if we just ignore versionName (high goal: https://gitlab.com/fdroid/fdroidserver/-/issues/1047) or if upstream has Tag = versionName

SamyDjemai commented 7 months ago

But the Git tags are in the vX.X.X format, hence the need for v%v if I'm not mistaken, since according to the doc:

The pattern is used to generate a value (tag name) used for the commit: property of new build blocks. It is simply text in which %v and %c are replaced with the required version name and version code respectively. The resulting string must match an existing tag in the app’s repo, which then will be used by F-Droid to build the corresponding version.

licaon-kter commented 7 months ago

yup that does it

...
AutoUpdateMode: Version v%v
UpdateCheckMode: HTTP
VercodeOperation:
  - 1 + %c
  - 2 + %c
  - 3 + %c
  - 4 + %c
UpdateCheckData: https://api.github.com/repos/element-hq/element-x-android/releases/latest|"name":"(\d+)-|.|"tag_name":"v(\d+\.\d+\.\d+)"
CurrentVersion: 0.4.6
CurrentVersionCode: 40004064
SamyDjemai commented 7 months ago

Great, thanks! From what I've assessed, in order to be compliant with F-Droid's inclusion policy, we still need to do the following:

@licaon-kter do you see any other blockers? do we want reproducible builds? @bmarty is the Element X team OK for F-Droid release?

licaon-kter commented 7 months ago

build F-Droid version without proprietary components

current recipe does that

Sentry: remove services/analyticsproviders/sentry in the F-Droid manifestJ

not a NEED, if it's on by default or opt-out, it will get a Tracking anti-feature, if it's off by default and opt-in, it will get nothing

get approval from the upstream team for F-Droid release (clear up the existing confusion regarding release)

mostly this yes

reproducible builds

the big IF

SamyDjemai commented 7 months ago

Nice catch, forgot that the Sentry SDK is open source :facepalm: There is an opt-in screen for tracking when first logging into the app, so there won't be a Tracking anti-feature.

reproducible builds

The "current" Element app currently doesn't have reproducible builds, but there will be no need to uninstall and reinstall for users as you mentioned here if we implement them later and publish both developer-signed and F-Droid-signed APKs, right?

In my opinion, reproducible builds are a nice-to-have feature regarding the "temporary" nature of this app, but they aren't a priority. The 52 people that liked the issue's first message probably just want to try out Element X, no matter if it has reproducible builds or not.

Mek101 commented 7 months ago

The 52 people that liked the issue's first message probably just want to try out Element X, no matter if it has reproducible builds or not.

I just want to try Element X without having the burden of keeping track of each release

Mynacol commented 7 months ago

Reproducible builds are always "nice to have."

However, there is exactly one time slot to determine which type of reproducible builds we choose for this app: before the first release in F-Droid. Adding reproducible builds from the start on means there is never a F-Droid signed build around, which reduces confusion of users (trying to update the F-Droid build with a GitHub release fails and users don't know why). It also preserves auto update functionality at F-Droid (adding reproducible builds later requires to manually add signature information for each new release). There is a decision to be made now. I hope the description above highlights the tradeoffs.

SamyDjemai commented 7 months ago

However, there is exactly one time slot to determine which type of reproducible builds we choose for this app: before the first release in F-Droid.

While I agree with that, the plan for this app is for its codebase to replace the existing Element app in app stores and in F-Droid, so even if we set up reproducible builds here before first release, we're still going to have to deal with an existing app that was already built and deployed before reproducible builds were set up.

That's why, in my opinion, they are nice-to-have, especially in this repo which is meant to be temporary.

adding reproducible builds later requires to manually add signature information for each new release

That's problematic, but from what I understand based on the GitLab pull request, updates won't be automated anyways

ildar commented 7 months ago

Let me comment as a user. Element (and X) is by its nature cloud-based. And local data is near zero. So uninstall+install is quite painless. I.e. reproducible builds aren't important. Thank you.

linsui commented 7 months ago

Reproducible Build doesn't only allow the user to install the apk from different channels, but also make sure that the apk built by F-Droid has no bug introduced by F-Droid. So F-Droid doesn't need to set a user agent for bug reports.

Please can you ensure that you set a user agent which clearly identifies the app as being the Fdroid build (so we can differentiate bug reports & debug appropriately)

bmarty commented 5 months ago

@licaon-kter I have published 4 APKs for F-Droid in the release 0.4.13: https://github.com/element-hq/element-x-android/releases/tag/v0.4.13

Let me know if it's OK and if you need anything else.

licaon-kter commented 5 months ago

well well well... I might even check that reproducibility then, brb

licaon-kter commented 5 months ago

First result

Unexpected diff output:                                    
Binary files /tmp/tmpx353fgjh/unsigned_binaries_io.element.android.x_40004132.binary/content/AndroidManifest.xml and /tmp/tmpx353fgjh/_tmp_tmpx353fgjh_sigcp_io.element.android.x_40004132/content/AndroidManifest.xml differ               
Binary files /tmp/tmpx353fgjh/unsigned_binaries_io.element.android.x_40004132.binary/content/assets/dexopt/baseline.prof and /tmp/tmpx353fgjh/_tmp_tmpx353fgjh_sigcp_io.element.android.x_40004132/content/assets/dexopt/baseline.prof differ                                                          
Binary files /tmp/tmpx353fgjh/unsigned_binaries_io.element.android.x_40004132.binary/content/classes.dex and /tmp/tmpx353fgjh/_tmp_tmpx353fgjh_sigcp_io.element.android.x_40004132/content/classes.dex differ                               
Binary files /tmp/tmpx353fgjh/unsigned_binaries_io.element.android.x_40004132.binary/content/classes3.dex and /tmp/tmpx353fgjh/_tmp_tmpx353fgjh_sigcp_io.element.android.x_40004132/content/classes3.dex differ                             
Binary files /tmp/tmpx353fgjh/unsigned_binaries_io.element.android.x_40004132.binary/content/resources.arsc and /tmp/tmpx353fgjh/_tmp_tmpx353fgjh_sigcp_io.element.android.x_40004132/content/resources.arsc differ

...will diff the APKs later

licaon-kter commented 5 months ago

In the meantime, the starting flow is funny/sad, always pushing the central server:

/LE: see https://github.com/element-hq/element-x-android/issues/2909

licaon-kter commented 5 months ago

Love you too thumbdowners :smile:

Anyway, @bmarty, the diff log for arm64 is small so we are close to being repro, all seem easily fixable: -uyVTbKjT3qFQHpitMUYEg.log

licaon-kter commented 5 months ago

Maybe reopen until those 2-3 repro issues get fixed?

bmarty commented 5 months ago

Thanks for the diff @licaon-kter .

The values are store as GitHub secret so you do not have it when you build yourself. We know that the secrets are easy to find, but the idea is to limit the usage of those keys by forks. Is it possible to provide the value in your build chain? You will need to set those 3 env variables:

For the diff in DefaultVersionFormatter, I think that: buildMeta.gitBranchName == "main" is not true in your build env. The value is equal to "HEAD".

Is it possible that your build chain patch the code here? So in your case the line will change from

fun Project.gitBranchName() = runCommand("git rev-parse --abbrev-ref HEAD") to fun Project.gitBranchName() = "main"

This is the same root cause for the following diffs.

There is also a diff in versionCode. On our build, the value is 40004130 and on your build, the value is 40004132. The last digit is for the device architecture.

We should not use BuildConfig.VERSION_CODE, (since it will not be impacted by this code) but read the value from the Manifest. I'll create a PR about it. I am not sure to understand why there is a diff in the build though, but it should fix the issue. I am still investigating this.

licaon-kter commented 5 months ago

The values are store as GitHub secret so you do not have it when you build yourself.

https://f-droid.org/docs/Inclusion_Policy/

F-Droid does not sign up for any API keys. Even if provided by a third party, we include them in both, binary and sourcecode releases.

they need to be here in your code or in the F-Droid recipe, no secrets allowed

Is it possible that your build chain patch the code here? So in your case the line will change from

we can do, yes

I am still investigating this.

great

bmarty commented 5 months ago

they need to be here in your code or in the F-Droid recipe, no secrets allowed

Yes, you can put the values in the F-Droid recipe, and set environment variables as described, this is fine (sorry if it was not clear). The 3 values are in the diff you provided.

Alternatively, you can put the values in the local.properties file. The values are read here and it's documented here.

bmarty commented 5 months ago

https://github.com/element-hq/element-x-android/pull/2911 should help on the versionCode diff.

bmarty commented 5 months ago

@licaon-kter #2911 has been merged, so I think the plan for us is to publish F-Droid APKs for the next release on Github v0.4.14. This should occur on June 4th.

On your side: change the recipe about the 2 points mentioned above: the map keys and the git branch name.

Hopefully the APKs will have the reproducible build green flag.

licaon-kter commented 5 months ago

I've already tested the map injection fix, will test the testother fixes when the Tag drops. Feel free to ping me.

licaon-kter commented 5 months ago

@bmarty tick-tock, 4-th was gone

bblacher commented 5 months ago

@licaon-kter They just bumped the version, in a few minutes the release will be out. https://github.com/element-hq/element-x-android/commit/4b27cda15c478049b7f6c2f2ab3b3a657a270af2

ganfra commented 5 months ago

@licaon-kter https://github.com/element-hq/element-x-android/releases/tag/v0.4.14 Have fun.

licaon-kter commented 5 months ago

Started a build

First issue though, I can't know the versionCode for autoupdates, we used to pick it up from the APK name in past releases, eg. https://github.com/element-hq/element-x-android/releases/tag/v0.4.13 -> 40004130.apk -> 40004130

but that's not a thing in latest https://github.com/element-hq/element-x-android/releases/tag/v0.4.14

ganfra commented 5 months ago

Ok, can rename the file.

-> Done.

licaon-kter commented 5 months ago

Ok, thanks, the question is, do you need that file or? maybe there's a way to name it so it's usable both by F-Droid and users that drop by (eg. Obtainium etc)

ganfra commented 5 months ago

This is the universal apk for gplay variant. I just renamed it manually so it was clear. The original name is the one provided by the playstore.

licaon-kter commented 5 months ago

Here's the first diff log: elem0414_2.log

this looks like https://f-droid.org/docs/Reproducible_Builds/#embedded-build-paths with some ways to fix it here: https://gist.github.com/obfusk/61046e09cee352ae6dd109911534b12e

not sure if the AndroidManifest.xml will be fixed too once baseline is

linsui commented 5 months ago

Given that we can't get the release files without using the api which is rate limited, can you add a version.txt with version code in it?