F-droid please #4

Closed amir28 closed 1 year ago

Please publish this on f-droid. thanks

Thanks @amir28 and sorry for the delay! We've had this in mind for a long time, but since we'll soon be working on updating CENO Browser to a more modern code base (as it still uses the very outdated Mozilla Fennec) and a more straightforward build process, we considered it better to wait until then for creating an F-Droid package.

F-Droid maintainer here. Thanks for your reply, @ivilata – what a coincidence that inclusion of your app was already requested in our issue tracker at … oof, 2 years ago. It somehow got stuck, but with your help we might get it kicking. I'm not a developer myself so I cannot help on that front, but here are the points I see missing:

could you establish Fastlane structures here in your repo for metadata (description, screenshots etc)? At F-Droid, we'd just need the metadata (which is all that the page behind the link describes). But if you want to, you could also use their binaries and have your app deployed to GooglePlay using the very same resources.
next we'd need some pointers on how to build the app. I saw no build.gradle (which is what we favor at F-Droid). As it's unlikely the Docker variant will work for us, I assume To Make A Release Build is what I should point our "builders" to?
as we need to build entirely from source (and thus no JARs/AARs are permitted as "input"), can you say if that would work?

Thanks in advance – and be welcome to chime in on the issue in our tracker directly, so we don't have to jump back-and-forth for information gathering :smile:

PS: As your project seems to be based on Fennec, and F-Droid has a Fennec build, I think I know who could help out with that – and maybe our Fennec recipe could be re-used for Ceno even (or at least parts of that recipe).

Thanks @IzzySoft for all the hints. The new generation of CENO Browser will be based upon modern GeckoView + Android Components (the Mozilla Reference Browser), which should make packaging much more straightforward with the hints that you provide. I'll drop a note in F-Droid's issue too.

Also note that the current Fennec package in F-Droid, despite its name, is actually based on the newer Mozilla Fenix. :wink:

Also note that the current Fennec package in F-Droid, despite its name, is actually based on the newer Mozilla Fenix.

Yupp, I vaguely remember that was brought up but decided to keep the name.

@jnny can you help us "getting things done"? The MR is currently stuck waiting for feedback (see my post from March above). Thanks in advance!

@IzzySoft + @ivilata - Thanks so much for following up! We are in the testing stages of releasing CENOv2 based on Mozilla Reference Browser. I'm tagging the core CENO/Ouinet devs to help facilitate the conversation: @paidforby @mhqz @kpetku

Hey @IzzySoft as @jnny points out we are in the process of getting things done. Points 1 and 2 you made in the above comment, are very doable. In fact, point 2 regarding a straightforward build system is mostly done. We build our new release candidates using a simple build script command, see here, but most of what is done by the script can be done manually and the apk can be built with ./gradlew assembleRelease like any old Android app.

However, I would like clarification on point 3,

as we need to build entirely from source (and thus no JARs/AARs are permitted as "input"), can you say if that would work?

I'd like to understand exactly what this requirement entails.

Since the new version of CENO is based on Mozilla's Android-Components, it uses lots of AARs, see the dependencies declared in app/build.gradle. So do we need to be able to build all of those from source? Furthermore, those Android-Components also contain their own dependencies. For example, you can build the engine-gecko component, but that depends on Mozilla's GeckoView AAR, see here. So then we need to build GeckoView from source, but geckoview depends on mozilla's exoplayer2.... etc etc.

Is this really all required? I looked at the source for Fennec on F-droid and did not see any indication that is it possible to build any of the Android-Components or GeckoView from that source, both of which are certainly being used by Fennec. So how did they handle this requirement?

I partially answered my own question. Found the build script being used to build Fennec for F-droid here, so if we follow the same general pattern (of running publishToMavenLocal) for each dependency as well as doing something similar for the Ouinet aar, we should be okay for publishing to F-droid, correct?

Hi @paidforby – great to see the points addressed! I'm no Android dev – and building Firefox & Co is even in the upper levels of that. At F-Droid, it's mostly Relan dealing with them; but I'd suggest you drop into the fdroid-dev channel at OFTC (IRC) or the corresponding Matrix room (see https://matrix.f-droid.org/alias/%23fdroid-dev:f-droid.org – should be #fdroid:f-droid.org) where you can raise your questions to our devs.

Hi, I'm a maintainer of https://gitlab.com/relan/fennecbuild.

Found the build script being used to build Fennec for F-droid here, so if we follow the same general pattern (of running publishToMavenLocal) for each dependency as well as doing something similar for the Ouinet aar, we should be okay for publishing to F-droid, correct?

Yes. Mozilla components include proprietary libraries (GeckoView, Application Services, Android Components), so we require them to be patched and built from source. The good news is that all this job is done for Fennec F-Droid. :slightly_smiling_face: We also rely on F-Droid server tools to fetch the source code from various repositories (so it's cached across builds), for example Mozilla Release repo is referenced as $$MozFennec$$ in our recipes.

Hi @relan :wave: thanks for the advice. The Fennec build system looks very nice, I think we can do something very similar for CENO. I also have a patch to apply to geckoview that could be done in the same way as the gecko-liberate.patch (right now I made a fork of gecko-dev for it, but that might not be sustainable in the long run and the change is relatively small, you can see it here if you are interested),

One important question. How do I actually use the fdroid build system? I tried to follow these docs both on a VPS and my local machine, but ran into some issues with virtualization at the .makebuildserver step on both. Do you have any tips for using this system? Is there some alternative for testing builds of an fdroid app or do I have to run my own build server?

I also have a patch to apply to geckoview that could be done in the same way as the gecko-liberate.patch (right now I made a fork of gecko-dev for it, but that might not be sustainable in the long run

Patches are more convenient to maintain than forks because they usually apply to multiple versions with no effort.

How do I actually use the fdroid build system? I tried to follow these docs both on a VPS and my local machine, but ran into some issues with virtualization at the .makebuildserver step on both.

There are two ways to run F-Droid server tool: with a Vagrant box (fdroid build --server appid:vercode) and in-place (same but without the --server option). The latter is quite straightforward as it does not involve virtualization, but you need to have all development tools already installed.

Thanks for the advice @relan. So I've gotten the fdroid build system working on an Ubuntu 22.04 VPS, I am able to successfully run, fdroid build org.fdroid.fdroid -l as a test. However, when I run fdroid build org.mozilla.fennec_fdroid -l, I get an error during the build,

Downloading Python-3.9.13.tar.xz...
-> https://www.python.org/ftp/python/3.9.13/Python-3.9.13.tar.xz
Installing Python-3.9.13...
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
WARNING: The Python curses extension was not compiled. Missing the ncurses lib?
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
ERROR: The Python ssl extension was not compiled. Missing the OpenSSL lib?

Please consult to the Wiki page to fix the problem.
https://github.com/pyenv/pyenv/wiki/Common-build-problems

BUILD FAILED (Ubuntu 22.04 using python-build 2.3.4)

See org.mozilla.fennec_fdroid.log for the full error.

Do you know why this might be happening? Maybe I'm missing a dependency? Is there a list of dependencies that I need installed to build fennec_fdroid? The VPS currently has Python 3.10.6 installed.

Also, related to this, I am unable to re-run the build easily because I get the following error on my second run,

ERROR: VCS error while building app org.mozilla.fennec_fdroid: HG purge failed

This is making is tough to debug because it takes very long to gather all the source code again.

Well, shortly after asking about that SSL problem, I figured out a solution was to install libssl-dev. Of course, I ran into a new problem during the build, it now looks like it is getting to the point of trying to do mach build for geckoview, but is complaining about a missing config object? See the log below.

 0:08.70 checking for Android build-tools... ['/home/fdroid/android-sdk-linux/build-tools/31.0.0']
 0:08.70 checking for Android platform-tools... ['/home/fdroid/android-sdk-linux/platform-tools']
 0:08.70 checking for zipalign... /home/fdroid/android-sdk-linux/build-tools/31.0.0/zipalign
 0:08.70 checking for adb... /home/fdroid/android-sdk-linux/platform-tools/adb
 0:08.70 checking for java... /usr/bin/java
 0:08.70 ERROR: GRADLE must be executable:
 Config object not found by mach.
*** Fix above errors and then restart with "./mach build"

Not sure how to fix this error, shouldn't the fennec build system create the config object for me?

Of course, I still have that issue with hg purge mentioned in the last comment, as well.

ERROR: VCS error while building app org.mozilla.fennec_fdroid: HG purge failed

Do you have the purge extension enabled? https://www.mercurial-scm.org/wiki/PurgeExtension

ERROR: GRADLE must be executable: Config object not found by mach.

I guess Gradle just wasn't found. Make a symlink gradle => fdroidserver/gradlew-fdroid in some directory in your $PATH.

Thanks for those hints @relan. Both suggestions solved their respective issues. After solving those I ran into several other issues related to dependencies, which I solved by running the command recommended when starting the fdroid build,

apt-get update && apt-get install -y cmake make m4 g++ pkg-config libssl-dev python-is-python3 python3-distutils python3-venv tcl gyp ninja-build bzip2 libz-dev libffi-dev libsqlite3-dev

Additionally, during one build, I also ran out of memory on my VPS, which only had 8GB of RAM, so I upgraded a to 16GB VPS.

The packages called out by the fdroid build weren't the only ones required to run fdroid. I've record all actions taken to get the build of fennec_fdroid working in a shell script that I will publish with the new CENO source code.

After all that, I finally succeeded in building fennec_fdroid. Now I can begin the work on porting what I need for CENO.

@relan, one point isn't clear to me. How does the fennecbuild repo link in with the fdroiddata? I can see that org.mozilla.fennec_fdroid.yml actually points to the fenix source code and that the fennecbuild scripts are inserted into the yml file, but how is this achieved in practice?

I think maybe see the answer to my question? Maybe. Looking further down in org.mozilla.fennec_fdroid.yml, e.g. at line 12409, the metadata makes more sense. And I can see that MozBuild.yml points to the fennecbuild repo. So I guess the question is, how would I initialize a fork of org.mozilla.fennec_fdroid. I suppose I could copy the metadata manually, but isn't the update of this metadata automated in some way?

Looking further down in org.mozilla.fennec_fdroid.yml, e.g. at line 12409, the metadata makes more sense. And I can see that MozBuild.yml points to the fennecbuild repo.

Exactly. Build entries in the metadata can reference srclibs of particular versions.

I suppose I could copy the metadata manually, but isn't the update of this metadata automated in some way?

It's not. Automatic metadata updating is possible for some simple recipes, but not for complex ones like this. Each new Fennec version has to be added manually.

It's not. Automatic metadata updating is possible for some simple recipes, but not for complex ones like this. Each new Fennec version has to be added manually.

Got it. As I looked at it more I realized it probably wasn't atuo-generated :sweat_smile:

So I can "fork" org.mozilla.fennec_fdroid by copying the yml metadata (or the parts of the metadata I care about) to a new yml file, (e.g. org.mozilla.ceno_fdroid as a test). This seems to be working. I've also forked fennecbuild, so I can point to that in a CenoBuild.yml file, to change the app id, and eventually make more changes, https://gitlab.com/paidforby/fennecbuild. Thanks for the thoughtful assistance @relan, I feel like I am on the right track now.

So I can "fork" org.mozilla.fennec_fdroid by copying the yml metadata (or the parts of the metadata I care about) to a new yml file, (e.g. org.mozilla.ceno_fdroid as a test). This seems to be working. I've also forked fennecbuild, so I can point to that in a CenoBuild.yml file, to change the app id, and eventually make more changes, https://gitlab.com/paidforby/fennecbuild.

Yep. There's a good example of such workflow: https://gitlab.com/divested-mobile/mull-fenix

I feel like I am on the right track now.

Great! Feel free to ask questions if you have any in the future :slightly_smiling_face:

Heads up, I've opened an issue against the new CENO repository here since that is where the changes will actually be made to make CENO F-Droid compliant. I'm closing this as "won't fix" since we will never release CENO v1 on F-Droid. Discussion on this topic should move to the new issue in the v2 repo.

censorship-no / ceno-browser-archive

F-droid please #4