search

ankitects / anki

Anki's shared backend and web components, and the Qt frontend
https://apps.ankiweb.net
Other
18.25k stars 2.08k forks source link

Is Anki too hard to build? (Or, make life easier for packagers.) #1378

Closed ropery closed 2 years ago

ropery commented 2 years ago

Our package maintainer decided to drop anki (having been stuck at 2.1.35 for months) from the official Arch Linux repos to the AUR, with the reasoning:

Dropping since they completely redid the build system yet again, but this time replaced it with bazel, which appears to be a rather unreasonable build system that no other TU seemed to understand either.

If you look at the anki packages in different distros, the majority are badly out of date.

The only exception is FreeBSD games/anki, whose maintainer is particularly tenacious; but if you look at the build files, and the packager's comments, it appears awfully tedious.

Edit: In the case of NixOS, they provide two versions of the package: the default build-from-source one stuck at 2.1.15 and anki-bin at 2.1.47 (= no building & uses official binaries & larger installation size).

So, is Anki too hard to build? Are packagers held back because of it?

escape0707 commented 2 years ago

I'm really curious to see whether Anki is truly a cross-platform app now.

dae commented 2 years ago

Bazel solves a lot of issues that the old Makefile-based solution had. It's a fairly recent technology, and I expect over time packagers will get more used to working with it as it becomes more widely adopted. The savings in build/test time for daily development alone are considerable, and I'm afraid I do not think it makes sense to prioritise simplicity for distro maintainers over the needs of developers who work on the code each day.

If you're looking for an easy way to install Anki without compiling it yourself, installing via pip is quite simple, and it can work with your system-installed version of Qt.

PureTryOut commented 2 years ago

and I'm afraid I do not think it makes sense to prioritise simplicity for distro maintainers over the needs of developers who work on the code each day.

I don't think you understand the implications of this. As OP mentioned, distributions are already dropping Anki from their repos. Arch Linux is the first, and I'm sure not the last. This means Anki will get a lot less users. Or, distros keep it but remain at an older version. Then users will miss out on all updates made and will only get security and crash fixes.

Both situations are not something you should want as a developer. You should want your users to get the latest and greatest.

Wanting to move away from Makefiles makes sense, but there are other build systems out there that can be used instead rather than Bazel that are much more distribution and developer friendly at the same time. Think of CMake and Meson for example. Especially the latter is getting a lot of interest nowadays, have you considered it?

Note again that basically every distribution out there (except for FreeBSD) is shipping an outdated Anki right now, and I'm afraid that won't change as long as Bazel remains in use.

escape0707 commented 2 years ago

and it can work with your system-installed version of Qt.

I'm nobody but a newbie Arch user, and I just found out how to do that:

  1. Install aur/anki-git's python-* dependencies with yay/paru
  2. Install python-pip with pacman
  3. pip install aqt

You can now run the installed binary. You can also create a .desktop entry for Anki according to anki-git, too. Input method engines like ibus and fcitx(5) will just work because anki is using distro PyQt5 and thus system Qt.

anki-git might be depending on packages more than it needs? Anyway, here is the result of pip show's dependency requirements:

pip show aqt:

Version: 2.1.48
Requires: flask, beautifulsoup4, send2trash, pyqtwebengine, jsonschema, flask-cors, waitress, anki, pyqt5, requests

pip show anki:

Version: 2.1.48
Requires: distro, markdown, protobuf, beautifulsoup4, requests, decorator, orjson, stringcase
Required-by: aqt

Example PKGBUILD.

Arbitrate3280 commented 2 years ago

This means Anki will get a lot less users.

Then users will miss out on all updates made and will only get security and crash fixes.

You should want your users to get the latest and greatest.

I'm not sure why you're acting like it's impossible to get Anki if it's not available on the user's distro repository. The packaged version for Linux worked just fine on all the distros I have tried. It has a simple installation and update process.

PureTryOut commented 2 years ago

I'm not saying or acting like it's impossible, but you should realize that most users will get applications from their system repositories. Note I said "less users", not "no users". Yes Pip packages will work for a few users which is nice, but in the end most people will want to do either apt install anki or use graphical package manager frontends like GNOME Software and KDE Discover.

In the end I'm just a passer-by and you should do whatever you guys think is best for Anki, as developers you always have the final say in everything of course, I'm just trying to warn you about the implications of using a build system like Bazel when clearly distributions are having lots of problems with it.

The packaged version for Linux worked just fine on all the distros I have tried.

Of course it's a target you probably shouldn't care about, but you clearly haven't tried it on a Musl system like Alpine Linux/postmarketOS :wink:

escape0707 commented 2 years ago

The packaged version for Linux worked just fine on all the distros I have tried.

You mean this? https://github.com/ankitects/anki/releases/download/2.1.48/anki-2.1.48-linux.tar.bz2

It seems that this version uses bundled python and qt5? and bundled Qt5 will need a patch to use Fcitx5 input method.

Plus, removed out of distros repos mean no sync for packages for mirrors. It really hurts if your country censors GitHub, etc.

Oh, I forgot to mention that using the bundled package will cause a lot of problems like the Fcitx one and waste system resources, too.

escape0707 commented 2 years ago

@PureTryOut , just FYI, @Vitorvlv is just another passer-by like you and me. I think he don't represent the dev of Anki, but discussion on the problem is welcomed anyway.

lilydjwg commented 2 years ago

I just tried to build anki-git from AUR. The build process seems scary (java + python + node + rust) but it does work and took 4m30s on our 40-core server.

OceanS2000 commented 2 years ago

On my Gentoo system I can build it with bazel but it will not work, neither binary release from GitHub, because it fetches binary PyQt packages with pip. However, as in Gentoo lots of my dependencies are trimmed down with compile time flags, these binary are not compatible.

 bin % pwd
/tmp/anki-2.1.48-linux/bin
 bin % ldd libQt5WebEngine.so.5 | grep 'not found'
    libsystemd.so.0 => not found
    libkrb5.so.3 => not found
    libk5crypto.so.3 => not found
    libkrb5support.so.0 => not found
 bin % ldd *.so.* | grep 'not found' | wc
    143     577    4553
 bin % ./Anki 
Traceback (most recent call last):
  File "runanki.py", line 3, in <module>
  File "/home/dae/venv/lib/python3.8/site-packages/PyInstaller-4.0.dev0+g2886519-py3.8.egg/PyInstaller/loader/pyimod03_importers.py", line 625, in exec_module
  File "aqt/__init__.py", line 21, in <module>
  File "/home/dae/venv/lib/python3.8/site-packages/PyInstaller-4.0.dev0+g2886519-py3.8.egg/PyInstaller/loader/pyimod03_importers.py", line 625, in exec_module
  File "aqt/gui_hooks.py", line 11, in <module>
  File "/home/dae/venv/lib/python3.8/site-packages/PyInstaller-4.0.dev0+g2886519-py3.8.egg/PyInstaller/loader/pyimod03_importers.py", line 625, in exec_module
  File "aqt/hooks_gen.py", line 18, in <module>
  File "/home/dae/venv/lib/python3.8/site-packages/PyInstaller-4.0.dev0+g2886519-py3.8.egg/PyInstaller/loader/pyimod03_importers.py", line 625, in exec_module
  File "aqt/qt.py", line 18, in <module>
ImportError: libsystemd.so.0: cannot open shared object file: No such file or directory
[16016] Failed to execute script runanki

In order to get things work what I have to do is basically remove all bundled shared library and bunch of hack with LD_LIBRARY_PATH and QTWEBENGINEPROCESS_PATH.

Indeed, for such a complex piece of software with multiple languages, there are few solutions besides bazel. But I definitely hope the build/release process can depend less on fetching random binary packages online. It can also be very time-wasting (like fetching entire rust toolchain even if I have a prefectly working one on my machine).

sobjornstad commented 2 years ago

I've been a Linux user for over 10 years, so I may not be completely representative.

That said, as a user literally all I've ever had to do to install Anki on any Linux over that time period is download the Linux package/tarball from the website, extract it, and run ./install (or the equivalent script at the time). This has worked across Ubuntu, Linux Mint, Arch Linux, and Manjaro, and five different computers. Although I'd certainly prefer to be able to pull it through my package manager, especially for ease of updates, I can't really accept that this is difficult. It's no harder than installing on Windows or Mac, and surely if you're using Linux you're comfortable with doing such things.

Even though I'm currently using an Arch-based distro, which has one of the better user package repos out there, I have quite a few pieces of software that need some other means of installation than either the official repos or the AUR (pip/npm, snap, Steam, downloads of archives like Anki, etc.). There are just a lot of ways to install software on Linux right now, and you kind of have to deal with that.

As far as Anki development goes, doing a full build from source is a bit of a pain in my experience, but I'll take @dae's word for it that it's more helpful for the folks that spend the most time on it than it is annoying for me.

Also, @lolilolicon, I don't know how long you've been using Anki, but over those same 10 years most distros' packages of Anki have always been out of date, some of them horrendously so (e.g., so old they're incapable of syncing with the current version of AnkiWeb). Maybe the new build system has made things a bit more difficult, but it's not by any means like we went from everything being neatly up-to-date to everything being a mess with that single change.

escape0707 commented 2 years ago

I can't really accept that this is difficult. It's no harder than installing on Windows or Mac, and surely if you're using Linux you're comfortable with doing such things.

I have to add that in some countries like China and Vietnam, pulling source / binary / tarballs from GitHub or Amazon AWS, etc IS difficult if not impossible. You have to use obscure ways to proxy through governments firewall (besides, it might be illegal). It's harder to setup proxies for so many tools trying to pull toolchains / source code / submodules from everywhere, the speed will be horrible and this produces extra financial costs for simply networking. This is true no matter which platform you try to download / build and install to, except iOS.

Linux distros' mirror servers hosted by universities and companies in the country have always been a better way to circumvent this kind of dilemma and it's a loss for Anki's (potentially or already?) big amount of users in China who need not to be a tech-savvy, and probably just a future writer / doctor / historian and simply want to use a reasonable open source language learning tool to learn EN, JP, RU, ES languages. I bet not all of them are rich and free enough to buy an iPhone and AnkiMobile to utilize Apple's possibly not blocked CDN. Most of them are just honest in-school students and don't even have a stable income yet. Shutting down another window for them to learn just makes me feel more horrible.

I don't know which side (dev of Anki or packagers of distros or even the government which obviously won't give a shit about all of us) could do a better job to make the software more reachable for these teenagers, I only know that as long as PyPi still have a mirror in their country , it's possible to install all dependencies with package managers and install wheels of aqt and anki with pip. And all I can do is consider at least write up an AUR PKGBUILD for automating this or some textual instructions to make the idea clearer for them.

ropery commented 2 years ago

Replying to @dae

Bazel solves a lot of issues that the old Makefile-based solution had. It's a fairly recent technology, and I expect over time packagers will get more used to working with it as it becomes more widely adopted.

Thanks for the explanation. I hope it gets easier for everyone.

The savings in build/test time for daily development alone are considerable, and I'm afraid I do not think it makes sense to prioritise simplicity for distro maintainers over the needs of developers who work on the code each day.

Understandable. However, it doesn't have to be either/or. I think it's hard to overstate the importance of the official distro repos for a Linux user. Bazel or not, there must be space for improving the build process for both upstream developers and downstream distro packagers.

If you're looking for an easy way to install Anki without compiling it yourself, installing via pip is quite simple, and it can work with your system-installed version of Qt.

It's good to have an alternative, but as @OceanS2000 demonstrates, even this doesn't always work. (And it's not just his/her "exotic" setup; other issues may arise, e.g. as reported by @escape0707 here. Installing an oversized binary bundle when the source is available is, in any case, a somewhat iffy last resort.) More to the point, I think this discussion should be around the challenges of building & packaging Anki for distros.

With so many distros falling hopelessly behind, let's face it: Anki has a packaging problem. In raising this discussion, I hope to help solve this problem. I hope especially for packagers to join the discussion with developers:

ropery commented 2 years ago

Replying to @sobjornstad

all I've ever had to do to install Anki on any Linux [...] is download the Linux package/tarball from the website, extract it, and run ./install [...] Although I'd certainly prefer to be able to pull it through my package manager, especially for ease of updates, I can't really accept that this is difficult.

Happy for you, but this is irrelevant. By "hard to build" I mean specifically for distro packagers; personal ad hoc success isn't a proper argument.

[...] There are just a lot of ways to install software on Linux right now, and you kind of have to deal with that.

Call me a neatnik, but I never install packages system-wide without a package manager. What's more, for regular open-source software like Anki, nobody should expect to have to install it without a package manager.

Also, @lolilolicon, I don't know how long you've been using Anki, but over those same 10 years most distros' packages of Anki have always been out of date, some of them horrendously so (e.g., so old they're incapable of syncing with the current version of AnkiWeb).

All the more reason to raise this discussion, don't you think? It says Anki has been challenging to package for quite long. And from what I hear the build system has been overhauled multiple times, each time requiring packagers to rewrite their build scripts. This could have been discouraging for two reasons: the upfront effort of adapting the build scripts, and the not unreasonable feeling that this is not the last such effort.

Maybe the new build system has made things a bit more difficult, but it's not by any means like we went from everything being neatly up-to-date to everything being a mess with that single change.

I never said Bazel changed things for the worse. Looking at the build script for anki 2.1.35-3 in Arch Linux, which required 5 patches plus 3 additional source tarballs, it's clear that anki hadn't been easy to build even before Bazel. Imagine maintaining a build script that required substantial effort to get right, and then have the whole thing scrapped, only to be met with a brand new build system that looks like it will take even more effort on your part.

In fact, my conjecture (not based on any evidence) is that perhaps Bazel was adopted to make Anki easier to build for packages, developers having realized Anki had been too difficult to build for packagers to catch up? After all, adoption by distros is crucial for Anki's wider adoption. (@PureTryOut's words sound harsh, and I'm not so confident Bazel is the problem, but he is correct in emphasizing the importance of the official distribution channels for a Linux user.)

In any case, "it's always been pretty bad" is no argument for not making things better now. This is not "if it ain't broke" cuz it is.

OceanS2000 commented 2 years ago

It's good to have an alternative, but as @OceanS2000 demonstrates, even this doesn't always work. More to the point, I think this discussion should be around the challenges of building & packaging Anki for distros.

To be clear, I believe my case is very exotic here: nowadays nearly every Linux distro is using systemd unless you are running Gentoo or Slackware. Even on Gentoo, it is rare to set USE= -systemd -elogind (elogind provides a somehow compatible libsystemd.so.0). So the binary release will work for most people and I'm not saying the current binary release is bad.

When I configure my system this way, I'm kind of prepared for such thing to happen and it is easy (for me) to make the binary release work on my machine.

Why Bazel? Are there better alternatives?

To my knowledge, Bazel might be the best solution here. Anki uses Python, Rust and Typescript. And all 'packager-friendly' build tools (pip, cargo) for them are only designed for mono language repo.

Other than Bazel, I can only come up with custom written Makefile or build scripts in python, which is arguably more difficult to maintain and more fragile when build environment changes.

The problem for Bazel, as it is designed by Google, is usually meant to be used in a corporate's monorepo, where all code including all dependencies is nicely sitting at the same place. This means the usual way to deal with dependencies in Bazel is to just fetch it. However, nearly all distribution packaging guideline requires no network connection during build and no vendor dependencies. But it doesn't seem to be what Bazel designers worry about.

I really appreciate the FreeBSD developers' hard work here. Indeed, as you can see nearly entire Bazel build definition is rewritten, and it is quite involving. But to me it is a good starting point for other distros' packagers including me.

As for advice for improvement, I suggest maybe some of FreeBSD's patch to the build system can be adopted upstream. I need some more time to read through the FreeBSD package but I believe Bazel can be made better than Makefile solution.

Nocifer commented 2 years ago

Just out of curiosity, can someone explain to me what is so hard about running a

bazel build --config opt dist

and building Anki?

Maybe I'm missing something here, but I'm under the impression that the drama has been cranked up to Over 9000! levels for no real reason.

If this were about possible licensing issues, or the need to use Java, or the fact that Bazel is a derivative of a proprietary Google tool, or even because the previous build process was a mess and people are so fed up that they don't want to bother anymore (just a hypothetical scenario, I have no personal experience with building Anki before Bazel was introduced); I'd certainly understand it.

But Bazel being "difficult for package maintainers to understand and use"? Really now? To the point that nearly all distros carry a terribly outdated Anki package? And some like Arch are now even dropping it from their repos altogether because "no one could understand how it works"?

Again, it really seems like I'm missing the elephant in the room here, so I'd appreciate it if someone could kindly point me to it.

Also, regarding this:

However, nearly all distribution packaging guideline requires no network connection during build and no vendor dependencies. But it doesn't seem to be what Bazel designers worry about.

How is this a problem when most all distros distribute their packages in pre-built binary form? I can understand it for the few outliers (in terms of package management) like Gentoo, but why should Average Joe care how a package was built before it was given to them to install via their distro's package manager?

Finally:

I just tried to build anki-git from AUR. The build process seems scary (java + python + node + rust) but it does work and took 4m30s on our 40-core server.

Weird. In my case the build only takes a couple of minutes on my 12 core desktop (Ryzen 3600). And how is the build process scary? Because it uses more than one language? Or because it uses multiple files to orchestrate the build process instead of a single 10k LoC makefile?

escape0707 commented 2 years ago

@Nocifer I saw your comments on AUR and I agree most of them. But just a note here:

However, nearly all distribution packaging guideline requires no network connection during build and no vendor dependencies.

I believe he mentioned packaging guideline which means the regulations a packager writing the build script should conform to. I guess this is because package builder should be able to check authority and integrity independently and apply patches like disabling auto-update or something when needed.

About anything else? I'm actually too inexperienced to packaging and writing PKGBUILD for anki-bin so I'm still reading the manuals XD.

lilydjwg commented 2 years ago

On Thu, Sep 23, 2021 at 06:29:59AM -0700, Alexander Michalopoulos wrote:

However, nearly all distribution packaging guideline requires no network connection during build and no vendor dependencies. But it doesn't seem to be what Bazel designers worry about.

How is this a problem when most all distros distribute their packages in pre-built binary form? I can understand it for the few outliers (in terms of package management) like Gentoo, but why should Average Joe care how a package was built before it was given to them to install via their distro's package manager?

This is a problem about reproducibility (and security). To be able to reproduce a package, nothing should be changed over time. So random numbers with fixed seeds (e.g. Python) and fixed timestamp (there is an environment variable that many tools respect). Mostly importantly, no Internet, because data from the Internet can change over time.

Finally:

I just tried to build anki-git from AUR. The build process seems scary (java + python + node + rust) but it does work and took 4m30s on our 40-core server.

Weird. In my case the build only takes a couple of minutes on my 12 core desktop (Ryzen 3600). And how is the build process scary? Because it uses more than one language? Or because it uses multiple files to orchestrate the build process instead of a single 10k LoC makefile?

Not only more than one language, but there are four languages (two of them I don't feel good about). And I don't know what's going on under the hood. Quite a few files in ~/.cache are touched. Perhaps it has downloaded a lot of data from around the world.

The output is mystery to me: what is //:dist and other // things? What are those linux-sandbox and ex processes I saw in htop? Where did they come from? Just built or from the Internet?

To be short, it's beyond my understanding, and goes into the world of magic.

-- Best regards, lilydjwg

OceanS2000 commented 2 years ago

Also, regarding this:

However, nearly all distribution packaging guideline requires no network connection during build and no vendor dependencies. But it doesn't seem to be what Bazel designers worry about.

How is this a problem when most all distros distribute their packages in pre-built binary form? I can understand it for the few outliers (in terms of package management) like Gentoo, but why should Average Joe care how a package was built before it was given to them to install via their distro's package manager?

Because most distros try not to use binary release build by upstream. Instead, they build packages from source release on their own. For example, here is the build farm used by Ubuntu. So, if package maintainers of distro cannot build the package cleanly, they just have no package to put into the package manager!

As for why distros doing so, there are a few reasons:

  1. Some packages, especially these fundamental ones like kernel and glibc, do not provide binary release at all. This is usually because there are no way for them to just 'extract and work', and these details are covered by distro packagers.
  2. By building software from source release only, you only need to trust your distro's maintainers are good people and they will not add malicious code into the software. If they use upstream binary directly, you need to additionally trust the upstream actually build from the published source code without secret 'sauce'.
  3. Regarding the last point, some distros like Debian and Arch are heading for reproducible builds, so you can verify how the distro build the package on your own. So you can even choose to not trust them.

Just out of curiosity, can someone explain to me what is so hard about running a

bazel build --config opt dist

and building Anki?

Firstly, it will not work on my computer.

anki % ./scripts/runopt
INFO: Analyzed target //qt:runanki (0 packages loaded, 3 targets configured).
INFO: Found 1 target...
Target //qt:runanki up-to-date:
  bazel-bin/qt/runanki
INFO: Elapsed time: 0.300s, Critical Path: 0.05s
INFO: 4 processes: 4 internal.
INFO: Build completed successfully, 4 total actions
INFO: Build completed successfully, 4 total actions
/home/<username redacted>/.cache/bazel/_bazel_<username redacted>/c121bf9bd26c8131616486fc51130231/execroot/ankidesktop/bazel-out/k8-opt/bin/qt/runanki.runfiles/pypi__39__stringcase_1_2_0/stringcase.py:247: DeprecationWarning: invalid escape sequence \W
  return re.sub("\W+", "", string)
Traceback (most recent call last):
<traceback redacted due to containing home path>
    from PyQt5.QtNetwork import QLocalServer, QLocalSocket, QNetworkProxy
ImportError: libgssapi_krb5.so.2: cannot open shared object file: No such file or directory

And it is because it try to fetch dependencies online instead of using locally installed one:

anki % cd /home/<username redacted>/.cache/bazel/_bazel_<username redacted>/c121bf9bd26c8131616486fc51130231/execroot/ankidesktop/bazel-out/k8-opt/bin/qt/runanki.runfiles
runanki.runfiles % ls -alh pyqt5
total 8.0K
drwxr-xr-x 1 <username redacted>  204 Sep 24 00:24 .
drwxr-xr-x 1 <username redacted> 1.4K Sep 24 00:24 ..
drwxr-xr-x 1 <username redacted> 2.7K Sep 24 00:24 PyQt5
drwxr-xr-x 1 <username redacted>  106 Sep 24 00:24 PyQt5-5.15.2.dist-info
drwxr-xr-x 1 <username redacted>  100 Sep 24 00:24 PyQt5_sip-12.8.1.dist-info
drwxr-xr-x 1 <username redacted>   74 Sep 24 00:24 PyQtWebEngine-5.15.2.dist-info
lrwxrwxrwx 1 <username redacted>   94 Sep 24 00:24 py.typed -> /home/<username redacted>/.cache/bazel/_bazel_<username redacted>/c121bf9bd26c8131616486fc51130231/external/pyqt5/py.typed
lrwxrwxrwx 1   <username redacted> 97 Sep 24 00:24 __init__.py -> /home/<username redacted>/.cache/bazel/_bazel_<username redacted>/c121bf9bd26c8131616486fc51130231/external/pyqt5/__init__.py
runanki.runfiles % rm -rf pyqt5/*
runanki.runfiles % cd pyqt5
pyqt5 % ln -s /usr/lib/python3.9/site-packages/PyQt* .
pyqt5 % cd ../ankidesktop/qt
qt % ls
aqt  bazelfixes.py  runanki  runanki.py
qt % ./runanki
... Now it works fine

This is why packaging guidelines are against fetching any binary dependency online. And in order to make the build adhere to these guidelines the build definition has to be patched.

But Bazel being "difficult for package maintainers to understand and use"? Really now? To the point that nearly all distros carry a terribly outdated Anki package? And some like Arch are now even dropping it from their repos altogether because "no one could understand how it works"?

Yes. It is mainly because (1) it is mostly designed for corporate monorepo and more importantly (2) there is few open source software using Bazel. To my knowledge, Anki is the first and only open source application designed for daily drive that using Bazel. So most distro maintainers out there just have no experience about it.

I just tried to build anki-git from AUR. The build process seems scary (java + python + node + rust) but it does work and took 4m30s on our 40-core server.

Weird. In my case the build only takes a couple of minutes on my 12 core desktop (Ryzen 3600).

It is mostly dependent on your network connection speed. On my computer the most time is consumed by fetching rust toolchain (a ~250 MB download). If your connection is bad then it can take up to 10 mins(!) for me... Once everything is downloaded the actually build process is fast.

OceanS2000 commented 2 years ago

@lilydjwg :

Quite a few files in ~/.cache are touched. Perhaps it has downloaded a lot of data from around the world.

Yes. To my knowledge the following are fetched and put in ~/.cache/bazel:

Also, all build output are put into ~/.cache/bazel.

The output is mystery to me: what is //:dist and other // things?

These are Bazel's notation for build targets. Instead of using plain text labels as Makefile Bazel uses this url-like notation to cope with more complex project structure.

What are those linux-sandbox and ex processes I saw in htop? Where did they come from? Just built or from the Internet?

They are part of Bazel. Basically Bazel uses its own sandbox to enforce dependency visibility rules so during build it can only see files exported in *.bzl build definition.

It is a good feature for development as basically it gives you a virtualenv but for all languages. However, this also makes patching the build definition to use dependencies from package manager quite a non-trivial work.

Nocifer commented 2 years ago

@escape0707 Online fetching is a point I can see as potentially problematic, though again, mostly in a scenario where the user builds from source, not when a package is distributed to them in binary form (which then makes this problem more a matter of principle and/or effort on the part of the packager than anything else).

Anyway, it's a valid concern and a reason to push against such a build process, but IMHO hardly a reason for keeping a package outdated for a year and a half or even dropping it completely from the repos. That's a tad too extreme.

@lilydjwg

This is a problem about reproducibility (and security). To be able to reproduce a package, nothing should be changed over time. So random numbers with fixed seeds (e.g. Python) and fixed timestamp (there is an environment variable that many tools respect). Mostly importantly, no Internet, because data from the Internet can change over time.

Yes, I'm beginning to get it now that lack of strict reproducibility is the main gripe people have with Anki's use of Babel. But again, as I just said to @escape0707, dropping a package from the repos because it's not reproducible sounds a bit extreme, especially when 1) full system reproducibility is still a future goal and not an established norm, and 2) we're talking about an educational (aka casual) application like Anki and not a system critical tool, so a bit of leeway would be advisable.

@OceanS2000

Because most distros try not to use binary release build by upstream. Instead, they build packages from source release on their own.

I never said that distros use or should use the ready made binary releases from upstream, I said that distros distribute binary packages and that Average Joe isn't affected by what goes on behind the scenes in order for these binary packages to be built. Barring the concept of reproducibility, if a packager manages to build a proper binary package and distribute it to the users, then this package is frozen in time and the users will never be affected by such things as online fetching during build, etc.

So in fact, dropping anki into the AUR makes the situation even worse, because now the users will be affected by the potential pitfalls of the build process, because they'll have to build the package from source themselves.

some distros like Debian and Arch are heading for reproducible builds

This is a very valid point/concern against systems like Bazel, but at the risk of repeating myself too many times, that's not really a reason to drop a package from the repos, especially a casual educational application like Anki. Remember that the point of contest here is not really whether Bazel is a good tool or even whether Anki is making good use of it, but whether dropping anki into the AUR was a good move.

Firstly, it will not work on my computer. And it is because it try to fetch dependencies online instead of using locally installed one:

I'd assume that it not working on your computer has something to do with you using Gentoo and having customized libraries, as you described in your previous post. And as you also said, having a heavily customized Gentoo system means you know to expect such incompatibilities to occur from time to time.

Still, trying to fetch even common system components like PyQt5 and the like is not good practice in my books, and now I begin to see why Anki is getting all this flak. It's one thing to fetch stuff like rust or npm dependencies, and another thing to fetch system components that can produce incompatibilities even in your typical binary distro.

Yes. It is mainly because (1) it is mostly designed for corporate monorepo and more importantly (2) there is few open source software using Bazel. To my knowledge, Anki is the first and only open source application designed for daily drive that using Bazel. So most distro maintainers out there just have no experience about it.

Lack of experience does not mean the tool is hard to use, it just means that we lack the experience to properly use it and we must first learn how it works. The debatable point here is whether Bazel brings enough to the table to warrant package managers spending their valuable time to do so.

Also, just for the record, there is at least Mozc which has also switched to using Bazel since a few months ago, and it's actually the application that first made me aware of Bazel in the first place.

It is mostly dependent on your network connection speed. On my computer the most time is consumed by fetching rust toolchain (a ~250 MB download). If your connection is bad then it can take up to 10 mins(!) for me... Once everything is downloaded the actually build process is fast.

Nah, that's can't be it. It's more probable that the person I was replying to made a typo and simply meant "4-core", not "40-core".

However, this also makes patching the build definition to use dependencies from package manager quite a non-trivial work.

Now this is the question I really wanted to ask in order to make up my mind on whether it's Bazel or Anki's devs at fault here: does Bazel force us by design to use bundled resources instead of local, system-wide ones? If that is so, then indeed Bazel is an unsuitable tool for building Linux distro packages, and Anki's at fault for choosing to use it. If not, then it's only Anki's fault for not creating a build process that respects Linux packaging standards. So either way, it seems Anki is at fault :P

PS: I looked a bit more into reproducibility and, given that this is the number 1 gripe most people seem to have with Bazel, you gotta love the irony that the one and only Platinum sponsor of the Reproducible Builds project is Google's Open Source Security Team (Bazel is Google software).

escape0707 commented 2 years ago

It's more probable that the person I was replying to made a typo and simply meant "4-core", not "40-core".

Uh well, at least I can confirm that lilydjwg is a very experienced packager and Arch user who is a main manager of ArchlinuxCN community and the unofficial repo.

I think by referring to the packaging machine's power she means at least fetching all the resources and build them is a painful problem for people with a less powerful rig, just the same point as you supported. Also, I don't think this will be the main reason for unofficial repos to refuse to package and distribute it, it's mainly security and packager's effort which are put into concern.

Also, just for the record, there is at least Mozc which has also switched to using Bazel since a few months ago

As of fcitx5-mozc-ut, it didn't make into the official repos either. Only the non bazel version without ut dictionary is there. I had a chance to ask farseerfc why he didn't consider packaging the ut version, and he said the vanilla version is sufficient for his scenario of using Japanese heavily everyday.

Though, I wish these packages could reside in the official repos, too.

ropery commented 2 years ago

Replying to @Nocifer

Remember that the point of contest here is not really whether Bazel is a good tool or even whether Anki is making good use of it, but whether dropping anki into the AUR was a good move.

You're going off-topic. The point of contest here is definitely not whether @kyrias made the right decision dropping the package. (I myself commend his disciplined approach.)

[...] which then makes this problem more a matter of principle and/or effort on the part of the packager than anything else [...] I said that distros distribute binary packages and that Average Joe isn't affected by what goes on behind the scenes in order for these binary packages to be built.

This discussion is specifically about making the packager's job reasonably easy. Be reminded that Arch Linux Trusted Users (and packagers of most distros) are volunteers. If the software makes unreasonable demands on their time & effort, it only makes sense to drop what is unmaintainable; the AUR exists to ensure the quality of the official repos, while giving users wider options and a place to collaborate.

The big question with Bazel is whether Anki can reconcile the Bazel way of doing things with distro packagers' requirements of independent verification and scrutability of the build process, etc.

lilydjwg commented 2 years ago

Nah, that's can't be it. It's more probable that the person I was replying to made a typo and simply meant "4-core", not "40-core".

It is 40-core, although there might be a concurrent build job to at most reduce the available CPU power to half. It's also in a Europe datacenter with good network connectivity.

I didn't know that it fetched rust toolchain (the PKGBUILD requires the rust package). This would take a lot of time for mainland Chinese users if they don't setup a proxy / mirror (and since it's like a virtualenv thing, I don't know if setting these up in it is easy or not). Also, would it pick up npm / pip mirror settings for those users?

dae commented 2 years ago

Firstly, for some perspective: less than 3% of Anki users are on a Linux machine. For the other 97% of users out there on a Windows or Mac machine, binary packages are the norm.

The comments about reproducibility in Bazel are somewhat off the mark. The dependencies Anki pulls in are pinned to specific versions, and the majority of them are checksummed, meaning changes to the remote files will result in the build aborting with a warning.

The pinning is important - they are versions we have tested and know will work, and it is not uncommon for a slightly older or slightly newer version of a dependency to break things. The dependencies are downloaded during build because that is the most convenient - to do otherwise, you'd need to install all the required dependencies yourself, and any version you accidentally got wrong could result in things breaking. Having a packager do that for you makes things easier, but even they don't always get it right, and there have been many Linux-specific issues over the years caused by distros switching out a different version without testing it properly, such as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=980344

There are no technical reasons preventing packages from overriding the dependencies if they wish. Bazel provides tools to override arbitrary dependencies with other versions in other locations. So this is not a case of packagers not being able package the app - it's just that it's more work than they want to do.

That's not to imply they are being lazy - the reality is that modern software development practices are somewhat at odds with the way Linux distros have historically swapped out dependencies with their own, and you can see this playing out with other projects as well. I don't really think it's realistic to expect developers to stand still for the sake of packagers - the onus is really on the distributions to keep up with the times. It may take them some time, but they'll likely get there in the end, and I believe Debian already have a group looking into improved tooling for packaging of Bazel projects.

Nocifer commented 2 years ago

@escape0707

Uh well, at least I can confirm that lilydjwg

Ah, sorry, I didn't even realize that the "40-core person" was @lilydjwg. I guess that if it were a typo she would have said so in her reply to me, so it's probably like @OceanS2000 said, that the stated time includes resolving dependencies over the network etc.

Also, I don't think this will be the main reason for unofficial repos to refuse to package and distribute it, it's mainly security and packager's effort which are put into concern.

I didn't mean to imply that, it's just that while skimming over the comments it hit me as weird, so I commented on it for the sake of discussion.

As of fcitx5-mozc-ut...

Yeah, again, I only mentioned that for the sake of discussion. I'm not involved enough with the Arch community to know the whys and hows of what package gets a place in the repos and what gets left out. I mean sure, Bazel notwithstanding I too wish that the UT packages were included in the repos (would save me the bother) but it's not really that important, especially compared to other missing packages like e.g. lib32-gst-plugins-{bad,ugly}.

@lolilolicon

You're going off-topic. The point of contest here is definitely not whether @kyrias made the right decision dropping the package. (I myself commend his disciplined approach.)

Well, I thought the point of contest was whether Bazel as a tool is too convoluted to the point that package maintainers are not able to understand and use it and thus are forced to drop packages that make use of it, like in the case of Anki. That struck me as very weird (because using Bazel is not that hard) so I questioned it. Now, since it turns out that @kyrias (on a side note, I thought the maintainer was @demize?) dropped the package due to other issues like lack of reproducibility, security concerns, the online fetching of dependencies, etc, then that's another matter and I'm not about to debate the decision, not the least because I don't really care about it, but also because I don't have any delusions that I'm knowledgeable enough to debate it.

As for the rest of what you said, it seems to me that my first hunch about "bad blood" was correct and I've inadvertently stumbled upon a long-standing internal affair/debate of the Arch community, and most people involved have already established some rather strong opinions on the matter, and I really have no desire to debate policies or established processes, so I'll kindly excuse myself out.

@lilydjwg Thank you for your reply but really, there is no need for you to explain yourself to me when it was a mistake on my part. It's just that it hit me as weird that a 40-core server would take more time to compile something than my petty 12-core desktop, and I never even considered that you may have been talking about the whole packaging process and not only the build stage.

escape0707 commented 2 years ago

And I believe Debian already have a group looking into improved tooling for packaging of Bazel projects.

Glad to see there are ongoing actions on this: https://salsa.debian.org/bazel-team/meta ... but oof this looks actually hard and takes many resources and efforts from both the Debian team and the bazel team to realize it ever since the COVID-19. And they are still not there yet.

I think the original goal of the discussion for sure cannot be meet now as it takes time for bazel to get integrated into Linux packaging tool sets, meanwhile Anki's development need this build system right now to move forward faster and easier.

So for now, the measures that can be taken are:

OceanS2000 commented 2 years ago

Well, if now we can not make Bazel easy to work with, why not try to at least get something usable? This is my take: https://github.com/OceanS2000/anki/commit/e4c820ced01d330418260bf6e8bce3d4a9bedd40

@dae Does you guys feel comfortable about releasing a source tarball containing all auto-generated source using this build rule along with pre-built binaries? I think at least it can provide something to work on before everyone adopts Bazel :)

The following is a full description about my commit:

This fork contains an additional build rule for a 'source distribution tarball'.

The original repo uses Bazel to build, which has been causing some headcache for distribution maintainers. This additional build rule tries to provide a stopgap solution before using Bazel in distribution packaging is well documented.

The idea is, a 'source distribution tarball' is created when releasing by

bazel -c opt '//src-dist:src-dist'

It will generate a tarball includes:

And then the software can be built without hassle of Bazel relatively.

Why prebuilt TypeScript part?

While we love all-source builds, vendoring some javascript code seems fine for me --- we can not really do much about its dependencies and our web browser downloads them all the day.

How to continue the build process, then?

You will need two snapshot from https://github.com/ankitects/anki-desktop-ftl and https://github.com/ankitects/anki-core-i18n for translations. The exact commit for them is documented in repos.bzl file, which is included in the generated tarball.

Extract them into the same working directory, and then build the Rust library:

work % ls
bazel-dist  core-i18n  desktop-ftl
work % BAZEL=1 \
    PYO3_PYTHON=/usr/bin/python3.9 \
    PROTOC=/usr/bin/protoc \
    BUILDINFO=$PWD/bazel-dist/buildinfo.txt \
    PROTO_TOP=$PWD/bazel-dist/proto/.top_level \
    BACKEND_PROTO=$PWD/bazel-dist/rslib/backend.proto \
    RSLIB_FTL_ROOT=$PWD/core-i18n/l10n.toml \
    EXTRA_FTL_ROOT=$PWD/desktop-ftl/l10n.toml \
    cargo build --manifest-path $PWD/bazel-dist/pylib/rsbridge/Cargo.toml --verbose --release

work % find . -name 'librsbridge.so'
./bazel-dist/target/release/librsbridge.so
./bazel-dist/target/release/deps/librsbridge.so
work % find . -name 'strings.json'
./bazel-dist/target/release/build/anki_i18n-15e40669e4befd1b/out/strings.json

librsbridge.so and strings.json are what we want. Move them into anki/_backend:

work % mkdir usr-share-anki
work % mv bazel-dist/pylib/anki usr-share-anki
work % mv bazel-dist/qt/aqt usr-share-anki
work % cp ./bazel-dist/target/release/librsbridge.so usr-share-anki/anki/_backend/rsbridge.so # The name is significant!
work % cp ./bazel-dist/target/release/build/anki_i18n-15e40669e4befd1b/out/strings.json usr-share-anki/anki/_backend

Generate the python file from strings.json:

work % export PYTHONPATH=usr-share-anki
work % cd usr-share-anki/anki/_backend
_backend % python3 genbackend.py generated.py
_backend % python3 genfluent.py strings.json fluent.py

Now (hopefully) you can try it by python3 -c 'import aqt; aqt.run()'.

To finish the build process, a launcher can be generated from qt/tools/runanki.system.in

#!/usr/bin/env python3

import sys

sys.path.append("@PREFIX@/share/anki")

import aqt

aqt.run()
sakkamade commented 2 years ago

Firstly, for some perspective: less than 3% of Anki users are on a Linux machine. For the other 97% of users out there on a Windows or Mac machine, binary packages are the norm.

Now, this is not too fair comparison, is it. It is actually an awful lot if one would consider usage share of Linux distributions:

https://store.steampowered.com/hwsurvey OS Share
Windows 96.49%
OSX 2.49%
Linux 1.02%

Yes, Steam may not be the best place to search for this kind of data, but it says a lot.

escape0707 commented 2 years ago

Yeah, as a Kant believer, I also felt that mentioning of mark share on a FOSS educational project will not solve any problem if not creating more.

ropery commented 2 years ago

It's also unclear on what that 3% is based. If it's by downloads, the Linux number would be a significant underestimate, since most Linux users get Anki from their own distro's repos, invisible to Anki download servers.

Another thing is that, presumably, there are would-be Linux Anki users who are deterred because their distro does not ship a working/up-to-date Anki package: they give up Anki, or opt to use only AnkiDroid on their phone, for example.

There's every reason to bet that Anki is much more widely adopted among Linux users than Windows users. Still, by sheer volume, we are no doubt a minority. Yet, given their more technical background, Linux users are much more likely to report bugs & give useful feedback to Anki developers, a good reason to promote adoption in Linux-land.

appetrosyan commented 2 years ago

I'll add my two cents to this discussion. Firstly, I've been trying to build anki for the past week from source. Either my system is far too unclean or Bazel solved the only problem that Makefiles had — it no longer produces an executable at the end of the process. I would imagine that the choice was induced by other factors: Makefiles are harder to write/read and they're not 100% portable. For a project as large as Anki, there may be benefits to Bazel that are completely invisible on Linux. Now, in my humble opinion a lot of the problems could have been solved by cleaning up the codebase, but I have the luxury of writing mostly smaller programs that rely on good design choices, rather than feature creep.

The removal of Anki from the official repositories is not very good for Anki. Recall that Steam Deck is about to be unleashed at the world: it's a small computer running a version of Arch Linux tailored to that small portable device, that is very likely to be used for Anki revisions. If I were you, the developers of anki, I'd be very interested in not missing out on being officially supported (and featured) on the app store of a device that has the potential to rival Nintendo. You not being included in the official binary repositories of said device's OS is a huge miss, particularly if you want to capitalise on the fact that the Steam Deck is likely to be popular among your core demographic: college and school kids who work and play. So whatever is the actual reason why you got kicked off of the official repos, I'd be working hard to fix it.

Another issue that I'd be more worried of, is the fact that Anki is likely to get forked in this eventuality. Do you want to compete with a fully compiled and fully KDE-compliant fork of Anki that is installed instead of anki on the aforementioned Steam Deck devices. I assure you that cleaning up the 2.1. family removing the Windows/Mac OS specific code and porting it to Kirigami is a lot easier than convincing developers that already kicked your application off of their repositories to take it back in. The install base on Linux may* be smaller now, but you would missing out on a golden opportunity.

As an end user on Linux, there's a lot to not like: it doesn't respect the Qt theme, it doesn't utilise the fact that it's running on a desktop to its fullest potential (in fact not a lot in general), It's not very cross-platform, since the IOS version costs an arm and a leg, and to top it all off it's not in the official repositories. Read that as people don't see it in the software centre. Being unable to install it (and I am unable to install a newer version), having to choose between a version that doesn't build, a version that's a docker container, and a version that needs Java and Rust to build, makes me consider forking the old version and porting it to C++. This is not a good look for Anki, and frankly something that I'd be concerned about in light of everything I mentioned so far.

escape0707 commented 2 years ago

Either my system is far too unclean or Bazel solved the only problem that Makefiles had — it no longer produces an executable at the end of the process.

This seems humorous to me, but to be honest, the excutable is here: https://github.com/ankitects/anki/blob/main/qt/runanki.py

Take a look at https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=anki-git#n102

Arbitrate3280 commented 2 years ago

lol, this thread started out fine, but I wasn't ready for Steam Deck to come into play and "it's going to get forked because it's hard to build and SOMETHING ABOUT KDE", and THE IOS APP IS SO EXPENSIVE.

@lolilolicon, are you still getting anything useful out of this issue? From dae answers Bazel is going to keep being used. OceanS2000 provided something that might help package mantainers, maybe a pull request can be opened and discussed there.

ropery commented 2 years ago

I would like to keep this discussion open, seeing the issue remains unresolved.

Given time, people more knowledgeable than us will chime in. Downstream packagers may voice their requirements explicitly, and offer suggestions.

I say let's ignore the occasional impertinence and ludicrous spiel.

appetrosyan commented 2 years ago

@escape0707 I did. I also tried installing a variety of PKGBUILDS, (including my own) to no avail.

lol, this thread started out fine, but I wasn't ready for Steam Deck to come into play and "it's going to get forked because it's hard to build and SOMETHING ABOUT KDE", and THE IOS APP IS SO EXPENSIVE.

@Vitorvlv Well, in all fairness this is half-serious. I don't get the arguments for "only X percent of our users use Linux", so I wanted something equally ridiculous.

I personally have no issues with using the old versions of Anki indefinitely. And judging by how many times the build system changed for Anki, I don't think Bazel is going to cause many problems long-term. These are typical issues for large projects. For a while, using pip to install the binary, or creating an AUR package like python-anki isn't too much of a hassle. I was surprised anki was kicked out of the official repos, so I wanted to see what was going on. From this discussion it seems that going either way will take a ton of time.

escape0707 commented 2 years ago

I also tried installing a variety of PKGBUILDS, (including my own) to no avail.

Well, sounds like a problem which should be asked at AUR before reporting here as a standalone issue. I'm fine building all of those.

dae commented 2 years ago

OceanS2000@e4c820c

@dae Does you guys feel comfortable about releasing a source tarball containing all auto-generated source using this build rule along with pre-built binaries? I think at least it can provide something to work on before everyone adopts Bazel :)

Bundled Javascript is not really "source code", and I suspect the majority of distros would not accept a package built from partially-compiled sources. No changes to Anki's build is required in any case - you should be able to extract those files from the existing wheels available on PyPi - they're just regular zip files.

To the people complaining about system themes and systemd, I repeat my recommendation from the first reply: the wheels on PyPi are easy to install, and can use the Python/PyQt you have already installed on your system.

appetrosyan commented 2 years ago

Pip installing anything on your system shouldn’t be a recommendation. I say that as someone who maintains four PyPi packages that I wrote. While it does indeed use the system Qt libraries so did the old version of anki, which IMHO is also perfectly fine. A desktop application should be allowed on the main repositories, and I think that what’s happening here is a temporary roadblock, which can be perfectly waited out with an older version of anki.

OceanS2000 commented 2 years ago

Bundled Javascript is not really "source code", and I suspect the majority of distros would not accept a package built from partially-compiled sources.

While bundled javascript is not technically 'source code', afaik, the whole conversion starts from Arch dropping anki from community repo, where bundled javascript is allowed just fine.

Moreover, Bazel itself include some generated source files in its release to help downstream compile it, so I think the request is well justifiable -- at least for a temporary solution.

No changes to Anki's build is required in any case That is a valid point.

At the end of day my goal is to use Bazel to build Anki for distribution, at that time some change to Anki's build must happen to make it look for local repos. I'll come up with a PR next month probably.

Yakkhini commented 2 years ago

btw, fcitx is not supported after 2.1.40 version. I'm an archlinux user and my local anki software was downloaded in AUR. The last version without this bug is 2.1.35. (I tried lots different version from official download page ) Maybe the issue you talk here cause this. I noted @escape0707 mentioned the input method (https://apps.ankiweb.net/downloads/archive/) needs an extra patch in Qt5 bundle. Could you fix this in next version? It would help a lot. ;)

criatle808 commented 2 years ago

To benefit from the security feature of Open Source, we need something that can be built offline from source code to fulfill the requirement of a secure, hence comprehensible built-process as part of the distribution packaging. As far as I can see it, pure Open Source distributions are locked-out from using Anki in the future due to Anki being hard to pack if you don't want to pull source code or even binaries from the Internet during the build process. As a user we rely on the package manager of our trusted distribution being able to build offline from source and then provide the package in the distribution's repository, so we can install it from there. I appreciate all the work that went into Anki, and it's Open Source, and I know it's the developer's work and choice, so only for making you understand why, please, continue to support the requirements of our trusted Open Source distributions by supporting an offline build process. Just because the majority of the users doesn't care and is on Windows or even Linux downloading a binary installer doesn't make it a feasible solution for the true Open Source Community that has contributed so much to software development and security, including software that Anki is based on.

On the latest Fedora release from last month, due to this issue, the Anki Version is still 2.1.15 from 2 years ago, and it crashes when closing the app or opening the Preferences dialog because it's incompatible with Python 3.10. It won't get fixed and updated and most-likely will be removed from Fedora and many other Linux distributions next year just because of Anki's current build-process.

Rogach commented 2 years ago

(disclaimer: I'm most familiar with Debian and Arch Linux packaging, so please correct me if I am missing something)

if you don't want to pull source code or even binaries from the Internet during the build process

As far as I know all packaging systems pull source code from the Internet somewhere during the build process - you still need to get the sources to your machine somehow. The rule "not pulling the files from Internet" comes from the common problems arising when random build scripts try to pull data without ensuring that received data exactly matches the expected checksums, resulting in non-reproducible builds and possible vulnerabilities. So distribution packaging requires that all downloads happen inside their own scripts/programs which are fully controlled by the distribution maintainers and can be ensured to be secure.

But Bazel doesn't suffer from the problem of non-reproducible network access - as far as I see all the downloads require explicit checksums, making the entire build process secure and reproducible. The only significant difference is that the downloads are happening during invocation of Bazel instead of makepkg. Same thing can be said for other popular tools, like npm or yarn - if everything is configured correctly then they will ensure that all downloaded packages explicitly match the checksums provided in the build definition files.

The distribution package maintainers may have doubts about whether the claims of reproducibility and security of any given tool are valid. That's entirely reasonable, but I think that it will be much easier if we will pour the resources into verifying the security of (common) build tools instead of trying to manually rewrite the build definitions to match our old build pipelines.

Please don't assume that I'm somehow promoting or supporting this proliferation of extremely complicated build processes and enormous amounts of dependencies. Quite the contrary, I'm very sad to see the old days of ./configure; make slowly fade away.

But software complexity is quickly rising, the amount of dependencies included grows. On top of that the whole ecosystem suffers from "regression to the mean" effect - if you have a project with huge dependency tree, you can't expect all of these dependencies to be of stellar quality. In fact, you are almost guaranteed that there will be some garbage included. There will be libraries that don't respect backward compatibility, you will have version mismatches, and lots of other interesting situations. It is entirely possible that in more convoluted cases you will need several different versions of a toolchain to even get some projects to build.

It's not reasonable to expect the successful projects to keep the old "simple" build tools - the ecosystem already moved onwards.

And on the other side it is unreasonable to expect the distribution maintainers to keep rewriting the build definitions to work with "simple" build tools. If previously packagers mostly only had to figure out what keys should be provided to configure invocation (I'm oversimplifying drastically, of course), now they need to basically rewrite the entire build definition - this is a multi-hour effort, a huge and unsustainable waste of time. We don't need to reach far for an example - just try creating a "proper" build of Anki here. Even BSD guys (the link to their build definition is somewhere above) decided that extracting all the dependencies and their checksums into the build file is not worth it - they simply use yarn invocation to download JS dependencies, for example.

In conclusion, I think we shouldn't fight against the rising complexity, lest we be left behind to watch a sad steamroller of binary flatpacks/snaps/containers conquering the world and burying our future. We need to adapt to this rising complexity, use the tools that allow us to live with it - and honestly Bazel doesn't seem to be such a bad starting point.

criatle808 commented 2 years ago

Thank you, I'm a programmer and not a package manager, so it's helpful you clarified what exactly package managers criticize of Bazel. You explain that it's because "the downloads are happening during invocation of Bazel instead of makepkg", and that "the distribution package maintainers may have doubts about whether the claims of reproducibility and security [of Bazel] are valid". I'm sure, if they didn't have a reasonable doubt, they would happily trust Bazel and keep the application, rather than deciding between rewriting the build process (which is a lot of work for them) or removing the application from the repository (which decreases the attractivity of the distribution).

So here we are. Package managers and Anki developers have each their own reasoning, there is no solution present yet, and the result is that Anki will be removed from most Linux distributions. It's like having removed a software from a mobile phone app store. That is really sad, for all of us who want to stick to installing from the distribution package system, but also for Anki. Sometimes it's not about statistics or the amount of users, but what kind of a user is being excluded. In this case it will be those who value security standards and the maintainability of their operating system and want to install only via their package system. And it's gonna happen, because package managers won't back down on their security standards, there is no solution yet, and the old Bazel-free Anki versions are broken now on current distributions.

appetrosyan commented 2 years ago

@criatle808 this isn't as sad as you think. The move to Basel was a calculated one, and Anki developers knew what they were getting into. They will get removed from most package repositories, and getting back in is going to take a not insignificant amount of time. This has less to do with technical issues of Basel and more to do with the present discussion and trust. The end users get the thin end of the wedge too, because while distribution package managers make sure that the app can run and that it has the right dependencies, and that these dependencies talk nicely to each other. Pip has no quality standards. After a while it ceases being a bad package manager and becomes a badly written app, and Anki will just not be as good.

At that time people might say, why not make it better, see this discussion and decide, that they"re better off forking the app, rather than contributing to it. This is really where the solution starts to come about. Anki doesn't have a significant user base on Linux. At the same time, there are Linux specific toolkits that could have benefited Anki, but didn't because it wouldn't work on Mac OS or Windows. This is where actually forking and making use of that fact can let Linux users enjoy coherent UX, while Anki developers don't need to worry about supporting a minority of their users. Since it's OpenSource, nobody's stopping anyone from doing that. Really all you need to worry about is the rendering, the rest is kinda simple.

So while it might seem sad for a proprietary application, since isn't one, this sort of "callousness" isn't really even offensive. I understand that the Mac Os and Windows install bases require a lot of attention. I understand that Basel was chosen because it addressed those issues. I also understand that Anki was written by mortal beings and re-writing it is not only feasible, but sometimes useful.

Don't be disparaged. One three things happens: 1) all is fine and Anki will keep using Basel, where either Linux distros come around and re-accept it into the repos or 2) they don't. You can use the older version. You can use the binary package. And you can also use the pip package. At that point 3) someone forks this and re-writes Anki in either GTK or Kirigami. These latter two programs fit into the system much nicer, because they use the system's paradigm of UX, rather than Anki's. Win-win.

criatle808 commented 2 years ago

Can we introduce a 2.1.15.x branch for compatibility updates of that pure Python version, so we can keep the version running that is accepted in Linux distributions (especially after it totally breaks with Python 3.10)? The compatibility fix for Python 3.10 went into version 2.1.49, but that commit is already down a path that introduced changes in 2.1.16 that were so severe so that many distributions didn't accept to increase versions, hence being unavailalbe for those distributions.

I am a bit surprised those major changes went into minor versions that are normally used for bug fixes. While it is discussable to take this decision of braking Linux distribution compliance by major refactoring, it is a totally different story to introduce major refactorings or new features incrementing minor version on the 2.1.x branch. Developers could have left the minor branch for bug and compatibilty fixes, as a pure Python package, while moving the development of the main branch into that new direction we see today, on a 2.2.x branch.

Can we at least open a minor branch in the current repository that keeps 2.1.15.x running on the latest versions of Python?

Bracket-H commented 2 years ago 
Traceback (most recent call last):
  File "runanki.py", line 3, in <module>
  File "/home/dae/venv/lib/python3.8/site-packages/PyInstaller-4.0.dev0+g2886519-py3.8.egg/PyInstaller/loader/pyimod03_importers.py", line 625, in exec_module
  File "aqt/__init__.py", line 21, in <module>
  File "/home/dae/venv/lib/python3.8/site-packages/PyInstaller-4.0.dev0+g2886519-py3.8.egg/PyInstaller/loader/pyimod03_importers.py", line 625, in exec_module
  File "aqt/gui_hooks.py", line 11, in <module>
  File "/home/dae/venv/lib/python3.8/site-packages/PyInstaller-4.0.dev0+g2886519-py3.8.egg/PyInstaller/loader/pyimod03_importers.py", line 625, in exec_module
  File "aqt/hooks_gen.py", line 18, in <module>
  File "/home/dae/venv/lib/python3.8/site-packages/PyInstaller-4.0.dev0+g2886519-py3.8.egg/PyInstaller/loader/pyimod03_importers.py", line 625, in exec_module
  File "aqt/qt.py", line 18, in <module>
ImportError: libsystemd.so.0: cannot open shared

Who is this dae? Why is there a hardcoded homedir of someone in any of this? What? Who? What's going on man? All I'm know is that I'm angry and this is some BS. Probably the fault of python because that thing is a plague upon the world.

I don't have a systemd system, why would ...you know...whatever. I'll just run Anki through wine or something.

criatle808 commented 2 years ago

@Bracket-H: This does not refer to building Anki for different Platforms, but is a problem that refers to the direct installation of Anki, so I suggest you open a separate bug ticket.

Apart from that, it would be helpul giving more information, e.g. what version you were trying to install, what system you are installing on, how you installed it (specific steps) ect. (In case you are interested in solving this issue rather than just complaining about it. ;)

Bracket-H commented 2 years ago

I got it to build from source. It created almost 4 gigabytes of bazel cache files in my home dir, that's almost as much as the entire system software package size of my distro. I don't know when this all changed, but I'm pretty sure Anki wasn't this bloated all the time. I opened a ticket with the request to unbloat it and remove all that rust and bazel crap, but instead I was linked to rick astleys never give you up and then this dae individual closed the issue.

This leads me to believe that this project is now moribund. I can smell the soy latte and estrogen pills clearly now.

criatle808 commented 2 years ago

The "unbloated" original version can be seen here, source code is 1.4 MB and pure Python code, the binary is 115 MB: https://github.com/ankitects/anki/releases/tag/2.1.15

Talking about solutions, it would be nice to keep that version compatible with current Linux distributions by backporting critical updates from time to time to that branch. There don't seem to be many anyways, as the 2.1.15 version is being used in distributions for years now, but sometimes there are updates to Python or Qt that break that version and need a small update.

It doesn't help to become too emotional here. We rather give constructive ideas or better code ourselves. If there is no Anki version that is compatible with true open source Linux distributions, I am thinking of creating a simple Anki version that can be cross-compiled to all platforms including mobile, iOS and Android.