Open deeplow opened 2 years ago
I checked on Dec. 12th 2022 the current availability of the python3-pyside6
package which we need for Qt6 in python. I checked these for the currently supported systems. It is not looking promising:
Distro | version | Qt6 | PySide6 |
---|---|---|---|
Ubuntu | 22.10 (kinetic) | :heavy_check_mark: | :x: |
Ubuntu | 22.04 (jammy) | :heavy_check_mark: | :x: |
Ubuntu | 20.04 (focal) | :x: | :x: |
Debian | 12 (bookworm) | :heavy_check_mark: | :x: |
Debian | 11 (bullseye) | :heavy_check_mark: | :x: |
Fedora | 37 | :heavy_check_mark: | :x: |
Fedora | 36 | :heavy_check_mark: | :x: |
Fedora | 35 | :heavy_check_mark: | :x: |
Another good website for checking this is repology (compare, for example, with PySide6).
For 0.4.1 we're aiming to make headway towards Qt6 support on Mac and Windows by direct bundling, with further testing and research for Linux distributions. Moving Linux distributions to Qt6 for 0.4.1 would be a stretch goal.
PySide6 won't be available for Fedora distributions any time soon. So we need to find a way to bundle it into our .rpm
ideally before it reaches EOL.
bdist_rpm
(will be deprecated)Note: In the process of looking into this issue, I found out that
bdist_rpm
is on its way to being deprecated (https://github.com/freedomofpress/dangerzone/issues/298).
bdist_rpm
is the build tool we're using for creating .rpm
files.
we generate python wheels for the missing packages.
pip install wheel
mkdir wheels
pip wheel --wheel-dir ./wheels PySide6
This will download the wheels for the missing packages and all of their dependencies. All we have to do after is packaging them into the RPM and install these wheels at install time.
Then we bundle them with bdist_rpm
. There may be two ways to do this:
Option B) by modifying the .spec
file generation. We need to add to the file extra extra lines to include %files
for wheels built in (1.) and perhaps a %py3_install_wheel
.
To achieve this, we can override the bdist_rpm
class and wrap its _make_spec_file() method (docs about extending commands). The final setup.py
would look something like this:
import distutils.command.bdist_rpm as orig
from distutils.core import setup
class bdist_rpm(orig.bdist_rpm):
"""Specialized Python source builder."""
def _make_spec_file():
spec = orig.bdist_rpm._make_spec_file(self)
# modify spec file here
setup(
...
cmdclass={'bdist_rpm': bdist_rpm},
...
)
Note: We should ensure that the installation of the PySide wheels are conditional (to ensure we don't overwrite package-manager-maintained ones) or that it somehow is installed in a virtual environment that only Dangerzone can use.
Bundling everything in our code may be another viable solution. We already do this for MacOS via CxFreeze, which also supports building RPMs with its relatively recent inclusion in its source code of the soon-to-be-deprecated bdist_rpm
.
Related docs: https://docs.python-guide.org/shipping/freezing/#freezing-your-code-ref
Instead of having the packaging files for fedora generated automatically from a setup.py
file, we can make them ourselves. https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/
We can use as the base the spec file generated by setuptools and keep developing it ourselves. It can be generated with python setup.py bdist_rpm --spec-only
.
Not sure this is fixed @apyrgio. Linux systems still run on it.
Oh, good catch, this should have been a "Refs".
As of 2023-06-13, I don't see any known distro, besides Alpine and Arch Linux that package PySide6. Also, the Qt5 support has officially ended in May 2023. Thankfully, MacOS and Windows builds use PySide6, but we immediately need to update our Debian/Fedora packages as well.
Regarding @deeplow's comment above, I'd be more in favor of option C, as it doesn't affect the package creation of Dangerzone, and because we can always update the package irrespective of the Dangerzone releases. For reference, this is the way Arch Linux packages PySide6: https://gitlab.archlinux.org/archlinux/packaging/packages/pyside6/-/blob/main/PKGBUILD
We had the same discussion on the SecureDrop Client side of things and decided that given that Debian is continuing to provide security updates, the migration timeline is not as critical as upstream EOL suggests, and we can wait until the Bookworm switchover: https://github.com/freedomofpress/securedrop-client/issues/1562#issuecomment-1464185785
That said, I don't know if Fedora applies security updates in a similar manner after a project has reached EOL upstream (@legoktm, tagging in case you have insights on that). And of course there are other reasons to make everything consistent.
That said, I don't know if Fedora applies security updates in a similar manner after a project has reached EOL upstream
Usually they do, but it can take longer, especially for the older supported release (currently F37). Looking at https://packages.fedoraproject.org/pkgs/qt5-qtbase/qt5-qtbase/fedora-38-updates.html and https://packages.fedoraproject.org/pkgs/qt5-qtbase/qt5-qtbase/fedora-37-updates.html the maintainer was applying security fixes from a month ago but that was also before the EOL, so it's unclear if they'll continue. (Also skimming those issues, none of them would have affected Dangerzone nor SDW AFAICT.)
This has started biting us back in Dangerzone development since PySide2 stopped being supported on Fedora 39.
Option C: our own fedora packaging
Instead of having the packaging files for fedora generated automatically from a setup.py file, we can make them ourselves. https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/
We can use as the base the spec file generated by setuptools and keep developing it ourselves. It can be generated with python setup.py bdist_rpm --spec-only.
Since we already tackled this part we may now consider want to consider the possibility of shipping PySide6 with our own package. Either that or us maintaining the PySide6 package on fedora.
Option C: our own fedora packaging
SecureDrop stumbled upon a similar issue: https://github.com/freedomofpress/securedrop/pull/6884. There a venv was used to ship newer versions of software that is unsupported by distros.
It seems that shipping PySide6 is the best way forward, so I did a bit of research on what that means, what are the alternatives, and how we can actually do it.
First thing I checked and want to address. I was afraid that providing a PySide6 package might break, if a Python binding points to a system Qt library that gets updated in the meantime. Thankfully, this is not the case. The PyPI version of PySide6 provides a Qt/
directory that surprisingly holds all the Qt libraries. The Python modules point to these libraries, and these libraries do not point to any Qt system libraries.
This is further explained in https://doc.qt.io/qtforpython-6/quickstart.html:
Having Qt installed in your system will not interfere with your PySide6 installation if you do it via pip install, because the Python packages (wheels) include already Qt binaries. Most notably, style plugins from the system won’t have any effect on PySide applications.
I have tested this by removing any Qt system library from my dev environment, and just had PySide6 installed. Spinning up the Dangerzone GUI works! At the end of the day, I guess this is expected, since Dangerzone works on Windows/MacOS as well, which do not have any Qt system libraries.
Note that in our development environment, we install a GUI-related Qt6 library (libqt6gui6
). I thought this was a requirement, but turns out we only use it to bring in the rest of the GUI libraries (xkb, fonts, opengl). Of course, we will need to depend on these libraries directly, if we want to package PySide6.
Ok, so we want to package PySide6. Let's assume that the Python wheel is the best candidate, since it contains the Qt libraries as well. Where do we download it from? How can we verify its contents?
So, we don't seem to have a way to verify Qt sources or binary packages via signatures (ideally), but we can use hashes provided by the maintainer and HTTPS (less ideal).
On the other hand, that's kind of the current situation on Windows/MacOS. We rely on a TOFU model of trust, in broad strokes. That is, we assume that the file hashes we grab from PyPI and store in our Poetry lock file are correct the moment we update our Poetry lock file, and malicious interference can only take place and get detected afterwards.
There are various ways we can bring PySide6 into the user's environment. I've tried to evaluate those:
Vendor PySide6 into the Dangerzone RPM package. Basically, this would require shipping /usr/lib/python3.11/site-packages/{PySide6,shiboken6}
within Dangerzone itself. It's a viable alternative, but this means that any global build flags for the Dangerzone RPM would apply to the PySide6 parts, which is not so simple as we will see next.
Create a virtualenv, install PySide6 in it, and ship it. SecureDrop used dh-virtualenv in the past successfully. In Fedora land, I see the following related projects:
I haven't experimented with them yet mostly due to time constraints, and because I wanted to have the option to exercise a bit more control on the contents on the virtualenv (e.g., not just blindly install the package from PyPI, without hash verification).
Package PySide6 as a separate RPM package. Let's see some tools that can help us in this job:
https://github.com/fedora-python/pyp2rpm: Throws the following error:
Unable to extract package metadata from .whl archive. This might be caused by an old .whl format version. You may ask the upstream to upload fresh wheels created with wheel >= 0.17.0 or to upload an sdist as well to workaround this problem.
https://github.com/openSUSE/py2pack: Probably does not work with binary-only wheels, and throws the following error:
KeyError: 'urls'
https://github.com/jordansissel/fpm: Does not work with binary-only wheels (see this open issue), and throws the following error:
fpm: ERROR: Could not find a version that satisfies the requirement PySide6 (from versions: none)
It seems that packaging PySide6 as a separate RPM gives us most control over the process, so we're going to evaluate this option first.
1. What will happen if Fedora actually packages PySide6 after our package is out?
This depends on which version they package:
In both cases, if another system package has PySide6 as a dependency and does not get the exact same version, this may lead to issues. It's good to make sure that we always work with the latest PySide6 version in any case.
2. How to build the package in a way that we install the necessary PySide6 requirements in the user's system (xkb, fonts, opengl)?
My guess is we have to specify them manually. Probably best to figure out which are missing in a clean Fedora container.
3. The resulting package size is big, roughly 150MiB compressed. Can we make it smaller?
My guess is that this is probably due to the bundled Qt libraries, since the same PySide2 package is ~7 MiB. We could probably remove the libraries that we are not using, but that could break any other program that attempts to import PySide6 (see 1).
4. PySide6's shared libraries are built with the manylinux project. Are they safe to be installed in the user's system?
I've already seen that automatic dependency detection goes haywire once rpmbuild
runs ldd
on the shared libraries. E.g.:
libstdc++.so.6()(64bit)
libstdc++.so.6(CXXABI_1.3)(64bit)
libstdc++.so.6(CXXABI_1.3.5)(64bit)
libstdc++.so.6(CXXABI_1.3.9)(64bit)
libstdc++.so.6(GLIBCXX_3.4)(64bit)
libstdc++.so.6(GLIBCXX_3.4.11)(64bit)
libstdc++.so.6(GLIBCXX_3.4.14)(64bit)
libstdc++.so.6(GLIBCXX_3.4.15)(64bit)
libstdc++.so.6(GLIBCXX_3.4.17)(64bit)
libstdc++.so.6(GLIBCXX_3.4.18)(64bit)
libstdc++.so.6(GLIBCXX_3.4.19)(64bit)
libstdc++.so.6(GLIBCXX_3.4.20)(64bit)
libstdc++.so.6(GLIBCXX_3.4.21)(64bit)
libstdc++.so.6(GLIBCXX_3.4.22)(64bit)
libstdc++.so.6(GLIBCXX_3.4.9)(64bit)
This probably affects tools that do some sort of analysis on these libraries, but Dangerzone does not seem to be affected.
5. Is it worth building the package on our own, based on OpenSUSE's specfile?
I guess that this will lead to a smaller package (because we will depend on the system Qt6 libraries), but this will resurface the concern regarding the divergence of the bindings and the system libraries, which I cannot answer yet confidently.
tl;dr: Building PySide6 from source requires digging into their build files. It's not worth the hassle though, since PySide6 relies on private Qt6 APIs that can change at any moment, and can easily break the user's system in case of Qt6 updates.
We have established that we can package PySide6 from a PyPI package, but this comes with some drawbacks:
So, it would be worth checking if building PySide6 from source will help alleviate these problems.
I've used Fedora 39 as a build platform, and tried to follow the official instructions in Qt's site. I also consulted OpenSUSE's python-pyside6
specfile at times.
After hunting down missing packages and fixing some CMake problems, I'm currently stuck at the following error:
[ 11%] Building C object sources/pyside6/qtexampleicons/CMakeFiles/QtExampleIcons.dir/module.c.o
/pyside-setup/sources/pyside6/qtexampleicons/module.c:4:10: fatal error: Python.h: No such file or directory
4 | #include <Python.h>
| ^~~~~~~~~~
compilation terminated.
I believe this error is CMake-related because:
Python.h
header file exists in /usr/include/python3.12/Python.h
CMake detects that it should include headers from /usr/include/python3.12
:
-- PYTHON_INCLUDE_DIRS: /usr/include/python3.12
-- SHIBOKEN_PYTHON_INCLUDE_DIRS computed to value: '/usr/include/python3.12'
I could dig up further, but at this point I've allocated too much time on this simple task.
While we haven't managed to build PySide6 from source and package it as an RPM, we can download the RPM from OpenSUSE's site, and examine it.
I'd like to draw your attention to these requirements:
[...]
libQt6Core.so.6()(64bit)
libQt6Core.so.6(Qt_6)(64bit)
libQt6Core.so.6(Qt_6.6)(64bit)
libQt6Core.so.6(Qt_6.6.0_PRIVATE_API)(64bit)
[...]
What do these requirements tell us? That the PySide6 package requires a Qt6 library that is compatible with the Qt 6.6 API, as well as the private Qt 6.6 API. This introduces two problems:
qt6-qtbase
has been upgraded from 6.3.1 -> 6.4.1 -> 6.4.2 (1 minor upgrade and 1 patch upgrade)Bottom line is: offering a PySide6 package that relies on system Qt6 libraries is bound to break, and cause system updates to stall. Fixing it requires us to be alert of such changes and publish a new PySide6 package whenever that happens.
Bottom line is: offering a PySide6 package that relies on system Qt6 libraries is bound to break, and cause system updates to stall. Fixing it requires us to be alert of such changes and publish a new PySide6 package whenever that happens.
Doesn't sound sustainable at all.
It seems that Debian has backported some patches from PySide6 in their own PySide2 package, in order to make it work with Python 3.12. We have to wait and see if Fedora picks up on those patches and reinstates PySide2.
We haven't tested rebuilding PySide2 in Fedora 39 with these patches, because they appeared after our packaging effort. We don't know if these patches can be cleanly applied to Fedora's PySide2 source, nor what happens if Qt5 gets updated after we publish a patched PySide2 (probably nothing because PySide2 development is stalled).
In any case, we can revisit this solution if our wheel approach is less maintainable, and maybe offer it upstream as well.
Quick update, Fedora Rawhide now provides python3-pyside6
6.7.0. We expect this to appear in the rest of the stable versions soon.
Another important update: Fedora 40 now provides python3-pyside6
6.7.2 through its updates channel: https://packages.fedoraproject.org/pkgs/python-pyside6/python3-pyside6/fedora-40-updates.html 🎉
The only version that does not have python3-pyside6
is Fedora 39, which will be EOL on November 12th, 2024
Mirroring https://github.com/freedomofpress/securedrop-client/issues/1562:
We have a PR that already does this change but it is blocked on distros lacking Pyside6 (Qt6) packages.