Closed solofoA45 closed 6 days ago
Here are some starting points and initial thoughts.
This should tie into the Usage : Security documentation since some issues can be mitigated or bypassed using a tight configuration.
The closest thing we have to an overview of the dependencies is here: https://github.com/Xpra-org/xpra/blob/master/docs/Build/Dependencies.md (and includes some pretty diagrams)
The MacOS builds are by far the easiest to track since we define and build every single library ourselves.
The list can be found by checking out the build repository at the date matching that of the release:
https://github.com/Xpra-org/gtk-osx-build
There are a few extra packages in there that are only used on MacOS for packaging (ie: xar
, etc), those are not shipped.
The full MSYS2
package list on the build system at time of writing has 179 packages:
$ pacman -Qe | wc -l
179
There are also python dependencies which do not have corresponding MINGW packages and are installed via pip: https://github.com/Xpra-org/xpra/blob/15bce9d2ec162ce4984c86aa4279ea2218445194/packaging/MSWindows/SETUP.sh#L38-L42 Then there are also some manual steps: https://github.com/Xpra-org/xpra/blob/15bce9d2ec162ce4984c86aa4279ea2218445194/packaging/MSWindows/SETUP.sh#L46 for:
There can be many reasons why some packages are not updated as regularly as others:
setuptools
, zeroconf
pip
): pip3 list --outdated
pyu2f
): should we remove them?libyuv
: google repository with no actual releases - who really knows what's going on there?Some issues are magnified in 3.1:
libwebp
and libvpx
, RHEL 7 does not):
libjpeg-turbo
: probably has CVEs against itlz4
+ python-lz4
: python wrapper abandoned python2.. v2.2.1 is the last version that can be built for python2python-pillow
stuck on 6.2.2On the whole, I don't think that it is reasonable to expect the 3.1.x to have the same level of maintenance as current versions. CUDA
We should probably split the dependencies into categories - this is probably too many:
gcc
though libgcc
might be?, bash
, etc)libtiff
CVE was for an executable we do not include, not the libtiff
library we do ship)--video-decoders=none
)GTK
, cairo
, etc)lz4
, rencodeplus
, python
interpreter.
etc.The MS Windows dependencies can be recorded in xpra/build_info.py
.
We can get most of the packages from pacman
- and perhaps trim most of the build time dependencies as those aren't very relevant?
The python pip
dependencies are going to be a pain.
This seems relevant: Understanding the NSA’s latest guidance on managing OSS and SBOMs
Both MacOS and MS Windows builds will now record the libraries and python modules present on the build system when the installer is generated. This will include dependencies we don't really care about: build tools, libraries we don't bundle, etc. But it is safer to record too much than too little, and filtering was hard and would also have required constant fine tuning.
The feature for the html5 client is now tracked here: https://github.com/Xpra-org/xpra-html5/issues/277
Next up:
pip3 freeze
output - what is the checksum for?https://pypi.org/pypi/$PROJECT/json
Another tricky one to handle is pdfium-binaries releases - this release page does show a line that says something like "This version was built with branch chromium/6337 of PDFium".
The archive containing the DLL we need also contains a VERSION
file:
$ cat pdfium/VERSION
MAJOR=124
MINOR=0
BUILD=6337
PATCH=0
The easiest way might be to create a "fake" pacman PKGBUILD
for it.
The new script that I am working on would flag:
liblzma-5.dll
as belonging to /mingw64/bin/liblzma-5.dll is owned by mingw-w64-x86_64-xz 5.6.1-2
.
(current DLL should be fine but it was previously owned by the vulnerable version of xz-utils: 5.6.1-1
)
Fixed in: https://github.com/msys2/MSYS2-packages/commit/eb7abbb627d1ccfc5a3a6ca31b98150ee733c366 First vulnerable version was added in: https://github.com/msys2/MSYS2-packages/commit/d153a0914cd45d74acae8a3ade74d0b0d8df5ec6 So any builds between 2024-02-25 and today are shipping the vulnerable library.
On the plus side, the exploit seems to target a specific function in openssh - with glibc, and we don't use openssh by default, and no glibc, and not as a server... So no need to panic.
Good links on the subject:
Forgot another packages missing from MSYS2 that we should contribute upstream:
https://github.com/Xpra-org/xpra/blob/79d8e18dc7544d019bef79374b0bdd51c6d42723/packaging/MSWindows/SETUP.sh#L70
Trivial to install: meson build && ninja install
.
cyclonedx-python-lib: This Python package provides data models, validators and more, to help you create/render/read CycloneDX documents.
As of 5d21b1b37385223e370cec280b97deba351a6fd1, a default build will add a lot of metadata in xpra/build_info.py
:
build = {'user': 'Windows 10 Test', 'on': 'Win10-Build-VM', 'date': '2024-10-20', 'time': '23:10', 'machine': 'AMD64', 'cpu': 'Intel64 Family 6 Model 94 Stepping 3, GenuineIntel', 'bit': '64bit', 'os': 'Microsoft Windows 10 Pro', 'type': '', 'python': '3.11.10', 'cython': '3.0.11', 'compiler': 'gcc (Rev1, Built by MSYS2 project) 14.2.0', 'nvcc': '', 'linker': 'GNU ld (GNU Binutils) 2.43.1'}
libs = {'apr': '1.7.5-1', 'apr-util': '1.6.3-2', 'asciidoc': '10.2.1-2', 'autoconf-wrapper': '20240607-1', ...
(only the first 4 packages are show here since there are literally hundreds of them, as we record all the packages installed on the build system - whether they are used or not, which is impossible to ascertain)
Then also:
BUILD_OPTIONS={'build': True, 'clean': True, 'cuda': False, 'desktop_logon': False, 'docs': False, 'fixups': True, 'html5': False, 'install': True, 'installer': False, 'light': True, 'manual': False, 'msi': False, 'numpy': False, 'openssh': False, 'openssl': False, 'paexec': False, 'putty': False, 'run': True, 'sbom': True, 'service': False, 'sign': True, 'tests': False, 'verbose': False, 'verpatch': True, 'zip': False, 'zip_modules': True}
dll_sbom={'lib/gdk-pixbuf-2.0/2.10.0/loaders/io-wmf.dll': ('mingw-w64-x86_64-libwmf', '0.2.13-1'), 'lib/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-ani.dll': ('mingw-w64-x86_64-gdk-pixbuf2', '2.42.12-3'), ...
Again, far too many to show here.
This one is smarter and only includes packages that we actually ship.
Only a few DLLs aren't identified yet, which we need to submit upstream: pdfium
, libspng
, win32com
.
This data can be converted to a JSON SBOM using cyclonedx - and maybe this should be a separate build arctifact?
@solofoA45 good news is that the scripting part is mostly done and working well, the bits that still need doing stick out from the script output:
$ python3.11 ./packaging/MSWindows/BUILD.py
* 21:42:25 Collecting version information
Python 3.11.10 (main, Sep 10 2024, 13:02:13) [GCC 14.2.0 64 bit (AMD64)]
Xpra-Light 6.3-r36810
using 4 cpus
* 21:42:31 locating `signtool`
* 21:42:31 Cleaning output directories and generated files
* 21:42:34 Building Cython modules
* 21:43:27 Generating installation directory
* 21:44:43 Fixups: paths, etc
* 21:44:44 Fixup GStreamer
* 21:44:44 Fixup DLLs
* 21:45:18 Deleting unnecessary DLLs
* 21:45:18 Removing unnecessary Python modules
Warning: glob 'lib/backports' did not match any files!
Warning: glob 'lib/yaml' did not match any files!
* 21:45:18 Removing unnecessary files
* 21:45:21 Removing unnecessary PIL plugins
* 21:45:21 Removing empty directories
* 21:45:22 cuda: False
Warning: glob 'lib/pycuda*' did not match any files!
Warning: glob 'lib/curand*' did not match any files!
* 21:45:22 numpy: False
Warning: 'dist/lib/numpy' does not exist
Warning: glob 'lib/libopenblas*' did not match any files!
Warning: glob 'lib/libgfortran*' did not match any files!
Warning: glob 'lib/libquadmath*' did not match any files!
* 21:45:22 zipping up some Python modules
Warning: glob 'lib/unittest' did not match any files!
Warning: glob 'lib/pynvml' did not match any files!
Warning: glob 'lib/ldap' did not match any files!
Warning: glob 'lib/ldap3' did not match any files!
* 21:45:28 Deleting unnecessary `share/` files
* 21:45:28 Removing empty icon directories
* 21:45:28 Adding EXE manifests
* 21:45:28 Generating gdk pixbuf loaders cache
* 21:45:28 Generating icons and theme cache
* 21:45:28 Recording SBOM
Warning: no package data found for 'lib/libspng-0.dll'
Warning: no package data found for 'pdfium.dll'
Warning: no package data found for 'C:/msys64/mingw64/lib/python3.11/site-packages/aioquic'
Warning: no package data found for 'C:/msys64/mingw64/lib/python3.11/site-packages/pylsqpack'
181 dist
* 21:48:25 Creating ZIP file:
72 Xpra-Light-x86_64_6.3-r36810.zip
* 21:48:44 Creating the installer using InnoSetup
51 Xpra-Light-x86_64_Setup_6.3-r36810.exe
* 21:49:28 Signing EXE
* 21:49:29 Running the new installer
And I think that this should be relatively straightforward to backport this code to older branches:
Here's the beginning of the raw SBOM data for a full build (this is parseable as python source):
# 307 SBOM path entries:
{
'lib/gdk-pixbuf-2.0/2.10.0/loaders/io-wmf.dll': \
(19975, '39f9911e319826a956a6a858dca4ad5d406e580db8eff3d760245b4b26435540', 'libwmf', '0.2.13-1'),
'lib/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-ani.dll': \
(26756, '9ed6fb05b84163c24ee2ecf3036b3a23a0f88e2bd29f45bbf1e97cfc079e7216', 'gdk-pixbuf2', '2.42.12-3'),
'lib/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-avif.dll': \
(25362, '6f99f0d702fc085fadf9d2a9033f0c8349c0b2d73019e5e32a4ffcaa50b3a0bf', 'libavif', '1.1.1-1'),
'lib/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-bmp.dll': \
(27093, 'bfbba366556397a66c82844a2fea656baa303ab501c112b1139143623a392d13', 'gdk-pixbuf2', '2.42.12-3'),
...
# 134 'mingw-w64-x86_64-' packages:
('aom', 'atk', 'boost', 'brotli', 'bzip2', 'cairo', 'crypto++', 'curl', 'cyrus-sasl', 'dav1d', 'dbus', 'dbus-glib', 'double-conversion', 'expat', 'faac', 'faad2', 'flac', 'fontconfig', 'freetype', 'fribidi', 'gcc-libgfortran', 'gcc-libs', 'gdk-pixbuf2', 'gettext-runtime', 'glib2', 'gmp', 'gnutls', 'gobject-introspection-runtime', 'graphite2', 'gss', 'gst-plugins-bad', 'gst-plugins-bad-libs', 'gst-plugins-base', 'gst-plugins-good', 'gst-plugins-ugly', 'gstreamer', 'gtk3', 'harfbuzz', 'highway', 'icu', 'jbigkit', 'lame', 'lcms2', 'lerc', 'libavif', 'libb2', 'libcroco', 'libdatrie', 'libdeflate', 'libepoxy', 'libffi', 'libgcrypt', 'libgpg-error', 'libheif', 'libiconv', 'libidn', 'libidn2', 'libimagequant', 'libjpeg-turbo', 'libjxl', 'libogg', 'libpng', 'libpsl', 'libraqm', 'librsvg', 'libsodium', 'libsoup', 'libssh2', 'libtasn1', 'libthai', 'libtiff', 'libunistring', 'libvorbis', 'libvpx', 'libwebp', 'libwinpthread-git', 'libwmf', 'libx264', 'libxml2', 'libyaml', 'libyuv', 'lz4', 'md4c', 'mpdecimal', 'mpg123', 'nettle', 'nghttp2', 'nghttp3', 'openblas', 'openh264', 'openjpeg2', 'openjph', 'openldap', 'openssl', 'opus', 'orc', 'p11-kit', 'pango', 'pcre2', 'pixman', 'python', 'python-bcrypt', 'python-cairo', 'python-cffi', 'python-charset-normalizer', 'python-cryptography', 'python-importlib-metadata', 'python-importlib_resources', 'python-ldap', 'python-netifaces', 'python-numpy', 'python-oauthlib', 'python-pillow', 'python-py-cpuinfo', 'python-pycryptodome', 'python-pycryptodomex', 'python-pynacl', 'python-pyopengl-accelerate', 'python-pyparsing', 'python-pywin32', 'python-watchdog', 'python-winkerberos', 'python-zeroconf', 'qrencode', 'qt6-base', 'rav1e', 'shishi', 'sqlite3', 'svt-av1', 'wavpack', 'xxhash', 'xz', 'zlib', 'zstd')
Some of the missing MSYS2 packages have been submitted upstream:
This one I will probably keep in my own repo as I don't think that will be acceptable for upstream:
Which only leaves CUDA
.
It makes sense to special case it since we already have a add_cuda
function and cuda
switch.
They ship their own metadata file which we'll need to parse:
$ cat /z/version.json
{
"cuda" : {
"name" : "CUDA SDK",
"version" : "12.6.0"
},
...
The MSYS2 packages have been merged upstream and the CUDA information is also recorded now using custom code: 8926b62f435548f05cb78df061c23bf54f5b67b3.
Unfortunately, the call order was wrong and zipping the python modules was hiding some packages from the SBOM code. So now after re-ordering these two steps, we see more missing modules:
* 17:30:35 Recording SBOM
Warning: no package data found for 'C:/msys64/mingw64/lib/python3.11/site-packages/browser_cookie3'
Warning: unknown source for filename 'browser_cookie3', tried [
'C:/msys64/mingw64/lib/python3.11',
'C:/msys64/mingw64/lib/python3.11/lib-dynload',
'C:/msys64/mingw64/lib/python3.11/site-packages',
'C:/msys64/mingw64/lib/python3.11/site-packages/win32',
'C:/msys64/mingw64/lib/python3.11/site-packages/pywin32_system32'
]
Warning: unknown source for filename 'pkcs11', tried [...
Warning: no package data found for 'C:/msys64/mingw64/lib/python3.11/site-packages/pyaes'
Warning: unknown source for filename 'pyaes', tried [...
Warning: no package data found for 'C:/msys64/mingw64/lib/python3.11/site-packages/pycuda'
Warning: unknown source for filename 'pycuda', tried [...
Warning: no package data found for 'C:/msys64/mingw64/lib/python3.11/site-packages/pyvda'
Warning: unknown source for filename 'pyvda', tried [...
* 17:35:16 zipping up some Python modules
Links:
browser_cookie3
- used for websocket cookies: https://github.com/Xpra-org/xpra/issues/2874#issuecomment-765550057 - this small library is what brought extra dependencies: edc92fd80c7693645e5f9cead81c86a2192c5ec5 (python-cryptodome
and pycryptodomex
)pkcs11
- which upstream is this? https://pypi.org/project/pkcs11/ or https://pypi.org/project/python-pkcs11/#history ? can be packaged either way?pyaes
added in 6cca4c44c760446d74045dd8cc1831e7e701f834, not clear which library needs it, can be packagedpycuda
: known issue, will make a PKGBUILD
for itpyvda
workspace support: #1442The commits above adds some data for macos builds.
This data is much less useful because jhbuild
doesn't expose any package file tracking so we can resolve specific files in our bundle to the package that installed them.
At least the packages
data lists which version of each package was installed.
(and we skip the ones only used for packaging)
This will do for now. The remaining warnings about unpackaged python dependencies will serve as a reminder that this can be improved further.
I would have liked to also generate the SBOM in a more standardized format, perhaps using cyclonedx-python but the lack of tools made this too difficult.
Example usage:
xpra_cmd.exe sbom | grep -A 5 xz:
xz:
Name : mingw-w64-x86_64-xz
Version : 5.6.3-1
Description : Library and command line tools for XZ and LZMA compressed files (mingw-w64)
Architecture : any
URL : https://tukaani.org/xz
xpra_cmd.exe sbom | grep `liblz4.dll`
'lib/liblz4.dll' : mingw-w64-x86_64-lz4 1.10.0-1
We now export a fairly comprehensive sbom using cyclonedx. (it must be installed separately in a Python-for-MSWindows installation since cyclonedx-py is not compatible with MSYS2)
The problem is that this now flags some missing package information via dependencies:
Warning: 'mingw-w64-x86_64-lzo2' not found, dependency of cairo
Warning: 'mingw-w64-x86_64-c-ares' not found, dependency of curl
Warning: 'mingw-w64-x86_64-ca-certificates' not found, dependency of curl
Warning: 'mingw-w64-x86_64-libwinpthread' not found, dependency of gcc-libs
Warning: 'mingw-w64-x86_64-nettle>=3.6' not found, dependency of gnutls
Warning: 'mingw-w64-x86_64-p11-kit>=0.23.1' not found, dependency of gnutls
Warning: 'mingw-w64-x86_64-adwaita-icon-theme' not found, dependency of gtk3
Warning: 'mingw-w64-x86_64-json-glib' not found, dependency of gtk3
Warning: 'mingw-w64-x86_64-shared-mime-info' not found, dependency of gtk3
Warning: 'mingw-w64-x86_64-gtk-update-icon-cache' not found, dependency of gtk3
Warning: 'mingw-w64-x86_64-aom' not found, dependency of libavif
Warning: 'mingw-w64-x86_64-dav1d' not found, dependency of libavif
Warning: 'mingw-w64-x86_64-rav1e' not found, dependency of libavif
Warning: 'mingw-w64-x86_64-svt-av1' not found, dependency of libavif
Warning: 'mingw-w64-x86_64-libsharpyuv' not found, dependency of libavif
Warning: 'mingw-w64-x86_64-aom' not found, dependency of libheif
Warning: 'mingw-w64-x86_64-dav1d' not found, dependency of libheif
Warning: 'mingw-w64-x86_64-kvazaar' not found, dependency of libheif
Warning: 'mingw-w64-x86_64-libde265' not found, dependency of libheif
Warning: 'mingw-w64-x86_64-libsharpyuv' not found, dependency of libheif
Warning: 'mingw-w64-x86_64-rav1e' not found, dependency of libheif
Warning: 'mingw-w64-x86_64-svt-av1' not found, dependency of libheif
Warning: 'mingw-w64-x86_64-x265' not found, dependency of libheif
Warning: 'mingw-w64-x86_64-giflib' not found, dependency of libjxl
Warning: 'mingw-w64-x86_64-libwinpthread' not found, dependency of libjxl
Warning: 'mingw-w64-x86_64-openexr' not found, dependency of libjxl
Warning: 'mingw-w64-x86_64-glib-networking' not found, dependency of libsoup
Warning: 'mingw-w64-x86_64-sqlite3' not found, dependency of libsoup
Warning: 'mingw-w64-x86_64-giflib' not found, dependency of libwebp
Warning: 'mingw-w64-x86_64-libjpeg' not found, dependency of libwmf
Warning: 'mingw-w64-x86_64-libjpeg' not found, dependency of libyuv
Warning: 'mingw-w64-x86_64-wineditline' not found, dependency of pcre2
Warning: 'mingw-w64-x86_64-mpdecimal' not found, dependency of python
Warning: 'mingw-w64-x86_64-ncurses' not found, dependency of python
Warning: 'mingw-w64-x86_64-sqlite3' not found, dependency of python
Warning: 'mingw-w64-x86_64-tzdata' not found, dependency of python
Warning: 'mingw-w64-x86_64-python-decorator' not found, dependency of python-gssapi
Warning: 'mingw-w64-x86_64-gss' not found, dependency of python-gssapi
Warning: 'mingw-w64-x86_64-python-jaraco.classes' not found, dependency of python-keyring
Warning: 'mingw-w64-x86_64-python-jaraco.context' not found, dependency of python-keyring
Warning: 'mingw-w64-x86_64-python-jaraco.functools' not found, dependency of python-keyring
Warning: 'mingw-w64-x86_64-python-pywin32-ctypes' not found, dependency of python-keyring
Warning: 'mingw-w64-x86_64-python-six' not found, dependency of python-paramiko
Warning: 'mingw-w64-x86_64-libjpeg' not found, dependency of python-pillow
Warning: 'mingw-w64-x86_64-python-six' not found, dependency of python-pyu2f
We already skip most of the ones we don't need:
https://github.com/Xpra-org/xpra/blob/0dfe55ed50404565e53d4fab4a47ea4d5295c4ec/packaging/MSWindows/cyclonedx_sbom.py#L73-L77
More should be added to this list.
Then others should be recorded properly (pyopengl
, libjpeg
, etc).
Others only provide data files, but we may still want to track these..
Remaining warnings for light builds during the cyclonedx export step that still need to be investigated:
Warning: 'mingw-w64-x86_64-c-ares' not found, dependency of curl
Warning: 'mingw-w64-x86_64-ca-certificates' not found, dependency of curl
Warning: 'mingw-w64-x86_64-nettle>=3.6' not found, dependency of gnutls
Warning: 'mingw-w64-x86_64-p11-kit>=0.23.1' not found, dependency of gnutls
Warning: 'mingw-w64-x86_64-glib-networking' not found, dependency of libsoup
Warning: 'mingw-w64-x86_64-sqlite3' not found, dependency of libsoup
Warning: 'mingw-w64-x86_64-libsystre' not found, dependency of ncurses
Warning: 'mingw-w64-x86_64-wineditline' not found, dependency of pcre2
Warning: 'mingw-w64-x86_64-mpdecimal' not found, dependency of python
Warning: 'mingw-w64-x86_64-sqlite3' not found, dependency of python
Warning: 'mingw-w64-x86_64-python-decorator' not found, dependency of python-gssapi
Warning: 'mingw-w64-x86_64-gss' not found, dependency of python-gssapi
With the latest dependency updates and tweaks, only a few corner cases remain for the "light" builds:
pyvda
- needs a PKGBUILD
c-ares
, nettle
, p11-kit
, wineditline
To make things even more "interesting", MSYS2 just switched from Python 3.11 to Python 3.12.
So the 6.3-r36873
beta builds are still using Python 3.11.10, but any later builds will be based on Python 3.12.x.
Good news is that the upgrade just worked. All that was needed afterwards - as expected:
pip3 install pyvda
- created a PR: https://github.com/msys2/MINGW-packages/pull/22482pip3 install browser_cookie3
- mehThe export-sbom
step of the new build script can be executed separately:
packaging/MSWindows/BUILD.py export-sbom
It requires:
C:\Program Files\Python312
cyclonedx
to it: Python.exe -m pip install cyclonedx-bom
There is now a link from the download page to the corresponding sbom files. (only for the beta releases for now, others will be added later)
Caveats:
pdfium
, pycuda
, browser_cookie3
, etc... should be packaged betterClosing.
With the latest fixes, the only missing piece is a PKGBUILD
for pycuda
.
For security and compliance concerns, it would be good to have a list of dependencies for example to assess which security vulnerabilities affect Xpra: https://en.wikipedia.org/wiki/Software_supply_chain
While this is rather clear for linux (RPM) packages, this is less clear for windows packages and HTML5 client packages.
Is there already a way to get these informations?