Debian 12 Bookworm - bump dependencies to uptodate versions of cx_Freeze, zipp and importlib_metadata

[as365n4]

MLAT client stopped working on Debian 12 Bookworm due to outdated cx_Freeze dependency.

This is becomming an issue soon as Debian 12 has entered Toolchain freeze on January 19th and is due for general release this year. https://lists.debian.org/debian-devel-announce/2023/01/msg00004.html

https://github.com/marcelotduarte/cx_Freeze/archive/refs/tags/6.13.1.tar.gz c302e15bc9c488704f0b79b3896101ae6fc81b405ce489c29b15937111bc18f3

https://github.com/jaraco/zipp/archive/refs/tags/v3.11.0.tar.gz d439d06f10385871258c29fb40839677a99b13720c2572b2673c14e73f2a550d

https://github.com/python/importlib_metadata/archive/refs/tags/v6.0.0.tar.gz 9e6fafdbf0601a5825c53a842f7bb6928c8e8eb6c46a228199ca02396cf6007a

[as365n4]

output from systemctl status piaware.service: piaware.service

For reference, this is a contiuation of https://discussions.flightaware.com/t/debian-bookworm-kills-piaware-service-minutes-after-starting/86018

[mutability]

Do you have a PR for this?

[as365n4]

No, I don't have a Pull request for this.

I am building piaware for arm64 platform and whilst the script builds without an error, as soon as piaware is started it complains about MLAT Client not working and refers to broken cx-Freeze as shown in systemctl output in previous comment.

Others building for amd64 platform and having exactly the same issue, as referenced in the FlightAware discussion.

Unfortunately, I do not have the knowlegde to provide a fix for this.

[mutability]

Okay. Can you verify that building with the newer cxfreeze fixes the problem? I do not have a bookworm system to verify against.

[as365n4]

This is not as easy as it seems to be...

zipp-3.12.0 does not have setup.py as specified in line 72 of the rules file importlib_metadata-6.0.0 does not have setup.py as specified in line 73 of the rules file

keeping zipp-0.5.0 and importlib_metadat-4.3.1 and just using cx_Freeze-6.14.2 results in:

Processing dependencies for cx-Freeze==6.14.2 Searching for patchelf>=0.14 Reading https://pypi.org/simple/patchelf/ Download error on https://pypi.org/simple/patchelf/: [Errno 111] Connection refused -- Some packages may not be found! Couldn't find index page for 'patchelf' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading https://pypi.org/simple/ Download error on https://pypi.org/simple/: [Errno 111] Connection refused -- Some packages may not be found! No local packages or working download links found for patchelf>=0.14 error: Could not find suitable distribution for Requirement.parse('patchelf>=0.14') make[1]: [debian/rules:74: build_cxfreeze] Error 1 make[1]: Leaving directory '//piaware_builder/package-bookworm' make: *** [debian/rules:104: build] Error 2 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

but patchelf 0.14 is installed already

ADSB-Receiver-test:~/piaware_builder/package-bookworm$ apt search patchelf Sorting... Done Full Text Search... Done patchelf/testing,now 0.14.3-1+b1 arm64 [installed] modify properties of ELF executables

pypi.org has patchelf 0.17.2.0 so installed this via pip as well, but still no success. Reinstalled with sudo pip and the build was successfully, no errors reported.

But there are warnings:

Finished processing dependencies for cx-Freeze==6.14.2 cd mlat-client && /home/user/piaware_builder/package-bookworm/debian/venv/bin/python setup.py install /usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( /usr/lib/python3/dist-packages/setuptools/command/easy_install.py:146: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools. warnings.warn(

And after installing piaware and starting the service via systemctl it reports that the mlat-client is still not working.

● piaware.service - FlightAware ADS-B uploader Loaded: loaded (/lib/systemd/system/piaware.service; enabled; preset: enabled) Active: active (running) since Sat 2023-02-04 12:44:08 GMT; 3min 7s ago Docs: https://flightaware.com/adsb/piaware/ Main PID: 23673 (piaware) Tasks: 3 (limit: 1045) Memory: 14.4M CPU: 19.761s CGroup: /system.slice/piaware.service ├─23673 /usr/bin/piaware -p /run/piaware/piaware.pid -plainlog -statusfile /run/piaware/status.json └─23750 /usr/lib/piaware/helpers/faup1090 --net-bo-ipaddr localhost --net-bo-port 30005 --stdout --lat 50.791 --lon 0.326

Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): module_init.run(name + "main") Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): File "/home/user/piaware_builder/package-bookworm/debian/venv/lib/python3.11/site-packages/cx_Freeze-6.14.2-py3.11-linux-aarch64.egg/cx_Freeze/initscripts/console.py", line 18, in run Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): exec(code, module_main.dict) # pylint: disable=exec-used Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): File "/home/user/piaware_builder/package-bookworm/debian/venv/bin/fa-mlat-client", line 4, in Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): import('pkg_resources').run_script('MlatClient==0.2.12', 'fa-mlat-client') Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): ModuleNotFoundError: No module named 'pkg_resources' Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: got EOF from multilateration client Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: fa-mlat-client exited with EXIT 255

So something else is broken, any ideas?

[mutability]

As you've discovered, getting this right is not as easy as it looks. A lot of the hoops that piaware_builder has to jump through are to do with getting cxfreeze working with the older versions of Python that Debian usually ships. Maybe bookworm is new enough that a recent cxfreeze can be installed directly with pip without needing to do the manual dependency dance.

That said, given that you still get the same error, it's presumably not the cxfreeze version that's causing problems at all. I have not looked into this at all yet, but maybe you need to force cxfreeze to include pkg_resources in the frozen binary.

[as365n4]

I wouldn't blame Debian on this one, because Bookworm ships with Python 3.11 which is the latest release of Python.

Which pkg_resources need to be included? pypi.org gives over 1300 hits when searched for pkg_resources, but cx_Freeze seems not to have a resource/dev package.

[mutability]

Sure, so if bookworm has a recent python you probably don't need to jump through hoops to get recent packages installed there and much of the work that piaware_builder does with mangling specific versions of packages to get them to work with the older python toolchain is unnecessary.

Re pkg_resources I mean tell cxfreeze to include the version that's already present in the venv, in the frozen binary, by using e.g. --includes or --packages on the cxfreeze command line. See https://cx-freeze.readthedocs.io/en/latest/script.html

(The issue here is that it looks like the version of setuptools et al that bookworm has, is synthesizing a fa-mlat-client script that uses pkg_resources, but pkg_resources is not being included by cxfreeze in the frozen binary)

[as365n4]

I've amended line 76/77 of the rules script to:

build_mlat-client: cd mlat-client && $(VENV)/bin/python setup.py install --includes=pkg_resources

And now gcc exits with a fatal error:

Finished processing dependencies for importlib-metadata==4.3.1 cd cx_Freeze-6.8.3 && /home/user/piaware_builder/package-bullseye/debian/venv/bin/python setup.py install running install /usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( /usr/lib/python3/dist-packages/setuptools/command/easy_install.py:146: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools. warnings.warn( running bdist_egg running egg_info writing cx_Freeze.egg-info/PKG-INFO writing dependency_links to cx_Freeze.egg-info/dependency_links.txt writing entry points to cx_Freeze.egg-info/entry_points.txt writing requirements to cx_Freeze.egg-info/requires.txt writing top-level names to cx_Freeze.egg-info/top_level.txt reading manifest file 'cx_Freeze.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' adding license file 'doc/src/license.rst' writing manifest file 'cx_Freeze.egg-info/SOURCES.txt' installing library code to build/bdist.linux-aarch64/egg running install_lib running build_py running build_ext running install_include aarch64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -O2 -ffile-prefix-map=/home/user/piaware_builder/package-bullseye=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/user/piaware_builder/package-bullseye/debian/venv/include -I/usr/include/python3.11 -c source/bases/Console.c -o build/temp.linux-aarch64-cpython-311/source/bases/Console.o In file included from source/bases/Console.c:45: source/bases/Common.c:7:10: fatal error: eval.h: No such file or directory 7 | #include | ^~~~ compilation terminated. error: command '/usr/bin/aarch64-linux-gnu-gcc' failed with exit code 1 make[1]: [debian/rules:74: build_cxfreeze] Error 1 make[1]: Leaving directory '/home/user/piaware_builder/package-bullseye' make: [debian/rules:104: build] Error 2 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

I think this needs more work to get piaware_builder ready for Bookworm. (I've used the Bullseye build script for simplicity this time)

Edit: Even with --includes=pkg_resources removed from the rules script the build fails with the above error.

[mutability]

Does /usr/include/python3.11/eval.h exist?

[mutability]

Also --includes needs to go on the invocation of cxfreeze, not the mlat-client install step.

[as365n4]

content of /usr/include/python3.11

user@ADSB-Receiver-test:~/piaware_builder/package-bullseye/debian$ ls /usr/include/python3.11 abstract.h descrobject.h graminit.h moduleobject.h pydtrace.h pystate.h structseq.h bltinmodule.h dictobject.h import.h object.h pyerrors.h pystrcmp.h sysmodule.h boolobject.h dynamic_annotations.h internal objimpl.h pyexpat.h pystrtod.h token.h bytearrayobject.h enumobject.h intrcheck.h opcode.h pyframe.h Python.h traceback.h bytesobject.h errcode.h iterobject.h osdefs.h pyhash.h pythonrun.h tracemalloc.h ceval.h exports.h listobject.h osmodule.h pylifecycle.h pythread.h tupleobject.h codecs.h fileobject.h longobject.h patchlevel.h pymacconfig.h pytypedefs.h typeslots.h compile.h fileutils.h marshal.h pybuffer.h pymacro.h rangeobject.h unicodeobject.h complexobject.h floatobject.h memoryobject.h pycapsule.h pymath.h setobject.h warnings.h cpython frameobject.h methodobject.h pyconfig.h pymem.h sliceobject.h weakrefobject.h datetime.h genericaliasobject.h modsupport.h py_curses.h pyport.h structmember.h

[as365n4]

Ok, I've put --includes=pkg_resources in line 74 of the rules script and the build fails with:

Finished processing dependencies for importlib-metadata==4.3.1 cd cx_Freeze-6.8.3 && /home/user/piaware_builder/package-bullseye/debian/venv/bin/python --includes=pkg_resources setup.py install unknown option --includes=pkg_resources usage: /home/user/piaware_builder/package-bullseye/debian/venv/bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ... Try `python -h' for more information. make[1]: [debian/rules:74: build_cxfreeze] Error 2 make[1]: Leaving directory '/home/user/piaware_builder/package-bullseye' make: [debian/rules:104: build] Error 2 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

Edit: even when changed to cd cx_Freeze-6.8.3 && /home/user/piaware_builder/package-bullseye/debian/venv/bin/python setup.py install --includes=pkg_resources or cd cx_Freeze-6.8.3 && /home/user/piaware_builder/package-bullseye/debian/venv/bin/python setup.py --includes=pkg_resources install results in the above error. Sorry, but I do not have the knowledge to code.

[mutability]

Well, 6.8.3 isn't going to work with Python 3.11 anyway? See https://cx-freeze.readthedocs.io/en/latest/versions.html You need 6.14.

Anyway, I can't really continue to remotely debug this one. If you do manage to get it working, I'll take a PR. Otherwise I think you'll need to wait until we officially look at bookworm support (which is more likely to be around the time that the Pi Foundation switches to basing their stuff on bookworm)

[WhoAmI0501]

I would like to draw your attention to my just opened pull request #24, in which context I am still looking for some more testers.

[iwikus]

I would like to draw your attention to my just opened pull request #24, in which context I am still looking for some more testers.

I have successfully used your repo to build and install on bookworm. Thank you.

ii piaware 8.2 amd64 Open Aviation Data Exchange Protocol for ADS-B FlightFeeders

[mutability]

9.0 has bookworm support now, so I'm going to close this.

[iwikus]

9.0 has bookworm support now, so I'm going to close this.

hm, but sensible-build.sh have no bookworm option

root@rx:~# git clone https://github.com/flightaware/piaware_builder
Cloning into 'piaware_builder'...
remote: Enumerating objects: 1252, done.
remote: Counting objects: 100% (312/312), done.
remote: Compressing objects: 100% (133/133), done.
remote: Total 1252 (delta 231), reused 255 (delta 179), pack-reused 940
Receiving objects: 100% (1252/1252), 223.59 KiB | 3.49 MiB/s, done.
Resolving deltas: 100% (841/841), done.
root@rx:~# cd piaware_builder
root@rx:~/piaware_builder# ls -la
total 104
drwxr-xr-x  8 root root  4096 Dec  8 16:06 .
drwx------ 13 root root  4096 Dec  8 16:06 ..
drwxr-xr-x  3 root root  4096 Dec  8 16:06 bookworm
drwxr-xr-x  3 root root  4096 Dec  8 16:06 bullseye
drwxr-xr-x  3 root root  4096 Dec  8 16:06 buster
-rw-r--r--  1 root root 38138 Dec  8 16:06 changelog
drwxr-xr-x  3 root root  4096 Dec  8 16:06 common
drwxr-xr-x  8 root root  4096 Dec  8 16:06 .git
-rw-r--r--  1 root root   120 Dec  8 16:06 .gitignore
-rw-r--r--  1 root root   883 Dec  8 16:06 importlib_metadata-2.1.2.patch
-rw-r--r--  1 root root   259 Dec  8 16:06 importlib_metadata-4.3.1.patch
-rw-r--r--  1 root root  2430 Dec  8 16:06 Jenkinsfile
-rw-r--r--  1 root root  2966 Dec  8 16:06 README.md
-rwxr-xr-x  1 root root  7190 Dec  8 16:06 sensible-build.sh
drwxr-xr-x  3 root root  4096 Dec  8 16:06 stretch
-rw-r--r--  1 root root   901 Dec  8 16:06 zipp-0.5.0.patch
root@rx:~/piaware_builder# ./s
sensible-build.sh  stretch/
root@rx:~/piaware_builder# ./sensible-build.sh
syntax: ./sensible-build.sh <stretch|buster|bullseye|xenial|bionic|disco>

Ah, seems it is only missing in that help message and accepting bookworm option.

[mutability]

It's there, I just forgot to update the usage message, yeah