Open holmanb opened 8 months ago
stefanor
commented
8 months ago Debian's interpretation here is that data should be installed into PREFIX (i.e. /usr). This matches what pip would do, and lines up with virtualenvs, which can be looked at as a different PREFIX.
So, yes, you can help distributions by laying out data usefully, but in Debian's case, we'll probably need to move the data around after we've installed the wheel.
dermotbradley
commented
7 months ago I had discussed this in the #cloud-init IRC channel several months ago as Alpine had switched a lot of packages from setup.py to gpep517.
After this change, tests on this branch started failing due to ds-identify and other "data_files" being installed in the wrong location.
That was exactly the same issue I experienced then when I tried to switch the Alpine cloud-init package to gpep517. At the time (from memory) Sam (_sam on IRC) indicated that gpep517 didn't support files outside of /var/lib/python/site-packages/ currently
holmanb
commented
7 months ago I had discussed this in the #cloud-init IRC channel several months ago as Alpine had switched a lot of packages from setup.py to gpep517.
After this change, tests on this branch started failing due to ds-identify and other "data_files" being installed in the wrong location.
That was exactly the same issue I experienced then when I tried to switch the Alpine cloud-init package to gpep517. At the time (from memory) Sam (_sam on IRC) indicated that gpep517 didn't support files outside of /var/lib/python/site-packages/ currently
@dermotbradley +1 Thanks for adding the extra context.
eli-schwartz
commented
7 months ago The python ecosystem officially considers it unsupported to use python packaging technology for anything other than:
The wheel standard supports installing files to hardcoded locations inside of /usr if and only if sys.prefix for python itself is /usr -- this means that installing into a virtualenv is probably broken as I somehow doubt the software will work correctly when installing /home/myname/.venvmanager/venvs/cloud-init/lib/udev/*, at which point you really, really should be asking "what am I getting out of this".
Note also that it is IMPOSSIBLE to install files to /etc using a wheel, since /etc inherently lies outside of python's sys.prefix.
On the flip side, you need to ask what you actually get out of wheels. It allows users to pip install the software, but do you need that functionality? Probably not... just get the software from the distro repository.
Effectively, python packaging is being used as a glorified makefile, and setup.py is kind of convenient because it supports locating the system site-packages directory.
I would recommend ditching the python packaging ecosystem entirely if you don't need it, and moving to a real build system such as meson.
You still get support for installing python modules using py_installation.install_sources(), but you also get to install files to standard locations such as --prefix, --sysconfdir, --localstatedir, and so on.
holmanb
commented
5 months ago This has been added to the 24.3 project milestone. FYI @dermotbradley @prometheanfire @thesamesam @rjschwei @igalic @ani-sinha @ncopa @major @nmeyerhans
Bug report
tl;dr: Changing cloud-init's packaging paradigm to support PEP517 packaging tooling appears unavoidable to support PEP517.
In https://github.com/canonical/cloud-init/commit/a476ecb5b1dd47527c907d2736cb2dd9ef845119 we tried switching debian packaging from dh-python's pybuild distutils plugin, (which depends on a standard library package which was removed from stdlib in 3.12) to dh-python's pybuild pyproject plugin.
After this change, tests on this branch started failing due to ds-identify and other "data_files" being installed in the wrong location.
After filing a bug with debian against dh-python, I had an IRC conversation in #debian-python on Libera. To summarize the conversation: other non-debian distros' packaging tooling similarly does wheel-based installations, so it's unlikely that we'll be able to keep the current packaging paradigm.