[Tracker]: Making subpackages act like regular packages.

TomJo2000 commented 4 weeks ago

Several issues have come up regarding the state of subpackages. This issue is intended to summarize those issues and track progress on solving them.

1.) Subpackage builds are a separate - incomplete - format.
2.) Subpackages do not behave like regular packages when included as dependencies in package builds.
3.) Subpackages cause cycles in the build order.

Potential solutions: - Phase out subpackages entirely. ^{Undesirable due to large amounts of duplicated build code.} - "Inline" subpackages, along the lines of what Arch Linux for example does. ^{https://man.archlinux.org/man/PKGBUILD.5#PACKAGE_SPLITTING} ^{Example: https://gitlab.archlinux.org/archlinux/packaging/packages/lpeg/-/blob/main/PKGBUILD?ref_type=heads} - "Fixing" the existing subpackage model ^{Would require additional capabilities, and/or clarification on the role and scope of subpackages.}

Biswa96 commented 4 weeks ago

Could you provide any package of this repository which needs this kind of fix?

TomJo2000 commented 4 weeks ago

Anything that has one or more subpackages. For example, specifying lua51-lpeg as a dependency pulls in all 4 of them, even though it doesn't depend on the parent package.

Grimler91 commented 4 weeks ago

Subpackages cause cycles in the build order.

At the moment subpackages automatically depends on parent, which is most of the time the correct thing we want. We have TERMUX_SUBPKG_DEPEND_ON_PARENT to change this behaviour, but that flag is easy to miss.

We have for example vim which depends on vim-runtime, which in turns depends on its parent vim since TERMUX_SUBPKG_DEPEND_ON_PARENT=no is not specified for it.

Linter could perhaps help us detect these issues, could run the buildorder script (after @tstein's improvements) and check so we don't have any cycles. Just need to actually fix all the package cycles first.. :)

I think fixing current model is the way to go. Archlinux's model works differently for example, they unlike us never(/seldom) splits library and binary (libsqlite vs sqlite for example), so model for splitting subpackages is different

Maxython commented 4 weeks ago

Subpackage builds are a separate - incomplete - format.

By subpackage builds, do you mean the *.subpackage.sh files? If so, can you indicate what they lack for a full-fledged format?

Subpackages do not behave like regular packages when included as dependencies in package builds.

Unclear. Write the problem in more detail.

Subpackages cause cycles in the build order.

Circular dependencies in packages are normal, as long as they are needed to compile the package itself. I think that such issues should be resolved not by changing the structure of subpackages, but by changing the logic in the scripts. It is worth noting that build-package.sh can already detect cyclic dependencies in packages and fix them. To do this, it installs the package that started the cycle with no version binding. And yes, the package that was installed to solve the cyclic dependency will definitely be recompiled with a certain version.

Biswa96 commented 4 weeks ago

If I understand correctly the issue is not about the sub-packages for the lua51-lpeg case. The problem occurs when a package is built which depends on a sub-package.

TomJo2000 commented 4 weeks ago

By subpackage builds, do you mean the *.subpackage.sh files? If so, can you indicate what they lack for a full-fledged format?

Yes, subpackages. By "missing features" I mean they don't have access to any of the usual build steps. They're essentially stuck at post-post_massage in the build step order.

Unclear. Write the problem in more detail.

When I set lua51-lpeg (which is a subpackage of lua-lpeg, i.e lpeg 5.4) as a dependency for neovim it pulls in:

lua51-lpeg
lua52-lpeg
lua53-lpeg
and lua-lpeg
as well as their respective Lua and Liblua versions.

Circular dependencies in packages are normal, as long as they are needed to compile the package itself. I think that such issues should be resolved not by changing the structure of subpackages, but by changing the logic in the scripts. It is worth noting that build-package.sh can already detect cyclic dependencies in packages and fix them. To do this, it installs the package that started the cycle with no version binding. And yes, the package that was installed to solve the cyclic dependency will definitely be recompiled with a certain version.

Point taken, and you have taken some steps in the fix-build-all branch to solve that particular issue. I realize I was a bit quick with my "tear it down and take a new approach" suggestion.

Maxython commented 4 weeks ago

When I set lua51-lpeg (which is a subpackage of lua-lpeg, i.e lpeg 5.4) as a dependency for neovim it pulls in

To avoid this you should configure the value TERMUX_PKG_SEPARATE_SUB_DEPENDS=true in the lua-lpeg package.

TomJo2000 commented 4 weeks ago

To avoid this you should configure the value TERMUX_PKG_SEPARATE_SUB_DEPENDS=true in the lua-lpeg package.

Would be nice if that were documented anywhere. Like say, the available control fields for subpackages, but the only reference to that variable is in buildorder.py. https://github.com/termux/termux-packages/blob/67f175612833f64bd9f2d05325c1c23eaedbe3a2/scripts/buildorder.py#L117 Which I for one, do not usually check when writing a package.

Anyway thanks for the tip.

Biswa96 commented 4 weeks ago

To avoid this you should configure the value TERMUX_PKG_SEPARATE_SUB_DEPENDS=true in the lua-lpeg package.

Should not that variable be always true? The build script should have the required dependencies irrespective of a package being a sub-package.

TomJo2000 commented 4 weeks ago

Well subpackages can have divergent dependencies. For example qbittorrent-nox doesn't need the X11 stuff.

Grimler91 commented 4 weeks ago

qbittorrent-nox seems like a good example of when TERMUX_SUBPKG_DEPEND_ON_PARENT=no can be used. qbittorrent-nox doesn't depend on anything in qbittorrent AFAIK, so it should be possible to specify TERMUX_SUBPKG_DEPEND_ON_PARENT=no for it

TomJo2000 commented 4 weeks ago

I think I'm just gonna open up a general "get some messed up subpackages sorted out" PR at this point.

qbittorrent-nox seems like a good example of when TERMUX_SUBPKG_DEPEND_ON_PARENT=no can be used. qbittorrent-nox doesn't depend on anything in qbittorrent AFAIK, so it should be possible to specify TERMUX_SUBPKG_DEPEND_ON_PARENT=no for it

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/x11-packages/qbittorrent/qbittorrent-nox.subpackage.sh#L1-L4 Nevermind, it already does that.

Grimler91 commented 4 weeks ago

Ah, yes, you are right, sorry for the noise

TomJo2000 commented 4 weeks ago

I'm still going to have a look through all our various subpackages for more stuff to keep track of here.

TomJo2000 commented 4 weeks ago

gpgv

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/gnupg/gpgv.subpackage.sh#L1-L4

can probably use the TERMUX_PKG_SEPARATE_SUB_DEPENDS=true treatment from what it looks like.
Same for:

libsmartcols

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/util-linux/libsmartcols.subpackage.sh#L1-L9

libltdl

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/libtool/libltdl.subpackage.sh#L1-L9

libuuid

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/util-linux/libuuid.subpackage.sh#L1-L9

uuid-utils

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/util-linux/uuid-utils.subpackage.sh#L1-L4

blk-utils

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/util-linux/blk-utils.subpackage.sh#L1-L6

libfdisk

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/util-linux/libfdisk.subpackage.sh#L1-L8

llvm-tools

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/libllvm/llvm-tools.subpackage.sh#L14-L18

sqlite

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/libsqlite/sqlite.subpackage.sh#L1-L4

gdbserver

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/gdb/gdbserver.subpackage.sh#L5-L9

rav1e

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/librav1e/rav1e.subpackage.sh#L1-L3

binutils

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/binutils-libs/binutils.subpackage.sh#L18-L21

More unclear cases:

nmap-ncat

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/nmap/nmap-ncat.subpackage.sh#L1-L9

xorg-server-xephyr

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/x11-packages/xorg-server/xorg-server-xephyr.subpackage.sh#L1-L4

emscripten-llvm

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/emscripten/emscripten-llvm.subpackage.sh#L1-L5

qbittorrent-nox

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/x11-packages/qbittorrent/qbittorrent-nox.subpackage.sh#L1-L4

openssh-sftp-server

https://github.com/termux/termux-packages/blob/91f1c47a5faca47740de34b17f875afbc6d73dec/packages/openssh/openssh-sftp-server.subpackage.sh#L5-L9

termux / termux-packages

[Tracker]: Making subpackages act like regular packages. #20500