[Stability]: making apt and its dependencies compile almost static

[twaik]

Everybody here knows that partial upgrades are not supported. But in the case it happens and one or more dependencies of apt or dpkg getting broken environment becomes non-recoverable (for most users). Can we make dpkg, apt and probably some other essential packages link against static libraries and not their dynamic versions? I know, it will make bootstrap archives heavier, but it will make environment less fragile.

[twaik]

I mean to avoid situations like this.

[Biswa96]

Couldn't the above situation happen with any package or its dependencies? The issue boils down to static vs. shared linking and their pros & cons. Though, I have not seen other Linux distribution statically links package manager.

[twaik]

Any other package can be reinstalled with apt or dpkg. But in the case if apt is broken users can only reinstall the whole environment since package installation and update mechanisms become broken. Also other Linux distributions support partial upgrades so I will not mention their behaviour.

[Biswa96]

What would happen if any dependency of the default shell is broken?

[twaik]

It may be reinstalled with apt in failsafe mode.

[Biswa96]

I assume, one could replace the broken .so file to repair apt/dpkg in failsafe mode. ar extracts deb > tar extracts data.tar. It may be hard for first time user.

[twaik]

Exactly. It will be better to ship less fragile versions of dpkg and apt (and probably pacman and its dependencies too) to avoid such situations.

[agnostic-apollo]

Static binaries wouldn't work for linker exec.

https://github.com/termux/termux-exec/pull/24#issuecomment-1743513480

[twaik]

I did not suggest using static binaries. Only using static libraries to make apt and dpkg dynamic binaries with no shared library dependencies.

[licy183]

It would be better to revise their dependencies and add stricter version number restrictions to ensure that no destructive changes will happen after package updates.

[twaik]

ensure that no destructive changes will happen after package updates.

Also we can try to make qemu-based autotests to ensure apt and dpkg will work after upgrade, but even in this case there is no 100% protection from fiascos. Shit happens. Making apt and dpkg dynamic binaries with no dynamic dependencies is not 100% wise, but at least it will protect apt and dpkg from breakage without much effort. They will work even in the case if some update will purge libssl or liblzma libraries completely by mistake or during sabotage actions made by third party packages or scripts. As I said before apt and dpkg are the most important packages. All other packages can be fixed with apt install --reinstall or apt update && apt upgrade, but apt breakage makes environment unrecoverable for most users because fixing environment is tough thing for them.

[sylirre]

apt depends on at least dash and coreutils external binaries. dpkg requires at least tar binary. Probably they have more external binary dependenices.

If you want to prevent failure due to missing library, you'll need to ensure that all used components have dependencies built-in.

[agnostic-apollo]

In addition to stricter version reqs and testing before package updates, another thing can possibly be used to reduce issues. Instead of changing apt, dpkg or pacman, we can add support in pkg install (our recommended tool for package management) where if a package is being updated, and it is a dependency of some critical package like apt, dpkg or pacman, bash, coreutils, tar, etc, then any of those critical dependent packages are also automatically updated by default along with the user supplied dependency package, possibly with some opt out flag, like --no-dep-install. We can hardcode a list of all the important dependency packages in pkg script for which such logic should run instead of always running.

The other way round will need to be handled as well where if apt or some other critical package is updated with apt install, its recursive dependencies may not be updated. Those should be automatically upgraded as well.

• https://stackoverflow.com/questions/22008193/how-to-list-download-the-recursive-dependencies-of-a-debian-package
• https://unix.stackexchange.com/questions/493306/how-to-upgrade-one-single-package-and-all-its-dependencies-with-apt

[twaik]

Yeah, there are pretty much lots of dependencies.

twaik@twaikpc:~$ termux apt-cache depends -i --recurse apt | grep Depends | sort | uniq
  Depends: bzip2
  Depends: ca-certificates
  Depends: coreutils
  Depends: diffutils
  Depends: dpkg
  Depends: findutils
  Depends: gpgv
  Depends: grep
  Depends: gzip
  Depends: less
  Depends: libandroid-glob
  Depends: libandroid-support
  Depends: libassuan
  Depends: libbz2
  Depends: libc++
  Depends: libcurl
  Depends: libevent
  Depends: libgcrypt
  Depends: libgmp
  Depends: libgnutls
  Depends: libgpg-error
  Depends: libiconv
  Depends: libidn2
  Depends: liblz4
  Depends: liblzma
  Depends: libmd
  Depends: libnettle
  Depends: libnghttp2
  Depends: libnghttp3
  Depends: libnpth
  Depends: libssh2
  Depends: libunbound
  Depends: libunistring
  Depends: ncurses
  Depends: openssl
  Depends: pcre2
  Depends: resolv-conf
  Depends: sed
  Depends: tar
  Depends: termux-keyring
  Depends: termux-licenses
  Depends: xxhash
  Depends: xz-utils
  Depends: zlib
  Depends: zstd

So making all of them semi-static will be a problem...

[twaik]

Ok, that is a lot of binaries. I did not think about that when I created this issue.

~ $ for pkg in $(apt-cache depends -i --recurse apt | grep Depends | cut -d: -f2 | sort | uniq); do     dpkg -L $pkg | grep '/usr/bin/'; done | sort | uniq | xargs ls -la | grep -v "\->"
-rwx------ 1 system system    2197 Apr 14 02:30 /data/data/com.termux/files/usr/bin/bzdiff
-rwx------ 1 system system    2085 Apr 14 02:30 /data/data/com.termux/files/usr/bin/bzgrep
-rwx------ 1 system system   26200 Apr 14 02:30 /data/data/com.termux/files/usr/bin/bzip2
-rwx------ 1 system system   13832 Apr 14 02:30 /data/data/com.termux/files/usr/bin/bzip2recover
-rwx------ 1 system system    1290 Apr 14 02:30 /data/data/com.termux/files/usr/bin/bzmore
-rwx------ 1 system system    6180 Nov 20 06:47 /data/data/com.termux/files/usr/bin/clear
-rwx------ 1 system system   34664 Jun  1  2023 /data/data/com.termux/files/usr/bin/cmp
-rwx------ 1 system system 1213800 Apr 27 04:19 /data/data/com.termux/files/usr/bin/coreutils
-rwx------ 1 system system    7364 Mar 27 12:23 /data/data/com.termux/files/usr/bin/curl-config
-rwx------ 1 system system  174964 Jun  1  2023 /data/data/com.termux/files/usr/bin/diff
-rwx------ 1 system system   48232 Jun  1  2023 /data/data/com.termux/files/usr/bin/diff3
-rwx------ 1 system system  254832 Mar 17 10:59 /data/data/com.termux/files/usr/bin/dpkg
-rwx------ 1 system system    1821 Mar 17 10:59 /data/data/com.termux/files/usr/bin/dpkg-buildapi
-rwx------ 1 system system  138632 Mar 17 10:59 /data/data/com.termux/files/usr/bin/dpkg-deb
-rwx------ 1 system system  124564 Mar 17 10:59 /data/data/com.termux/files/usr/bin/dpkg-divert
-rwx------ 1 system system   17433 Mar 17 10:59 /data/data/com.termux/files/usr/bin/dpkg-fsys-usrunmess
-rwx------ 1 system system  130760 Mar 17 10:59 /data/data/com.termux/files/usr/bin/dpkg-query
-rwx------ 1 system system    4243 Mar 17 10:59 /data/data/com.termux/files/usr/bin/dpkg-realpath
-rwx------ 1 system system  103640 Mar 17 10:59 /data/data/com.termux/files/usr/bin/dpkg-split
-rwx------ 1 system system   68808 Mar 17 10:59 /data/data/com.termux/files/usr/bin/dpkg-trigger
-rwx------ 1 system system    9768 Dec 25 13:41 /data/data/com.termux/files/usr/bin/dumpsexp
-rwx------ 1 system system     135 May 14  2023 /data/data/com.termux/files/usr/bin/egrep
-rwx------ 1 system system     135 May 14  2023 /data/data/com.termux/files/usr/bin/fgrep
-rwx------ 1 system system  245216 Apr 14 02:30 /data/data/com.termux/files/usr/bin/find
-rwx------ 1 system system   24944 Mar 17 09:10 /data/data/com.termux/files/usr/bin/gpg-error
-rwx------ 1 system system    2176 Mar 17 09:10 /data/data/com.termux/files/usr/bin/gpg-error-config
-rwx------ 1 system system   16333 Mar 17 09:10 /data/data/com.termux/files/usr/bin/gpgrt-config
-rwx------ 1 system system  457112 Jan 29 15:06 /data/data/com.termux/files/usr/bin/gpgv
-rwx------ 1 system system  191740 May 14  2023 /data/data/com.termux/files/usr/bin/grep
-rwx------ 1 system system    2368 Aug 21  2023 /data/data/com.termux/files/usr/bin/gunzip
-rwx------ 1 system system    6577 Aug 21  2023 /data/data/com.termux/files/usr/bin/gzexe
-rwx------ 1 system system   98472 Aug 21  2023 /data/data/com.termux/files/usr/bin/gzip
-rwx------ 1 system system   10508 Dec 25 13:41 /data/data/com.termux/files/usr/bin/hmac256
-rwx------ 1 system system   20368 Apr 14 02:30 /data/data/com.termux/files/usr/bin/idn2
-rwx------ 1 system system  180056 Jun  1  2023 /data/data/com.termux/files/usr/bin/less
-rwx------ 1 system system    5876 Jun  1  2023 /data/data/com.termux/files/usr/bin/lessecho
-rwx------ 1 system system   13608 Jun  1  2023 /data/data/com.termux/files/usr/bin/lesskey
-rwx------ 1 system system    2914 Jun 19  2023 /data/data/com.termux/files/usr/bin/libassuan-config
-rwx------ 1 system system    4568 Dec 25 13:41 /data/data/com.termux/files/usr/bin/libgcrypt-config
-rwx------ 1 system system    7612 Mar 29 21:21 /data/data/com.termux/files/usr/bin/lzmadec
-rwx------ 1 system system    6852 Mar 29 21:21 /data/data/com.termux/files/usr/bin/lzmainfo
-rwx------ 1 system system   11152 Dec 25 13:41 /data/data/com.termux/files/usr/bin/mpicalc
-rwx------ 1 system system    8937 Nov 20 06:47 /data/data/com.termux/files/usr/bin/ncursesw6-config
-rwx------ 1 system system    3169 Jan 21 18:11 /data/data/com.termux/files/usr/bin/npth-config
-rwx------ 1 system system    2318 Feb 16 18:16 /data/data/com.termux/files/usr/bin/pcre2-config
-rwx------ 1 system system   37692 Jun  1  2023 /data/data/com.termux/files/usr/bin/sdiff
-rwx------ 1 system system  149480 Apr  1  2023 /data/data/com.termux/files/usr/bin/sed
-rwx------ 1 system system   31216 Mar 17 10:59 /data/data/com.termux/files/usr/bin/start-stop-daemon
-rwx------ 1 system system  442912 Jul 21  2023 /data/data/com.termux/files/usr/bin/tar
-rwx------ 1 system system   15236 Nov 20 06:47 /data/data/com.termux/files/usr/bin/tset
-rwx------ 1 system system    2368 Aug 21  2023 /data/data/com.termux/files/usr/bin/uncompress
-rwx------ 1 system system   38440 Mar 17 10:59 /data/data/com.termux/files/usr/bin/update-alternatives
-rwx------ 1 system system   59520 Apr 14 02:30 /data/data/com.termux/files/usr/bin/xargs
-rwx------ 1 system system   49012 Jul 21  2023 /data/data/com.termux/files/usr/bin/xxhsum
-rwx------ 1 system system   64128 Mar 29 21:21 /data/data/com.termux/files/usr/bin/xz
-rwx------ 1 system system    7596 Mar 29 21:21 /data/data/com.termux/files/usr/bin/xzdec
-rwx------ 1 system system    7455 Mar 29 21:21 /data/data/com.termux/files/usr/bin/xzdiff
-rwx------ 1 system system   10366 Mar 29 21:21 /data/data/com.termux/files/usr/bin/xzgrep
-rwx------ 1 system system    1846 Mar 29 21:21 /data/data/com.termux/files/usr/bin/xzless
-rwx------ 1 system system    2223 Mar 29 21:21 /data/data/com.termux/files/usr/bin/xzmore
-rwx------ 1 system system   29320 Mar 17 09:10 /data/data/com.termux/files/usr/bin/yat2m
-rwx------ 1 system system    2006 Aug 21  2023 /data/data/com.termux/files/usr/bin/zcat
-rwx------ 1 system system    1711 Aug 21  2023 /data/data/com.termux/files/usr/bin/zcmp
-rwx------ 1 system system    6513 Aug 21  2023 /data/data/com.termux/files/usr/bin/zdiff
-rwx------ 1 system system      62 Aug 21  2023 /data/data/com.termux/files/usr/bin/zegrep
-rwx------ 1 system system      62 Aug 21  2023 /data/data/com.termux/files/usr/bin/zfgrep
-rwx------ 1 system system    2109 Aug 21  2023 /data/data/com.termux/files/usr/bin/zforce
-rwx------ 1 system system    8263 Aug 21  2023 /data/data/com.termux/files/usr/bin/zgrep
-rwx------ 1 system system    1870 Aug 21  2023 /data/data/com.termux/files/usr/bin/zmore
-rwx------ 1 system system    4605 Aug 21  2023 /data/data/com.termux/files/usr/bin/znew
-rwx------ 1 system system  174488 Mar 27 00:33 /data/data/com.termux/files/usr/bin/zstd
-rwx------ 1 system system    3900 Mar 27 00:33 /data/data/com.termux/files/usr/bin/zstdgrep
-rwx------ 1 system system     228 Mar 27 00:33 /data/data/com.termux/files/usr/bin/zstdless

[twaik]

The other way round will need to be handled as well where if apt or some other critical package is updated with apt install, its recursive dependencies may not be updated. Those should be automatically upgraded as well.

It can protect end users from partial upgrade issues, but it can not protect them from breaking updates. I mean if some maintainer upgraded essential package without testing it properly. So users still will have no protection against cannot locate symbol issues. I think the best thing we can do is to mix your solution with building bootstrap and invoking some tests (probably we should start with simple invoking apt, invoking package installing and updating) and make build-package.sh script fail in the case if the package is essential and installing it breaks an ability to install/update/upgrade packages.

[twaik]

20944 made me think it is still a good idea to make apt and its executable dependencies non-dependent on shared libraries. Even in the case if dependency will be updated there will be no shared library conflict and linker will not raise CANNOT LINK EXECUTABLE exception.