New debootstrap is unable to bootstrap pre-bookworm debian derivatives

[sjoerdsimons]

Debian unstable/testing as well as the latest debian stable release have a patch to the debian-common script which requires usr-is-merged on anything that's not an older known Debian release..

Which breaks using that debootstrap for any derivative distribution with a different codename, which is rather unfortunate as that's a reasonable common usage.

Without changing debootstrap we have the following options:

• 16 - Making the debootstrap script configurable.. While that would give an escape hatch, it will require using a different

     script entirely as the offending code is in debian-common

• 127 - Support another bootstrapping tool; That won't fix current recipes unless it's done as a drop-in replacement. But switching

        the bootstrapping tool underneath the user opens a whole other can of worms. And really i'd like to avoid having to
        support multiple tools, even with all its issues debootstrap is still the official bootstrapping tools

• 358 - Expose debootstraps include and exclude options; Which we avoided thusfar as debootstrap really isn't the right location

        to add more packages (use apt actions for that) or to remove package as that's risky as well. When using a minbase 
        variant the latter really shouldn't be needed. Not unsurprising it was only proposed due to this issue. And ofcourse, this 
        still requires users to switch their recipes so it's quite annoying.

TL;DR we can't have nice things; and in particular all these options require users to chagne their recipes which is really not great

[obbardc]

One attempted modification to debootstrap was to only install usr-is-merged if the suite is testing or unstable; it wasn't accepted and the developer's suggestion was to exclude usr-is-merge by debootstrap's --exclude argument.

[sjoerdsimons]

Digging into this further with some pointers from @smcv; The current overview is:

• usr-is-merged is a dummy package to say:
  • This system was created in a merged usr form
  • This system expclitly opted out from merged-usr (flag file in etc: unsupported-skip-usrmerge-conversion); Which is mostly intended for QA reasons.

New debootstrap (stable and onwards) will on codenames that it doesn't recognize as an older debian version always require this package and if --no-merged-usr is given will create the flag file mentioned above. Ideally with the goal of avoiding the installation of usrmerge. Though when testing with debootstrapping sid for the new init-system-helpers (with updated dependencies) it seems that still pulls in both usrmerge and usr-is-merged.

Given that as a quick fix i'm planning to simply hardcode --exclude=usr-is-merged which will mostly get us back to the behaviour we had before. The impact on non-debian suites will be the flag file gets dropped in etc when running a non-usr-merged setup and no changes otherwise. For sid (and soon bookworm) based systems it will mean that usr-is-merged doesn't get installed, but usrmerge is (as it would anyway as things stand at the moment).

For a longer term solution on the debos side we probably want to have a field in the debootstrap action to indicate what "base" distribution the bootstrapping should be done for, which will then also be used for the debootstrap script and potentially to do workarounds like the above only for older debian releases.. That said to avoid dropping the flag file that would still need some fixes to debootstrap as it bases its decision on the codename of the release file rather then the debootstrap script being used

[obbardc]

This has been fixed in #362 but let's keep this open to track the longer-term solution.

[Penlect]

Dear maintainers, I would like to bring something to your attention that I discovered today when using debos (in Debian bookworm, version 1.1.1-2) that seems related to this issue.

When using bookworm as suit in the debootstrap action

  - action: debootstrap
    suite: bookworm
    components:
      - main
      - contrib
      - non-free
    mirror: https://deb.debian.org/debian
    variant: minbase

any following apt-action step will fail with the following message:

2022/12/13 09:01:26 ==== Install system packages ====
Hit:1 https://deb.debian.org/debian bookworm InRelease
Get:2 https://deb.debian.org/debian bookworm/main Translation-en [5941 kB]
Get:3 https://deb.debian.org/debian bookworm/contrib Translation-en [46.9 kB]
Get:4 https://deb.debian.org/debian bookworm/non-free Translation-en [92.3 kB]
Fetched 6080 kB in 1s (9733 kB/s)                                      
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
2022/12/13 09:01:27 apt | You might want to run 'apt --fix-broken install' to correct these.
2022/12/13 09:01:27 apt | The following packages have unmet dependencies:
2022/12/13 09:01:27 apt |  init-system-helpers : Depends: usrmerge but it is not going to be installed or
2022/12/13 09:01:27 apt |                                 usr-is-merged
2022/12/13 09:01:27 apt |  systemd : Depends: libcryptsetup12 (>= 2:2.4) but it is not going to be installed
2022/12/13 09:01:27 apt |            Depends: libfdisk1 (>= 2.33) but it is not going to be installed
2022/12/13 09:01:27 apt |            Depends: libkmod2 (>= 15) but it is not going to be installed
2022/12/13 09:01:27 apt |            Depends: libsystemd-shared (= 252.2-2) but it is not going to be installed
2022/12/13 09:01:27 apt | E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
2022/12/13 09:01:27 Action `Install system packages` failed at stage Run, error: exit status 100

However, installing usr-is-merged immediately after the deboostrap-action seems to be a workaround

  - action: apt
    packages:
      - usr-is-merged

[elboulangero]

@obbardc The commit https://github.com/go-debos/debos/commit/18998ffaf78321e111d9823b3180eca3fa4593f6 doesn't work for derivatives based on recent Debian's suites (eg. Kali Linux based on testing). But I guess you already know it and there's no simple solution.

The workaround is simple though, just this step after debootstrap:

  - description: "Install usr-is-merged (cf. debos #361 and #362)"
    action: apt
    packages: [ usr-is-merged ]

[obbardc]

@obbardc The commit 18998ff doesn't work for derivatives based on recent Debian's suites (eg. Kali Linux based on testing).

Hi @elboulangero sorry for the late response, it got lost in my backlog. Can you please check the implementation under https://github.com/go-debos/debos/pull/424 ?

[elboulangero]

Hello @obbardc, I just tried #424 on my side, it works great for Kali. Outstanding work! Thank you very much!

[fortysixandtwo]

@obbardc The commit 18998ff doesn't work for derivatives based on recent Debian's suites (eg. Kali Linux based on testing). But I guess you already know it and there's no simple solution.

The workaround is simple though, just this step after debootstrap:

  - description: "Install usr-is-merged (cf. debos #361 and #362)"
    action: apt
    packages: [ usr-is-merged ]

Just a datapoint: This workaround did not work for me on arm64 (tried with PureOS crimson, which is based on bookworm) unless I also set variant: minbase on the debootstrap step.

I haven't checked in detail what the difference is between amd64 and arm64 in this case.

Anyway, personally my preferred solution would be #358 as this gives users control (and get rid of the logic that adds --exclude=usr-is-merged). I prefer it over #424 because that extra layer effectively does the same thing for this case.

[obbardc]

@fortysixandtwo #352 is not related. Can you check which PR you mean ? thanks.

[fortysixandtwo]

My bad, I meant #358 (edited in the above post as well)