client layering fails with custom image without `/etc/passwd`

orowith2os commented 1 year ago

Host system details

~ took 5s 
❯ rpm-ostree status
State: idle
Deployments:
● fedora:fedora/37/x86_64/silverblue
                  Version: 37.20221231.0 (2022-12-31T00:39:25Z)
               BaseCommit: c5e55106da4dd577c162e9d89077bd18e1175c7d14d6ca0e2de15e783251bb6b
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
      RemovedBasePackages: firefox firefox-langpacks 108.0.1-3.fc37
          RemoteOverrides: repo=copr:copr.fedorainfracloud.org:kylegospo:gnome-vrr
                           ├─ gnome-control-center 43.1-1.fc37 -> 43.1.vrr.3-1.fc37
                           ├─ gnome-control-center-filesystem 43.1-1.fc37 -> 43.1.vrr.3-1.fc37
                           └─ mutter 43.2-1.fc37 -> 43.2.vrr.11-1.fc37
          LayeredPackages: corectrl fish langpacks-en

  fedora:fedora/37/x86_64/silverblue
                  Version: 37.20221217.0 (2022-12-17T00:43:29Z)
               BaseCommit: 214e87c60213bc9066243aeb975b30287af585291f5f0424ade3b444c4c7e46a
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
      RemovedBasePackages: firefox firefox-langpacks 108.0-2.fc37
          RemoteOverrides: repo=copr:copr.fedorainfracloud.org:kylegospo:gnome-vrr
                           ├─ gnome-control-center 43.1-1.fc37 -> 43.1.vrr.3-1.fc37
                           ├─ gnome-control-center-filesystem 43.1-1.fc37 -> 43.1.vrr.3-1.fc37
                           └─ mutter 43.2-1.fc37 -> 43.2.vrr.11-1.fc37
          LayeredPackages: corectrl fish langpacks-en

Expected vs actual behavior

~ took 8m49s 
❯ sudo rpm-ostree rebase --experimental ostree-unverified-registry:ghcr.io/orowith2os/pitti-workstation-oci:latest
Pulling manifest: ostree-unverified-image:docker://ghcr.io/orowith2os/pitti-workstation-oci:latest
Checking out tree 3db15ab... done
Inactive base removals:
  firefox
  firefox-langpacks
Inactive base replacements:
  remove
  gnome-control-center
  gnome-control-center-filesystem
  mutter
Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora copr:copr.fedorainfracloud.org:kylegospo:gnome-vrr updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2022-10-06T11:01:40Z solvables: 4
rpm-md repo 'fedora-modular' (cached); generated: 2022-11-10T09:23:24Z solvables: 1454
rpm-md repo 'updates-modular' (cached); generated: 2022-12-30T01:52:36Z solvables: 1460
rpm-md repo 'updates' (cached); generated: 2023-01-01T01:29:53Z solvables: 14917
rpm-md repo 'fedora' (cached); generated: 2022-11-10T09:30:00Z solvables: 66822
rpm-md repo 'copr:copr.fedorainfracloud.org:kylegospo:gnome-vrr' (cached); generated: 2022-12-10T11:03:31Z solvables: 13
rpm-md repo 'updates-archive' (cached); generated: 2023-01-01T01:53:37Z solvables: 15642
Resolving dependencies... done
Checking out packages... done
error: No such file or directory (os error 2)

Expected: rpm-ostree would rebase successfully

Steps to reproduce it

Not sure, try sudo rpm-ostree rebase --experimental ostree-unverified-registry:ghcr.io/orowith2os/pitti-workstation-oci:latest

Would you like to work on the issue?

I can try to help with bug testing and whatnot, but I don't know C/C++, and very little Rust.

cgwalters commented 1 year ago

Do you really want to be carrying these local overrides on top of the remote image? It would likely work to drop those overrides, then rebase like:

$ rpm-ostree reset -lo
$ rpm-ostree rebase ...

(Which, hm we should definitely add an rpm-ostree rebase --reset too, though of course this gets into https://github.com/coreos/rpm-ostree/issues/2326 and https://github.com/coreos/rpm-ostree/issues/3403 too)

error: No such file or directory (os error 2)

Hmm. The dreaded unprefixed error; debugging this will be a bit of a pain without me trying to setup a full reproducer environment. Do you know how to use strace? Basically from a privileged container (or installed/layered strace binary), do e.g. rpm-ostree status && strace -f -p $(systemctl show -p MainPID rpm-ostreed | cut -f 2 -d =) -o /tmp/strace.log & and then run the rebase command, then something like 'grep ENOENT /tmp/strace.log` would be helpful.

orowith2os commented 1 year ago

Do you really want to be carrying these local overrides on top of the remote image? It would likely work to drop those overrides, then rebase like: ...

Shouldn't rpm-ostree automatically make those overrides inactive? IIRC it does that for some remove overrides, and I want to keep the overrides active for whenever I use a GNOME-based image.

I also do not know how to use strace, but I can run the commands you listed. One moment.

orowith2os commented 1 year ago

Update: The commands you showed didn't quite work, it just put an empty file into /tmp.

cgwalters commented 1 year ago

OK https://github.com/coreos/rpm-ostree/pull/4246 gets me:

error: Preparing passwd content: Preparing pwgrp: Renaming original etc/passwd: No such file or directory (os error 2)

Which is far more helpful, and shows that this image is missing /etc/passwd. In theory perhaps we could try to support this in some way, but for now...don't delete /etc/passwd? Where's the source code that builds this?

orowith2os commented 1 year ago

The image I'm trying to rebase to is located at https://github.com/orowith2os/pitti-workstation-oci/blob/main/Containerfile

cgwalters commented 1 year ago

Right, so https://github.com/orowith2os/pitti-workstation-oci/blob/956f7fdd97d3cc0d4cecbd027b0dff55101fdb1a/setup.sh#L25 is introducing this problem right now.

Now...the thing is, ultimately having user/groups only in /usr/lib does make some sense, but there's a conflict right now with trying to do that and local package layering.

I think the semantics you want are to rpm-ostree reset -ol before doing the rebase anyways so that you're in "pure image" mode.

coreos / rpm-ostree

client layering fails with custom image without `/etc/passwd` #4244