bootc status failed with error `Reading deployment metadata: Missing base image ref ostree/container/blob/e05a2` - Githubissues

containers / bootc

Boot and upgrade via container images

https://containers.github.io/bootc/

Apache License 2.0

781 stars 84 forks source link

bootc status failed with error `Reading deployment metadata: Missing base image ref ostree/container/blob/e05a2` #800

Closed henrywang closed 2 months ago

henrywang commented 2 months ago

This issue is for anaconda installation only (bib anaconda-iso included), bootc install does not have this issue. Test result: https://gitlab.com/redhat/rhel/bifrost/tests/bootc-image/-/issues/219

Deploy bootc image with anaconda kickstart and boot the system. System can be booted without error. Check ostree status with rpm-ostree successful. But check with bootc status failed.

$ sudo bootc status
ERROR: Status: Computing status: Booted deployment: Reading deployment metadata: Missing base image ref ostree/container/blob/e05a222db71963792f1e74c3ca70a0009ae55e8794de45e8f8d7c9cea416936e"

$ rpm-ostree status
Deployments:
    ● ostree-unverified-registry:192.168.100.1:5000/hidden:22t5
                       Digest: sha256:d1aa7635c6fa50c70e8cfd110e229d1a2329a48609616e1bc585b9c8ed8bf3f4
                      Version: 9.20240920.0 (2024-09-20T21:59:00Z)

This issue has been found since bootc-202409192026.g85b2ca5256-1.el9.x86_64.
The bootc-202409191923.geea3996f8f-1.el9.x86_64 does not have this issue.
Check bootc commit from eea3996f8f to 85b2ca5256. The https://github.com/containers/bootc/commit/a8737d9e1456ca85db99d8911a6768f0f2aab21f might be the root case.

cgwalters commented 2 months ago

Yes I saw this on one machine too, it's really concerning.

We may have broken the semantics for how we write refs as part of the upgrade in https://github.com/ostreedev/ostree-rs-ext/pull/663

cgwalters commented 2 months ago

PR in https://github.com/ostreedev/ostree-rs-ext/pull/666

cgwalters commented 2 months ago

And https://github.com/containers/oci-spec-rs/pull/229

cgwalters commented 2 months ago

Should be fixed by https://github.com/containers/bootc/pull/804 - thanks for the report.

henrywang commented 2 months ago

Verified on bootc-202409242020.g34e104d217-1.el9.x86_64.

cgwalters commented 2 months ago

This issue is for anaconda installation only (bib anaconda-iso included),

To explain the reason for this (it's important) - it's because the way Anaconda works today we rely on the version of the ostree-container code which is embedded in the ISO (ref https://github.com/rhinstaller/anaconda/discussions/5197 ). So we have version skew.

But we really need to test the version-skew case in general, i.e. we want an "upgrade from stable" test that needs to gate PRs on this repo by default probably.

henrywang commented 1 month ago

But we really need to test the version-skew case in general, i.e. we want an "upgrade from stable" test that needs to gate PRs on this repo by default probably.

Ack. It's in my list. The test flow would be start vm from fedora/cs-bootc image, and upgrade bootc with dnf upgrade (after bootc usr-overlay), then try bootc status. Adding this test into integration test is more reasonable.

henrywang commented 1 month ago

To explain the reason for this (it's important) - it's because the way Anaconda works today we rely on the version of the ostree-container code which is embedded in the ISO (ref rhinstaller/anaconda#5197 ). So we have version skew.

Hi @cgwalters, do you mean the ostree-rs-ext version bumped by rpm-ostree (which is used by anadonda) is different from ostree-rs-ext version bumped by bootc in system. That caused this issue. Do I understand correct? Thanks.

cgwalters commented 1 month ago

Hi @cgwalters, do you mean the ostree-rs-ext version bumped by rpm-ostree (which is used by anadonda) is different from ostree-rs-ext version bumped by bootc in system. That caused this issue. Do I understand correct? Thanks.

Yep!