Support for manifests with local storage.

[cdrage]

When trying to implement https://github.com/containers/podman-desktop/issues/6236 related to the bootc extension, I had an intrusive thought, does bootc-image-builder support manifests?

Unfortunately I ran into issues using a locally created manifest with local storage.

It works when NOT passing in --local and having bootc-image-builder pull the manifest.

But manifests do not work with a local storage.

Steps to reproduce:

 git clone https://gitlab.com/bootc-org/examples
 cd httpd
 podman build --platform linux/arm64,linux/amd64 --manifest testmanifest .
 podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v $(pwd)/output:/output \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    quay.io/centos-bootc/bootc-image-builder:latest \
    --type qcow2 --target-arch arm64 \
    --local \
    testmanifest
Generating manifest manifest-qcow2.json
WARNING: target-arch is experimental and needs an installed 'qemu-user' package
panic: failed to resolve container: '': error getting manifest: reference "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/usr/lib/containers/storage,overlay.mountopt=nodev,metacopy=on]docker.io/library/testmanifest:latest" does not resolve to an image ID: identifier is not an image

goroutine 1 [running]:
main.cmdBuild(0x40001c2800?, {0x400008ad20, 0x1, 0x6})
    /build/bib/cmd/bootc-image-builder/main.go:367 +0xaf0
github.com/spf13/cobra.(*Command).execute(0x400040ec00, {0x400008ac60, 0x6, 0x6})
    /root/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983 +0x840
github.com/spf13/cobra.(*Command).ExecuteC(0x400040e900)
    /root/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
    /root/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
main.run()
    /build/bib/cmd/bootc-image-builder/main.go:511 +0x664
main.main()
    /build/bib/cmd/bootc-image-builder/main.go:515 +0x1c

Also tested with a different example image (seeing it it's issues with just the naming...), tested by naming it quay.io/test/image and still unable to build unfortunately.

[cdrage]

Ref: @kingsleyzissou @cgwalters

[kingsleyzissou]

What's the output of podman images?

[cdrage]

▶ podman images
REPOSITORY                                TAG                IMAGE ID      CREATED       SIZE
localhost/testmanifest                    latest             a1b05be52529  20 hours ago  1.08 kB
<none>                                    <none>             4a4b48a8f979  20 hours ago  1.64 GB
localhost/testimage123                    latest             010035bc30fe  22 hours ago  1.08 kB
<none>                                    <none>             2ea77ed59d9c  22 hours ago  1.64 GB
<none>                                    <none>             24544ba7c526  22 hours ago  1.41 kB
localhost/foobar                          latest             51055376f7cb  22 hours ago  486 B
quay.io/centos-bootc/centos-bootc-dev     stream9            a0273f465573  23 hours ago  1.54 GB
localhost/test                            latest             6a105d8fe5a7  24 hours ago  486 B
quay.io/test/image                        latest             263b0748604f  24 hours ago  486 B
quay.io/centos-bootc/bootc-image-builder  latest             5813f79c0726  28 hours ago  843 MB
<none>                                    <none>             3027951d09e2  29 hours ago  1.08 kB
<none>                                    <none>             6aed315874ac  29 hours ago  1.64 GB
<none>                                    <none>             10f120cbda1b  29 hours ago  1.64 GB
<none>                                    <none>             f114b942e8dd  8 days ago    1.65 GB
quay.io/centos-bootc/bootc-image-builder  latest-1710916056  0c39315d3899  2 weeks ago   842 MB

(excuse the other output).

FYI, even if you pass in localhost/testmanifest, or even if you tag it and name it quay.io/test/image it will not work. I've added the steps in the initial issue so you can reproduce.

[cdrage]

Here is the output of manifest and image inspect:

▶ podman manifest inspect testimage123
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 11204,
            "digest": "sha256:97f07804c174675610516fbb3af7486a7a80b2b105693ef875ce16af038f3cc2",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 11200,
            "digest": "sha256:f133768ad5f62cd9a74e3288711c5498613c6c285db560cefa448d1e6bb2bbed",
            "platform": {
                "architecture": "arm64",
                "os": "linux"
            }
        }
    ]
}

podman image inspect testimage123
[
     {
          "Id": "2ea77ed59d9c38f0233bb724aaba3e5c4443090db2b4a954f08ad816b8183049",
          "Digest": "sha256:f133768ad5f62cd9a74e3288711c5498613c6c285db560cefa448d1e6bb2bbed",
          "RepoTags": [],
          "RepoDigests": [],
          "Parent": "362fdd0b92b2cb152a4d5b4746947c47f3ae0d42955226421b551db900100372",
          "Comment": "",
          "Created": "2024-04-03T21:43:19.084473161Z",
          "Config": {
               "ExposedPorts": {
                    "80/tcp": {}
               },
               "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
               ],
               "Cmd": [
                    "/sbin/init"
               ],
               "Labels": {
                    "containers.bootc": "1",
                    "io.buildah.version": "1.35.0",
                    "org.opencontainers.image.version": "stream9.20240402.0",
                    "ostree.bootable": "true",
                    "ostree.commit": "6e1a3dd7fede025ee449c82bfc94f562477ce98d2d0c9a81ed1c93c0dcbd04cd",
                    "ostree.final-diffid": "sha256:12787d84fa137cd5649a9005efe98ec9d05ea46245fdc50aecb7dd007f2035b1",
                    "ostree.linux": "5.14.0-432.el9.aarch64",
                    "redhat.compose-id": "CentOS-Stream-9-20240304.d.0",
                    "redhat.id": "centos",
                    "redhat.version-id": "9",
                    "rpmostree.inputhash": "f5175ad2e1cb7b38eaea41a410559d28f94823aca74861b7d388c14eb92a1977"
               },
               "StopSignal": "SIGRTMIN+3"
          },
          "Version": "",
          "Author": "",
          "Architecture": "arm64",
          "Os": "linux",
          "Size": 1644818604,
          "VirtualSize": 1644818604,
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/var/lib/containers/storage/overlay/7fc8dd2514846362aa2466266378abb63a9d187e05ecbd7eb4f90c5878f9fe5f/diff:/var/lib/containers/storage/overlay/aabfc2393939bd1098a7edce4da3ac53d8d0c56ef52ac61f719ba8bdf92edd48/diff:/var/lib/containers/storage/overlay/09d7b380212847617b9056dc02187ac0589446aeae2032faadb54876ca1b440e/diff:/var/lib/containers/storage/overlay/82cf30724b9d2c2741df24ad0b166a71cda24b67304c2f0c1f4e866e639f5e69/diff:/var/lib/containers/storage/overlay/a879d700586d66e3c14a4238663bf8cc12d4bc923b516bb2686e81c2a4f4eb01/diff:/var/lib/containers/storage/overlay/0ba0a6907647821bbe7e06916ee6ccd2adec85ff2370a5db89ab1bce809daf30/diff:/var/lib/containers/storage/overlay/7fee544a02a460fe3af198fd9fa4fe33a2c7fa59e5d61678505d8700f2bf5eb2/diff:/var/lib/containers/storage/overlay/7074b51a2ef9cf9bd3781e2d7dc0e48513c00389d9230e0a87a21a791be66433/diff:/var/lib/containers/storage/overlay/ea9c87d374c4b61f2a418f262a6b7f88f239a0f507f3508e48c5950c0963ab15/diff:/var/lib/containers/storage/overlay/8e6c1f96d9a9eb40ed59e99f78bb2fce5d8d78fdfc860fadf209541b98977433/diff:/var/lib/containers/storage/overlay/73cc2db7dd00bdb2ccebfcc59ad1b7c627661ed741387c81bb99801c959854ad/diff:/var/lib/containers/storage/overlay/b42f347995c1681bc7e3be3309186599c1b91edf32d43e8c063f5db40890f0bf/diff:/var/lib/containers/storage/overlay/ee3206d6ff1299f316528268bec6464fea9f784a11d77de2afceaf56ad5fd7ea/diff:/var/lib/containers/storage/overlay/e3d4088d45fc43ff6afbf4c8bc838a5fa4c93eb88ee33ab6e27d1efc987bda57/diff:/var/lib/containers/storage/overlay/da3452a40e881f4949a71453c51c8560747cb21ebe08ccdf30188fb9859ec4f3/diff:/var/lib/containers/storage/overlay/aedd87d49940041fc43d6f3d1a84a1976111459b261087466d7bcf4060fcc8e0/diff:/var/lib/containers/storage/overlay/2391b36df63c73ccaa870f230d6db52098b7fcdd160d760582bd6db9b82198bf/diff:/var/lib/containers/storage/overlay/3204712b1472d0dc05bfbfc53653f21ef6f205b23f484bbbd834b410e75f999d/diff:/var/lib/containers/storage/overlay/1de94245110cbabc8ff741ddd2d72e70e62a2b86cc14095553f22650813946d5/diff:/var/lib/containers/storage/overlay/77bfad060d0bb21d5f2d15ef0c55513e12c9011a69ca06e79d2c81414f361fb8/diff:/var/lib/containers/storage/overlay/030a2cbc76333b2c4e89798f6093519b9ab0e174d6b172eb5456fbdd775b6f00/diff:/var/lib/containers/storage/overlay/089ed597e40b2a636b3e137bd43206d365f76318966ccf4e733a13becf35c169/diff:/var/lib/containers/storage/overlay/635ecbcb29c695ee7d10d3abbc41531550056875233e781402b88be3a716d210/diff:/var/lib/containers/storage/overlay/d69eae3d354a3d80c46b0ff627873c84184add997c01bb7075db4149f012f986/diff:/var/lib/containers/storage/overlay/a02ff525f5927af9f8b6d64d4a9cecaac1ea852ff531455817123b989b47477e/diff:/var/lib/containers/storage/overlay/c052a7983165baaa7fd9cbde8e02f043b10cb105c67284a7f8e054d03ac838ea/diff:/var/lib/containers/storage/overlay/19f38c02ef265556233f0a632abb5243afe6743d2bd1a3a9cbf05c6059b542a8/diff:/var/lib/containers/storage/overlay/4c269bd56cb3f3e6055ef4c2d295474c8d10df4f8a79cfb2062dc7937e6860a5/diff:/var/lib/containers/storage/overlay/79d122f7b07f777d9ccd72ba7b65a64c2e42a69542dea5e254a583755172b972/diff:/var/lib/containers/storage/overlay/ed0ff037068421a9f5a5de7cf7aa48f1448bedf75f171e833de1534a86a1ec54/diff:/var/lib/containers/storage/overlay/0ef8fb00bc7373eeef81831806265cc0b7aed62c00217403292f545e3caf7b2d/diff:/var/lib/containers/storage/overlay/fb0f6a70e60b8f60de64725e89329e5624f064ab13e3d4e1b392a736329ab2d8/diff:/var/lib/containers/storage/overlay/2ffcff9b511b3b18971bd011e862ebef701616496c0a71e52c2314ecb5484d77/diff:/var/lib/containers/storage/overlay/d8c8599e4cbd75f8d96dec78497e21be315a0c786a18a0f5214855f6d634a688/diff:/var/lib/containers/storage/overlay/493f184e773535150ddfa7df13143fe8dfa075bbb7d75ecaf6bf56bf0240e608/diff:/var/lib/containers/storage/overlay/5c32bc81cfaf26ad7062002efb8ee93892954eba99f7384febee5249a660d1a6/diff:/var/lib/containers/storage/overlay/572b1ceb586356352eeb4243f80ef0d2d0953d069b2315327ff663e41e6bdbff/diff:/var/lib/containers/storage/overlay/297e765c16e411af97ea014d56911af0067f31e2522d3a2f50d23eab789b5634/diff:/var/lib/containers/storage/overlay/2deadb9d2c687a157d1e71a342c564bfa03f3440c2232ecbc11ddbceae35012b/diff:/var/lib/containers/storage/overlay/68f87f75c4f9bec1c48e35451da2c17a093475f998b3ba110ad21f55c332f47c/diff:/var/lib/containers/storage/overlay/5b00e5ffcd990671c27fb3054e5f5316286cc36a181d1b84407e9be3c80821a5/diff:/var/lib/containers/storage/overlay/ed096cfc47d58f71dfb4f00e8c3913895b34bbaa2d052e53fe0c1ffa0df40c4c/diff:/var/lib/containers/storage/overlay/d935a94fd0e731fcf3f2ecc5154cbe20a8061d379c4619a843763ccd7be5463f/diff:/var/lib/containers/storage/overlay/83a13f72b34838b8451be805bac8ecf083e436e327bee6de920c8c2814dcc728/diff:/var/lib/containers/storage/overlay/9d96068cfa20e015ea1dce90bed81e5103e86d47b58e3cedfa1579582bd3d468/diff:/var/lib/containers/storage/overlay/8c631cd0a1f40979be56b28e692964803e1cef06e1c62baee0cbe5a0cd6d436b/diff:/var/lib/containers/storage/overlay/d115ca19e8fbf3d0b908631ca680b1131a5af69d05c742d75bc7e144f817feec/diff:/var/lib/containers/storage/overlay/df5ba027f59e915de5834ddb0062c4a44e0b48270e8b4b69f06f3ee4dfb1c9c1/diff:/var/lib/containers/storage/overlay/454eb548275cf87e41b40edfb97282e1974bd63c511f7009b9544d354f1187fe/diff:/var/lib/containers/storage/overlay/23df818095d871d59cac2e1caae4e79ce7ab03f931f99ec1a4958334516e9b19/diff:/var/lib/containers/storage/overlay/f157b01c7227e95d1d6de510ef7158fa412b68d1f08a12bd10d2d131405a5157/diff:/var/lib/containers/storage/overlay/b6bc080f66cc8a0457a34cd3a8bc84865a6d60d8daa4de6d782bea5372cace7b/diff:/var/lib/containers/storage/overlay/6ec384b1c478dfd3fd7d3b67cde66935bc1158932ac3f5c9890eb9fae4220e74/diff:/var/lib/containers/storage/overlay/45422b35e81fa9142055c265ab8f54b258c3d6b58f4ef975cc72903c13aef50a/diff:/var/lib/containers/storage/overlay/8b6ca5f9d243181f903139dc1583bd0525c54281f4e40c9db1d7f7ebfd317f9d/diff:/var/lib/containers/storage/overlay/ea0e817743d90a080a544b8d3d6320c5477fa91e2c24305f1800d6cc4166bb0a/diff:/var/lib/containers/storage/overlay/0ce559398de36cff7ad296f4a227dfce2202b63280203956b8f9dbccd883a6af/diff:/var/lib/containers/storage/overlay/7910a0fcc9a0096a773a1b5996ab64c54c83ef348d7aecff6677cb60427f6e6c/diff:/var/lib/containers/storage/overlay/b8f1d17d5e1d22471d995efc845123c7af290794c7bd66283003c89978f7e653/diff:/var/lib/containers/storage/overlay/26df861c92e509c0f06385f594ec8cd1de4bc8903a6a16cbdb2896c1c2072ac0/diff:/var/lib/containers/storage/overlay/a98043ad721a6be72f3cdcaa17ccb0bd503cdaffdbd6d2aec12a25c29a7c5838/diff:/var/lib/containers/storage/overlay/1872c90b225525decc16c522186800b1888b6b8e48c7ac342d7b7574db550e72/diff:/var/lib/containers/storage/overlay/96b0c247c6ea65198469b9b201732b85e143837f45b9a97ea7349b2e8ba6dd33/diff:/var/lib/containers/storage/overlay/b9dad65bdb4ec9106e9f4366cb120433c224cf5a2ea0e9f06611e47367c2cb0f/diff:/var/lib/containers/storage/overlay/101ce5b06ebdd494fb3bdf72e76f53ab7cb4c7ad5b53899a294aaef84bbd4279/diff:/var/lib/containers/storage/overlay/61960520aa5aeee0176e4df630dd89f4f81fb0ceff6690cbfb351a9859e0de44/diff:/var/lib/containers/storage/overlay/0c83d16e4e9836210a493cba3cd69c56b2ca6104514873ee15cd88a63218f13c/diff:/var/lib/containers/storage/overlay/560ec81bc83e5f6b9194ca7c887d52b4894be5032aec859388e54c4528e26884/diff:/var/lib/containers/storage/overlay/805949604c8cb5cb74b24cc6720ad769d994d174e213f0c0a4a0c910e19f9f58/diff",
                    "UpperDir": "/var/lib/containers/storage/overlay/74754964e0e9d88924adf8900067e9209491cd06c10937616e8bd489329150d8/diff",
                    "WorkDir": "/var/lib/containers/storage/overlay/74754964e0e9d88924adf8900067e9209491cd06c10937616e8bd489329150d8/work"
               }
          },
          "RootFS": {
               "Type": "layers",
               "Layers": [
                    "sha256:805949604c8cb5cb74b24cc6720ad769d994d174e213f0c0a4a0c910e19f9f58",
                    "sha256:88aab01220f9684503fb67e4ac6c7548d5503bec19e1b9935e04f86119f72f5c",
                    "sha256:1398857a59385bf21b05036c99b41b3ffa2ccf989a99dd66092c8d17d927d47c",
                    "sha256:ecabd238e09c883475a9b6b2b76a4e40dc6b734575a27223846fd4667ab7a42f",
                    "sha256:bc6a2ddc91e10b1a426be528ef61ccab8eaddd5fbc7f294fdfc315c0439c3df8",
                    "sha256:159348fa9cfbb75c5cb57e9fac24b9f28477412149e901bdadb909bfaeb84dad",
                    "sha256:ea21f664c19ce57ea2cbe3b21be9a5c4069c096e8f01afaa4fb0a14fed6918ee",
                    "sha256:e3b6dcdf32c3f7ca77eeff8f174f3b59169b5a583cb4a1831eef699cc6609e3e",
                    "sha256:ed6e50b45a9355d57f5593c64d35b4e8eb7b953bc402815a201cb4986a2f7d03",
                    "sha256:5db477f4687d9f8e0b3c733273e95c5d2c0d216d0a8932ff3024c2231061b586",
                    "sha256:d14dafd0d84f06d1724f6fa5fcb78c438b53ca1e605dc3668779f5a39347e16d",
                    "sha256:19e94fc9f561b63f00bc4f29cae6f3d66dfd17f88e2cc453dd7c95122dac2c97",
                    "sha256:d49f1af481997d13dc624f2a27605cf272f8385f9c4a29da03297800f55c8a2f",
                    "sha256:5b8cb2c5a76d77f168728d7ec9e8140d969299446b088fd76386277d54b7622e",
                    "sha256:1e4036afc4e15b9f799bcdf6157fdf1e788dae66c64ef6f46b411094ffc8ed25",
                    "sha256:ff18962ccfa3557da881ec43f19936d9add6053fbaea093bf549fce38e4c82fd",
                    "sha256:8264b77d5359dc3f95123f650aefa1d7b61f604ed879ec4fe2742fd9ee991642",
                    "sha256:28adee727adc25437e2497271a281dfa7395894839d7cbe42ad10b6a96c839e9",
                    "sha256:98ca63d3e1db3121180b82e806387a7337da2d445d07d315cb2fcfb525dd7247",
                    "sha256:312bd44fbfb86e4a31816751d39104a1bdd0e1ffed486ed5df080bcf92d99885",
                    "sha256:84f756f2a76cba323a1bf8d06a65e641d9cf9b3946a1cf99e2301d06b2fb0bb8",
                    "sha256:03f8187d462decd660e9d518a136c79c867bae006e6d4ad0e29cd476b7f81f5d",
                    "sha256:b4d285fa41e72c1c08d693065fddaabeabb529893d391c1a68f306238bcaf899",
                    "sha256:6afbc9f1ff554f2e93f9da5adfcf78c19079797baf26704239c9671dbdb06843",
                    "sha256:d85d11b65e436925fb2ae85483fe3d9fe6a26d2df1253fc82d2b88e1186e61b1",
                    "sha256:936b50b851a1abbd62542c1fca6b841243dd2a3ff4913311fcfe0046149e52e7",
                    "sha256:4a90b950ef7b87cf74480bb547118e60c67f11a0be6f1ef5d6b7c5cccb628ad2",
                    "sha256:d62c908cc2e2e6e131f07330f2a20aa290f9e929b5d9a59bec88a900e36ce47b",
                    "sha256:0cd464dec7d8ce1e3a829526f61c64247550a0709f79006f46d19145697e1e21",
                    "sha256:b2cd4d821498ffedc06b956d1b07f53ff8d7585816aac0bf755de926c90a5e78",
                    "sha256:47236f97ecf75b6666c2f9f981f5d8ba04210117424e90d739fad05382d13789",
                    "sha256:fafcf64fcc84bf1ba4177396edfd6022ed4be63044e4ad617b985a2bc769efed",
                    "sha256:5d6a623c38136b8dfec219954165cfca204c830265c75e2adc85155dfef21829",
                    "sha256:20595d276e7c26092001d0172a6b8fd106e463be83ddc43ee772cf99c98e1689",
                    "sha256:2972ed6b11ebeb23fc8caa2eef95f49f34ec577ade17b8709388d6124598822a",
                    "sha256:50b1ed09271c82cd00f37c92243fc0dc28d222731c8f94af7ec8f9c5b8b5ce84",
                    "sha256:83e6ac618a47cd5dde370b59c684af7e878139b301c5c32f3e38d141cfde34f5",
                    "sha256:fcc19d0ee42df736a635e7a88bc21f1804f3f233ab8a0145681b8665e1fdc49f",
                    "sha256:d3eb22aa466d90069d019016cae45d5fd05999b3f51abd81548283d2c3357ff4",
                    "sha256:3fbf1e0ab541ad69c8e7b77be27d45c24bf4234ecdbbe206b719c449e313ed2e",
                    "sha256:be537f1055b6e6ffb0e83e5c231bab666a328efeffb014badc248778662746e6",
                    "sha256:9641efc7a7c78c16ce06c366c28b12b905467f803a3efa16935717c28b797310",
                    "sha256:d86b46bff3e9614d567fdeb80c66637ca931f051c3d57e15086e8fdb7b519271",
                    "sha256:ea7fa05cd3a55e0f0c4a665564300672d5cc47ede75d3326c4f31d0b354e5d27",
                    "sha256:ef9287c95def475b5f31c5790f1e24d0b83651a8843d3918feb4742ebf5e0b95",
                    "sha256:fd6c6acfc0a4b9d471a6eb426b250039bb15719728077bb3738e598ea3d2c91e",
                    "sha256:1b4b2c78c49c98ae115cdf7c3c4535ec13463d7cfa2159400a26a970ff2df86c",
                    "sha256:60c34157f07651d4e063c708b7326f0a0bb96db6cdf858667a687efc3125b38d",
                    "sha256:d9e013ae94cde55900cc4e5a7a45e7b241b9d989c202d84938c80848d233e6cc",
                    "sha256:3cab2654156eaf0b46610d3b3acbe0969c72ec2ab89e17e14987476a21b8585c",
                    "sha256:5e0b21051039c2e57a542424f39348315e902cdcaaa78f249157b4cd6b21e3a0",
                    "sha256:8d38ac40b9eaa36b646a131117f1991b3edc873125b2b28649c268ad798e1f4c",
                    "sha256:596a87c17c00dc94d6e5975a8f56345307b47b90edbbe9e3d747c508fe9b1d3c",
                    "sha256:c99557a4c9ff38114e7f993909c7d4a8d1b7c55cfcc3b784fa5b3e8e92daeae7",
                    "sha256:6cf37683867820281dc0de77921774d072064c67217070f2a0f8a2ab4ed68797",
                    "sha256:2f89708153501705a90ebb7cd7f086490bcf1f476ff2b78ab5b897495780ceee",
                    "sha256:7ba6701df7c2e6720c0e96d289289d82bb577a0d32b549a5390fdb5a0087cca6",
                    "sha256:67ac3e56d2a0828b1e568517f3f7140948d970425203c57b2d3051d27b2f68f0",
                    "sha256:c3f500141a25e664ac63c280e1a3eb97b823198471262c6f99a99745b22f4596",
                    "sha256:9089241606b121ed752d472255f619a27143e773a6b896fbfff6c7108aa3be08",
                    "sha256:e5957753dc01c18e1127a270baa527ca23b8ce902f8bced376785a4ddfd47809",
                    "sha256:0608674f733ef64e9c26fe8793ce5f48c836fd5a94f63341cd7dcd6947cabcc0",
                    "sha256:14b39f017de7d7393dbccd7b52782e69fae6f1df7018a63ea513af58ffbe079e",
                    "sha256:1f67296bb947333fad65730fe40d81336c65b3f1f6fd6afad5384240e3764e00",
                    "sha256:12787d84fa137cd5649a9005efe98ec9d05ea46245fdc50aecb7dd007f2035b1",
                    "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
                    "sha256:4f0d87b73a0d8f68830e269ddab370267fc0b2ff622190ea747d60dd8fe83ff2",
                    "sha256:8e5600affccb420b197addba59571377a986d2a1b4805c8512c93e83c26be1ab",
                    "sha256:a832e22f4db2cb8b17127751a998cac575f02f36bb96de51458c36c21289a67e",
                    "sha256:1e03cd1d0e0ae0c1e77828510693221ec2b9000bc587d0b98222d50906341e64"
               ]
          },
          "Labels": {
               "containers.bootc": "1",
               "io.buildah.version": "1.35.0",
               "org.opencontainers.image.version": "stream9.20240402.0",
               "ostree.bootable": "true",
               "ostree.commit": "6e1a3dd7fede025ee449c82bfc94f562477ce98d2d0c9a81ed1c93c0dcbd04cd",
               "ostree.final-diffid": "sha256:12787d84fa137cd5649a9005efe98ec9d05ea46245fdc50aecb7dd007f2035b1",
               "ostree.linux": "5.14.0-432.el9.aarch64",
               "redhat.compose-id": "CentOS-Stream-9-20240304.d.0",
               "redhat.id": "centos",
               "redhat.version-id": "9",
               "rpmostree.inputhash": "f5175ad2e1cb7b38eaea41a410559d28f94823aca74861b7d388c14eb92a1977"
          },
          "Annotations": {
               "org.opencontainers.image.base.digest": "sha256:9a8fd5a569825f474a1f6b28bc3067f5fa8098c125b36ad5cf20227e1897967c",
               "org.opencontainers.image.base.name": "quay.io/centos-bootc/centos-bootc-dev:stream9"
          },
          "ManifestType": "application/vnd.oci.image.manifest.v1+json",
          "User": "",
          "History": [
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "ostree export of commit 6e1a3dd7fede025ee449c82bfc94f562477ce98d2d0c9a81ed1c93c0dcbd04cd"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "linux-firmware-20240219-143.el9.noarch"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "podman-2:5.0.0-1.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "python3-libs-3.9.18-3.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "libicu-67.1-9.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "skopeo-2:1.14.3-0.1.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "polkit-libs-0.117-11.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "kernel-core-5.14.0-432.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "binutils-2.35.2-43.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "kernel-modules-core-5.14.0-432.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "samba-client-libs-4.19.4-104.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "glibc-gconv-extra-2.34-103.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "kernel-modules-5.14.0-432.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "netavark-2:1.10.3-1.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "systemd-252-32.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "util-linux-2.37.4-18.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "glib2-2.68.4-14.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "rpm-ostree-2024.4-2.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "systemd-udev-252-32.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "coreutils-common-8.32-35.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "NetworkManager-libnm-1:1.47.2-1.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "cracklib-dicts-2.9.6-27.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "gnupg2-2.3.3-4.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "coreutils-8.32-35.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "file-libs-5.39-16.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "bash-5.1.8-6.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "fwupd-1.9.13-2.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "grub2-tools-1:2.06-68.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "iptables-libs-1.8.10-2.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "sssd-common-2.9.4-2.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "NetworkManager-1:1.47.2-1.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "openssl-libs-1:3.0.7-27.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "glibc-2.34-103.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "gettext-0.21-8.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "sg3_utils-1.47-9.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "bootc-0.1.7-1.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "grub2-common-1:2.06-68.el9.noarch"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "104 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "nvme-cli-2.6-5.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "systemd-pam-252-32.el9.aarch64 and crun-1.14.4-1.el9.aarch64 and NetworkManager-cloud-setup-1:1.47.2-1.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "8 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "9 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "11 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "11 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "11 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "samba-common-libs-4.19.4-104.el9.aarch64 and libsmbclient-4.19.4-104.el9.aarch64 and samba-common-4.19.4-104.el9.noarch"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "11 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "11 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "11 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "sssd-client-2.9.4-2.el9.aarch64"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "7 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "11 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "11 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "11 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "11 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "11 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "22 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "22 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "12 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "22 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "22 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "22 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "6 components"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "rpmostree-unpackaged-content and initramfs (kernel 5.14.0-432.el9.aarch64)"
               },
               {
                    "created": "2024-04-02T19:30:20Z",
                    "created_by": "Reserved for new packages"
               },
               {
                    "created": "2024-04-02T19:33:09.601690127Z",
                    "created_by": "/bin/sh",
                    "comment": "FROM quay.io/redhat-user-workloads/centos-bootc-tenant/centos-bootc/centos-bootc:on-pr-19ead2b16af28551532f54343208f9da3d29d4d4-arm64"
               },
               {
                    "created": "2024-04-03T20:38:48.297321639Z",
                    "created_by": "/bin/sh -c #(nop) COPY dir:445c8168edb33dfeb0b9ae5972e5649699cf93559b3342854e6af466659223c4 in usr ",
                    "empty_layer": true
               },
               {
                    "created": "2024-04-03T20:38:48.431057729Z",
                    "created_by": "/bin/sh -c #(nop) COPY multi:eda71433ead939c77e0327fd58dbd240f09214383065b444d817e92e0d14d6f2 in /etc/yum.repos.d/ ",
                    "empty_layer": true
               },
               {
                    "created": "2024-04-03T20:39:18.478935198Z",
                    "created_by": "/bin/sh -c dnf --disablerepo='*' --enablerepo=copr-coreos-continuous --enablerepo=copr-rhcontainerbot-bootc -y distro-sync &&     dnf clean all",
                    "empty_layer": true
               },
               {
                    "created": "2024-04-03T20:42:06.022113353Z",
                    "created_by": "/bin/sh -c set -x; kver=$(cd /usr/lib/modules && echo *); dracut -vf /usr/lib/modules/$kver/initramfs.img $kver",
                    "comment": "FROM quay.io/centos-bootc/centos-bootc:stream9"
               },
               {
                    "created": "2024-04-03T21:43:09.884335302Z",
                    "created_by": "/bin/sh -c dnf -y install httpd &&     systemctl enable httpd &&     mv /var/www /usr/share/www &&     sed -ie 's,/var/www,/usr/share/www,' /etc/httpd/conf/httpd.conf",
                    "comment": "FROM quay.io/centos-bootc/centos-bootc-dev:stream9"
               },
               {
                    "created": "2024-04-03T21:43:14.623712931Z",
                    "created_by": "/bin/sh -c rm /usr/share/httpd/noindex -rf",
                    "comment": "FROM a2c7324b4f32"
               },
               {
                    "created": "2024-04-03T21:43:19.018208562Z",
                    "created_by": "/bin/sh -c #(nop) COPY file:f6b79b470adcd6d977c80d5175bdf25d6eec249dfb80aede13a61d0957137506 in /usr/share/www/html ",
                    "comment": "FROM 9222e6d93e0e"
               },
               {
                    "created": "2024-04-03T21:43:19.085409369Z",
                    "created_by": "/bin/sh -c #(nop) EXPOSE 80",
                    "comment": "FROM 362fdd0b92b2",
                    "empty_layer": true
               }
          ],
          "NamesHistory": [
               "docker.io/library/869790e350a248b5355ffd56a70e371e751f615a3a2be49fe6b1cd79a21163e6-tmp:latest"
          ]
     }
]

[kingsleyzissou]

I'll try see if I can reproduce on my end.

Are you running on Mac or Linux?

[cdrage]

I'll try see if I can reproduce on my end.

Are you running on Mac or Linux?

mac M1

[kingsleyzissou]

I am seeing something slightly different but quite similar on my end. I suspect it has something to do with how we're resolving the containers. But will have to do some more digging on this.

[ondrejbudai]

@kingsleyzissou This was implemented, right?

[kingsleyzissou]

Yeah it got fixed with https://github.com/osbuild/images/pull/595 and #398