identifier is not an image, when push the iamge.

fanux commented 2 years ago

I want to use storage interface to create an image, but when push the image it failed, my code:

func wrongManifestDigest(b []byte) (digest.Digest, error) {
    return digest.Canonical.FromBytes(b), nil
}

// fake manifest
var manifest = `{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1457,
    "digest": "sha256:24d4f50961544fdb669075c442509f194bdc4c0e344bde06e35dbd55af842a32"
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.descriptor.v1+json",
      "size": 772812,
      "digest": "sha256:50e8d59317eb665383b2ef4d9434aeaa394dcd6f54b96bb7810fdde583e9c2d1"
    }
  ]
}`

func CreateImages(images []string, target string) error {
    options, err := storage.DefaultStoreOptions(unshare.IsRootless(), unshare.GetRootlessUID())
    store, err := storage.GetStore(options)

        // create layer and Set layer big data
    _, err = store.CreateLayer("foo", "", nil, "", false, nil)
    b := bytes.NewBuffer([]byte(`{"key":"value"}`))
    store.SetLayerBigData("foo", "key", b)

        // create image
    imageOptions := &storage.ImageOptions{
        Digest: digest.Digest("sha256:24d4f50961544fdb669075c442509f194bdc4c0e344bde06e35dbd55af842a32"),
    }
    im, err := store.CreateImage("24d4f50961544fdb669075c442509f194bdc4c0e344bde06e35dbd55af842a32", []string{"docker.io/fanux/merge6"}, "foo", `{"signatures-sizes":{"sha256:24d4f50961544fdb669075c442509f194bdc4c0e344bde06e35dbd55af842a32":[]}}`, imageOptions)

        // set image manifest bigdata
    store.SetImageBigData("24d4f50961544fdb669075c442509f194bdc4c0e344bde06e35dbd55af842a32", "manifest", []byte(manifest), wrongManifestDigest)

    return nil
}

Then the layer and image created success:

[root@iZ2ze4vo1tjx7vf1b0ab2tZ sealer]# tree /var/lib/containers/
/var/lib/containers/
└── storage
    ├── mounts
    ├── overlay
    │   ├── foo
    │   │   ├── diff
    │   │   ├── empty
    │   │   ├── link
    │   │   ├── merged
    │   │   └── work
    │   └── l
    │       └── G5FNAUEYDBYKH6C35KI4JCBCNW -> ../foo/diff
    ├── overlay-containers
    │   └── containers.lock
    ├── overlay-images
    │   ├── 24d4f50961544fdb669075c442509f194bdc4c0e344bde06e35dbd55af842a32
    │   │   └── manifest
    │   ├── images.json
    │   └── images.lock
    ├── overlay-layers
    │   ├── foo
    │   │   └── key
    │   ├── layers.json
    │   └── layers.lock
    ├── storage.lock
    ├── tmp
    └── userns.lock

[root@iZ2ze4vo1tjx7vf1b0ab2tZ sealer]# buildah images
REPOSITORY               TAG      IMAGE ID       CREATED         SIZE
docker.io/fanux/merge6   <none>   24d4f5096154   9 minutes ago   491 B

But when I push the image:

[root@iZ2ze4vo1tjx7vf1b0ab2tZ sealer]# buildah push docker.io/fanux/merge6
error pushing image "docker.io/fanux/merge6" to "docker://docker.io/fanux/merge6": error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/fanux/merge6:latest" to "docker://fanux/merge6:latest": Error initializing source containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/fanux/merge6:latest: reference "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/fanux/merge6:latest" does not resolve to an image ID: identifier is not an image

It failed.

huanghantao commented 2 years ago

it seem that the layers.mediaType cannot use application/vnd.oci.descriptor.v1+json when use docker v2s2, have to use the application/vnd.docker.image.rootfs.diff.tar.gzip. Also, should calculate the diff of the layer. The code is as follows:

package main

import (
    "bytes"
    "io"
    "log"
    "os"

    "github.com/containers/storage"
    "github.com/containers/storage/pkg/reexec"
    "github.com/containers/storage/pkg/unshare"
    "github.com/opencontainers/go-digest"
)

func wrongManifestDigest(b []byte) (digest.Digest, error) {
    return digest.Canonical.FromBytes(b), nil
}

// fake manifest
var manifest = `{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1457,
    "digest": "sha256:be447ede66d9dcdac3ed6ee4f26b0e267b79e867ba277b0fb889f4399c3a11d3"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 772812,
      "digest": "sha256:faa6a7e1d2a5cf50be575c20d09864262ee59feb6d2d69cc0f49ee8482d6e59a"
    }
  ]
}`

func CreateImages(images []string, target string) error {
    options, err := storage.DefaultStoreOptions(unshare.IsRootless(), unshare.GetRootlessUID())
    if err != nil {
        return err
    }
    store, err := storage.GetStore(options)
    if err != nil {
        return err
    }

    // create layer and Set layer big data
    _, err = store.CreateLayer("faa6a7e1d2a5cf50be575c20d09864262ee59feb6d2d69cc0f49ee8482d6e59a", "", nil, "", false, nil)
    if err != nil {
        return err
    }
    b := bytes.NewBuffer([]byte(`{"key":"value"}`))
    store.SetLayerBigData("faa6a7e1d2a5cf50be575c20d09864262ee59feb6d2d69cc0f49ee8482d6e59a", "key", b)

    { // diff
        f, err := os.Create("faa6a7e1d2a5cf50be575c20d09864262ee59feb6d2d69cc0f49ee8482d6e59a.tar.gz")
        if err != nil {
            log.Fatal(err)
        }
        diffStream := f

        diffOptions := storage.DiffOptions{}
        reader, err := store.Diff("", "faa6a7e1d2a5cf50be575c20d09864262ee59feb6d2d69cc0f49ee8482d6e59a", &diffOptions)
        if err != nil {
            log.Fatal(err)
        }
        _, err = io.Copy(diffStream, reader)
        reader.Close()
        if err != nil {
            log.Fatal(err)
        }
    }

    { // apply diff
        f, err := os.Open("faa6a7e1d2a5cf50be575c20d09864262ee59feb6d2d69cc0f49ee8482d6e59a.tar.gz")
        if err != nil {
            log.Fatal(err)
        }
        diffStream := f
        defer f.Close()

        _, err = store.ApplyDiff("faa6a7e1d2a5cf50be575c20d09864262ee59feb6d2d69cc0f49ee8482d6e59a", diffStream)
        if err != nil {
            log.Fatal(err)
        }
    }

    // create image
    imageOptions := &storage.ImageOptions{
        Digest: digest.Digest("sha256:be447ede66d9dcdac3ed6ee4f26b0e267b79e867ba277b0fb889f4399c3a11d3"),
    }
    _, err = store.CreateImage("be447ede66d9dcdac3ed6ee4f26b0e267b79e867ba277b0fb889f4399c3a11d3", []string{"codinghuang/merge6"}, "faa6a7e1d2a5cf50be575c20d09864262ee59feb6d2d69cc0f49ee8482d6e59a", `{"signatures-sizes":{"sha256:be447ede66d9dcdac3ed6ee4f26b0e267b79e867ba277b0fb889f4399c3a11d3":[]}}`, imageOptions)
    if err != nil {
        return err
    }

    // set image manifest bigdata
    store.SetImageBigData("be447ede66d9dcdac3ed6ee4f26b0e267b79e867ba277b0fb889f4399c3a11d3", "manifest", []byte(manifest), wrongManifestDigest)

    return nil
}

func main() {
    reexec.Init()

    err := CreateImages([]string{}, "")
    if err != nil {
        log.Fatal(err)
    }
}

tree /var/lib/containers/
/var/lib/containers/
├── cache
│   └── blob-info-cache-v1.boltdb
└── storage
    ├── defaultNetworkBackend
    ├── mounts
    ├── overlay
    │   ├── backingFsBlockDev
    │   ├── faa6a7e1d2a5cf50be575c20d09864262ee59feb6d2d69cc0f49ee8482d6e59a
    │   │   ├── diff
    │   │   ├── empty
    │   │   ├── link
    │   │   ├── merged
    │   │   └── work
    │   └── l
    │       └── W45JQTJCYUVVWSZYNSI54A34D6 -> ../faa6a7e1d2a5cf50be575c20d09864262ee59feb6d2d69cc0f49ee8482d6e59a/diff
    ├── overlay-containers
    │   ├── containers.json
    │   └── containers.lock
    ├── overlay-images
    │   ├── be447ede66d9dcdac3ed6ee4f26b0e267b79e867ba277b0fb889f4399c3a11d3
    │   │   └── manifest
    │   ├── images.json
    │   └── images.lock
    ├── overlay-layers
    │   ├── faa6a7e1d2a5cf50be575c20d09864262ee59feb6d2d69cc0f49ee8482d6e59a
    │   │   └── key
    │   ├── faa6a7e1d2a5cf50be575c20d09864262ee59feb6d2d69cc0f49ee8482d6e59a.tar-split.gz
    │   ├── layers.json
    │   └── layers.lock
    ├── storage.lock
    ├── tmp
    └── userns.lock

17 directories, 15 files

However, when uploading the image, I encountered a new problem:

buildah push codinghuang/merge6
Getting image source signatures
Copying blob bd9ddc54bea9 skipped: already exists
Copying config be447ede66 [--------------------------------------] 0.0b / 1.4KiB
error pushing image "codinghuang/merge6" to "docker://codinghuang/merge6": reading config blob sha256:be447ede66d9dcdac3ed6ee4f26b0e267b79e867ba277b0fb889f4399c3a11d3: error locating item named "sha256:be447ede66d9dcdac3ed6ee4f26b0e267b79e867ba277b0fb889f4399c3a11d3" for image with ID "be447ede66d9dcdac3ed6ee4f26b0e267b79e867ba277b0fb889f4399c3a11d3" (consider removing the image to resolve the issue): file does not exist

rhatdan commented 2 years ago

@nalind @giuseppe @flouthoc PTAL

nalind commented 2 years ago

At the storage library level, names are only matched exactly, so in @fanux's example, and after "docker.io/fanux/merge6" is canonicalized to "docker.io/fanux/merge6:latest", there's no image record that matches it. This usually gets resolved when a higher-level library calls into containers/image to locate an image with a given name.

Both @fanux's and @huanghantao's examples attach manifests to an image record, and both manifests reference a config blob which is not being saved to the image, which triggers the error that @huanghantao's example runs into.

In either case, I'd recommend using APIs from buildah or podman, which take care of these and other details.

mtrmac commented 1 year ago

Closing because the provided code samples are not creating correct images, as described above. Please reopen, or file new bugs, if there are other concerns.

containers / storage

identifier is not an image, when push the iamge. #1202