Error loading OCI archive with multiple manifest descriptors

[TadCordle]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I want to be able to tag an oci archive multiple times (similar to the results of podman tag, but automatically for oci tars that are loaded into podman). To do this I duplicate the manifest descriptor in index.json and give it a different image ref name, but when I try to load the new oci archive into podman, I get an error.

Steps to reproduce the issue:

1. Run:

   podman pull gcr.io/distroless/java
   podman save -o distroless.tar --format oci-archive gcr.io/distroless/java
   mkdir distroless-oci
   tar -xvf distroless.tar -C distroless-oci

2. Duplicate the manifest descriptor in distroless-oci/index.json and modify the org.opencontainers.image.ref.name, so the contents are as follows:

{
  "schemaVersion":2,
  "manifests":[
    {
      "mediaType":"application/vnd.oci.image.manifest.v1+json",
      "digest":"sha256:3a5a334aff603a9f5914640802c4d909c9df285e504c05968d51105adfaa161c",
      "size":821,
      "annotations":{
        "org.opencontainers.image.ref.name":"gcr.io/distroless/java"
      }
    },
    {
      "mediaType":"application/vnd.oci.image.manifest.v1+json",
      "digest":"sha256:3a5a334aff603a9f5914640802c4d909c9df285e504c05968d51105adfaa161c",
      "size":821,
      "annotations":{
        "org.opencontainers.image.ref.name":"gcr.io/distroless/java:tag2"
      }
    }
  ]
}

3. Run: tar -czf new-distroless.tar distroless-oci/

4. Run either podman load < new-distroless.tar or podman pull oci-archive:new-distroless.tar

Describe the results you received:

Using podman load < new-distroless.tar:

  open /var/tmp/podman675604817/manifest.json: not a directory
Error: error pulling "": unable to pull dir:/var/tmp/podman675604817: unable to pull image: Error determining manifest MIME type for dir:/var/tmp/podman675604817: open /var/tmp/podman675604817/manifest.json: not a directory

Using podman pull oci-archive:new-distroless.tar:

Error: error pulling image "oci-archive:jib-image.tar": unable to pull oci-archive:jib-image.tar: error determining pull goal for image "oci-archive:jib-image.tar": error loading manifest for {"jib-image.tar" "/usr/local/google/home/tcordle/IdeaProjects/hellogradle/oci/jib-image.tar" ""}: error loading index: more than one image in oci, choose an image

Describe the results you expected:

podman images loads the oci archive successfully, and lists both gcr.io/distroless/java:latest and gcr.io/distroless/java:tag2.

Additional information you deem important (e.g. issue happens only occasionally):

I'm essentially trying to replicate what the RepoTags field in a docker tar manifest does when you docker load the tar. (https://github.com/opencontainers/image-spec/issues/796)

Output of podman version:

Version:            1.6.4-dev
RemoteAPI Version:  1
Go Version:         go1.11.6
Git Commit:         c673ff8cb63c71a39e1a1d561373af42bfea7f24
Built:              Wed Nov 20 11:16:05 2019
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: c673ff8cb63c71a39e1a1d561373af42bfea7f24
  go version: go1.11.6
  podman version: 1.6.4-dev
host:
  BuildahVersion: 1.11.5
  CgroupVersion: v1
  Conmon:
    package: Unknown
    path: /usr/local/libexec/podman/conmon
    version: 'conmon version 2.0.4-dev, commit: f6d23b51a3c56cccd4e13b4eea361e98e1855802'
  Distribution:
    distribution: debian
    version: rodete
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 89939
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 535966
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 15417282560
  MemTotal: 67534381056
  OCIRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
      spec: 1.0.1-dev
  SwapFree: 68631392256
  SwapTotal: 68631392256
  arch: amd64
  cpus: 12
  eventlogger: journald
  hostname: tcordle33.nyc.corp.google.com
  kernel: 4.19.67-2rodete2-amd64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: Unknown
    Version: |-
      slirp4netns version 0.4.2
      commit: 69153b0d1cb82216d6782179ff7c3c5e91c731a1
  uptime: 510h 47m 45.11s (Approximately 21.25 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  ConfigFile: /usr/local/google/home/tcordle/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: {}
  GraphRoot: /usr/local/google/home/tcordle/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 0
  RunRoot: /run/user/535966
  VolumePath: /usr/local/google/home/tcordle/.local/share/containers/storage/volumes

[mheon]

@vrothberg Does this fall under the same category as multiple images in the same tar with the Docker format? It's just tags, but it does seem to be multiple manifests

[vrothberg]

@vrothberg Does this fall under the same category as multiple images in the same tar with the Docker format? It's just tags, but it does seem to be multiple manifests

error loading index: more than one image in oci, choose an image

We're correctly detecting the index but we didn't specify which image(s) to select. @nalind is the expert of manifest lists and indexes, so I'll assign it to him.

[github-actions[bot]]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[nalind]

For docker-archive: sources, I think we should be able to get the list of images using the source's LoadTarManifest() method and then iterate over them. (It wouldn't be fast, and it would require some logic outside of the image library.) If I'm reading it right, we'd need to add something similar for oci-archive:'s LoadManifestDescriptor() to avoid an ErrMoreThanOneImage error, but then we could do the same for it.

[rhatdan]

@TadCordle Any chance you could work on this. We don't have anyone with the time to work on it right now.

[rhatdan]

@QiWang19 PTAL

[rhatdan]

@QiWang19 @vrothberg Do we still have this issues, with some of the fixes for load that have been happening?

[QiWang19]

Still an issue

[rhatdan]

Still an issue, that I look forward to fixing? :^)

[vrothberg]

Note that multi-image support is limited to docker-archive. oci-archive may be an entirely different beast. I've not looked into though.

[vrothberg]

I guess that's something we should discuss and size during planing with @mtrmac

[mtrmac]

Purely structurally, the OCI archive and docker-archive are pretty similar, so it’t very tempting to use a a docker-archive-like multi-reader/writer .

The way OCI can use an index to represent a multi-archive image (or IIRC something else, to an arbitrary depth of nesting in theory), and the way c/image actually implements multi-archive in OCI in practice, may interact/conflict with that, though, and I’m afraid I have never read the OCI multi-arch implementation (by @nalind) in full, so I’m not immediately sure about the impact.

I also suspect that the current OCI code could do something unexpected if there are two entries pointing at the same manifest; this was IIRC never a core concern during implementation/reviews in that core area, so it may well behave in surprising ways.

[chanseokoh]

Note that this issue has a much narrower scope than the multi-architecture support with multiple images. The feature is merely to give multiple tags to a single image.

And if you follow the issue on the OCI image spec repo, they do suggest to have multiple entries pointing to the same manifest to assign multiple tags.

[mtrmac]

It’s not really narrower: right now the top level blindly assumes that the archive contains a single image (or that the user has specified a single image out of the archive). We need to build an infrastructure to list all images and handle them appropriately, and the “list all images” part necessarily includes “detect multi-arch images and treat them appropriately”.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@QiWang19 @mtrmac Any movement on this?

[mtrmac]

@rhatdan See the PRs by @QiWang19 linked just above.

[rhatdan]

Well any movement on these PRs?

[QiWang19]

They are still under review and working in progress PRs.

[rhatdan]

@mtrmac Could you update the state on this one and the PRs?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[vrothberg]

Unassigned, @QiWang19 who's currently busy in CRI-O land. I also marked it as a feature since OCI archives are not documented to support multi images.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[vrothberg]

@umohnani8 is currently working on it

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@umohnani8 Ping.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@umohnani8 Progress?