RFE: create multi-arch manifests

[terinjokes]

I created a multi-image OCI path, and can see these have been properly merged in the index

$ skopeo copy docker-archive:./save1.tar.gz oci:manifest
$ skopeo copy docker-archive:./save2.tar.gz oci:manifest

# manifest/index.js
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:5cb5de57b9476b44186e34c5520ce550ee840b9d9bf4a2cf55c9f69a92a8bd63",
      "size": 1446
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:67a7232cc67fda26632946325ada75c35b60eca578c9cfcaca6e3d98a4094562",
      "size": 974
    }
  ]
}

I am, however, unable to push this to Docker Hub to use as a multi-arch image.

$ skopeo copy --all --format v2s2 oci:./manifest docker://terinjokes/test123:skopeo
FATA[0000] Error initializing source oci:./manifest:: more than one image in oci, choose an image

[terinjokes]

For completion, I also tried copying the different style of OCI list generated in containers/buildah#2858, but got the same error message.

[mtrmac]

Thanks for your report. The copy pipeline in skopeo copy copies a single (possibly multi-arch) image, so this seems to work as currently designed at a first glance. (That said, it should be possible to work with multi-arch images, so the Buildah issue might be a real bug.)

[mtrmac]

Looking further, if the destination is docker://…test123:skopeo, how would you expect the two images to be represented? That command just doesn’t make much sense to me.

You can copy one image at a time, currently only if they have different tags; https://github.com/containers/image/pull/1072 will add a way to reference unnamed images in an OCI index.

[terinjokes]

@mtrmac my goal is to make a multi-architecture image. I expect them to appear as such on Docker Hub. As far as I can tell, there is no way to create a proper multi-architecture image with skopeo, though the tooling gets very close.

[mtrmac]

Both Buildah and Podman have a manifest subcommand intended for creating such images, so it seems superfluous to provide that functionality in Skopeo.

[terinjokes]

Both of those require user namespace support, and a fully setup /etc/containers and storage engines. Those are very different requirements than Skopeo, and I should be able to create multi-arch manifests from existing images without them.

Skopeo already generates the OCI directory merged from multiple images. It just generates the wrong type of JSON manifest.

[darktempla]

I wanted to be able to copy an image from dockerhub and import it into my private repo including all multi-arch refs. Skopeo seemed like a logical choice.

However in alignment with this issue it seems skopeo (I am running latest version 1.2.0) cannot write/read dockerhub multi-arch manifests. I first started trying to just read the manifest but ran into the following (see Observations below).

Perhaps I am doing something wrong?

Would be great if skopeo did get multi-arch support with ARM based processors becoming more prominent now but more so in the future.

Observations

skopeo inspect docker://ubuntu:21.04
FATA[0003] Error parsing manifest for image: Error choosing image instance: no image found in manifest list for architecture amd64, variant "", OS darwin

However when you add the --raw command line switch it pulls the manifest. So seems to be skopeo cannot parse this format.

skopeo inspect --raw docker://ubuntu:21.04 | jq

Output:

{
  "manifests": [
    {
      "digest": "sha256:eb9086d472747453ad2d5cfa10f80986d9b0afb9ae9c4256fe2887b029566d06",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 943
    },
    {
      "digest": "sha256:017b74c5d97855021c7bde7e0d5ecd31bd78cad301dc7c701bb99ae2ea903857",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      },
      "size": 943
    },
    {
      "digest": "sha256:bb48336f1dd075aa11f9e819fbaa642208d7d92b7ebe38cb202b0187e1df8ed4",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm64",
        "os": "linux",
        "variant": "v8"
      },
      "size": 943
    },
    {
      "digest": "sha256:29c2f09290253a0883690761f411cbe5195cd65a4f23ff40bf66d7586d72ebb7",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux"
      },
      "size": 943
    },
    {
      "digest": "sha256:e8e0c3580fc5948141d8f60c062e0640d4c7e02d10877a19a433573555eda25b",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "s390x",
        "os": "linux"
      },
      "size": 943
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}

If you inspect an image that does not support multi-arch then the manifest returned is different in how it is presented.

skopeo inspect --raw docker://jenkins/jenkins:latest | jq

Output:

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 16488,
      "digest": "sha256:f98e5f96106f5484d49bd725e1bac1fa92974ec2688783b153ef815a33680f70"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 45380658,
         "digest": "sha256:3192219afd04f93d90f0af7f89cb527d1af2a16975ea391ea8517c602ad6ddb6"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 10797219,
         "digest": "sha256:17c160265e75550c2ed099aa7d3906b3fef0bf046a2aeead136f8e587a015159"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 4340216,
         "digest": "sha256:cc4fe40d0e618e3319afb689c3570bb87e8e8cf51bca080364d1552317bc66c2"
      },
     ... redacted for brevity
   ]
}

[terinjokes]

@darktempla It does know how to read the from the registry, but it resolves to a specific OS and architecture: by default your local one. Ubuntu doesn't have an amd64/darwin build, so there's no image that matches your local architecture. Try using --override-os and --override-arch.

I can't remember if skopeo-sync handles all architectures or not. You might want to give that a try.

[mtrmac]

@darktempla Yeah, you’re basically running into an artifact of skopeo inspect being created before multi-arch images, so it continues to report single-image data.

It’s certainly plausible to have a multi-arch inspect/format/… feature, but --raw has so far apparently been good enough that I am not aware of even a request for that, so far.

---

Returning to your original request to copy a multi-arch image, both skopeo copy and skopeo sync have an --all option.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rverma-dev]

Is there any resolution for this?

[mtrmac]

@rverma-nsl The current recommendation is to use {podman,buildah} manifest. It does have more system requirements than Skopeo, but seems to me that building multi-arch images usually immediately follows building single-arch images, so requiring tools that can build images is not usually too much to ask.

I can certainly imagine exceptions (and it would be interesting to hear about them), so this issue is not closed yet, but it’s not something we are too likely to work on soon.

(I’d certainly not want for Skopeo to have and maintain an independent reimplementation of the manifest building logic; the existing one should be shared, possibly after being made more general.)

[andrewshadura]

@mtrmac, not necessarily. It is quite typical to build images in GitLab CI using Kaniko, in a setup that cannot run docker or podman. Once images are built, there needs to be a way to assemble them into a multi-arch image, and requiring docker or podman is quite limiting.

[bastianbeier]

@mtrmac, not necessarily. It is quite typical to build images in GitLab CI using Kaniko, in a setup that cannot run docker or podman. Once images are built, there needs to be a way to assemble them into a multi-arch image, and requiring docker or podman is quite limiting.

Yeah we are facing the same issue right now. We would as well build in GitLab CI using Kaniko. Afterwards assemble them in a multi-arch image. And we do not want / are not able to provide extended permissions to use e.g. docker, buildah, podman.

[epDHowwD]

Same here. Our images are build on a specifically protected infrastructure and the resulting images end up in an intermediate registry. We now search for a valuable solution to generate a multi-arch image from arising separate images that are even build on physically separated machines. The general CI/CD infrastructure is using containers as they are perfectly suited for task isolation. This infrastructure does not allow privileged execution. Right now we use skopeo in this infrastructure to publish images from intermediate internal registry to various endpoints. It would be perfect if this step could also include to combine images from different architectures into a single multi-arch image using skopeo. Thus, please consider this valuable user story.

[zrzka]

I've got a similar use case. Two archives with x86_64 & aarch64 images. I can publish them via:

skopeo copy docker-archive:path-x86_64.tar.gz docker://index.docker.io/foo/bar:latest-x86_64
skopeo copy docker-archive:path-aarch64.tar.gz docker://index.docker.io/foo/bar:latest-aarch64

As a final step, one has to create manifest & push it:

export DOCKER_CLI_EXPERIMENTAL=enabled
docker manifest create --insecure latest --amend latest-x86_64 --amend latest-aarch64
docker manifest push --insecure -p latest

Would be nice if skopeo can do the final step too. docker requires Docker engine, which is not available in some environments (or is "hard" to set up, Docker inside Docker ...).

[epDHowwD]

Maybe https://github.com/estesp/manifest-tool#createpush is an option for this use case.

manifest-tool is a command line utility used to view or push multi-platform container image references located in an OCIv1 or Docker v2.2 compatible container registry. Manifest creation is also a use case on this tool.

[terinjokes]

You can also use buildah, as mentioned way up thread.

buildah manifest create example
buildah manifest add example docker-archive:./docker-amd64
buildah manifest add example docker-archive:./docker-arm64
buildah manifest push --all -f v2s2 example docker://terinjokes/example:latest

[zrzka]

Thanks, aware of buildah, manifest-tool, ... I'm not a big fan of yet another dependency. Quite a big one (buildah) for what I need. I've ended up with a couple of HEAD, GET requests followed by final PUT for the manifest list. It's not that hard to assemble a manifest list by myself. The Service I'm building is a publisher, which just consumes Docker image archives (built elsewhere, one archive per architecture) and distributes them to Docker Hub, AWS ECR, Google Cloud, ... as multi-arch images.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[andrewshadura]

A friendly reminder that ‘stale issues’ is a misnomer.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[andrewshadura]

Because it’s still an issue. Can somebody please disable this annoying bot?

[rhatdan]

This annoying bot is the only thing that keeps the issue in the foreground, without it, I doubt anyone will pay attention to the issue, it will get lost in the forest.

It is always best if you have an issue to step forward and open a PR to solve it.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[andrewshadura]

@mtrmac, any updates on this please?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[mccaig]

Adding a vote for this feature. Our use case: We run image builds on a restricted k8s cluster using kaniko, and push the resulting image archive to a registry using skopeo. We're now creating images for multiple architectures, so it would be great to be able to use skopeo to push a multi arch image to a registry. We cant use the buildah method as suggested by @terinjokes, as buildah manifest requires elevated permissions, which aren't an option for us.

[panpan0000]

vote +1

[d1nfinite]

vote +1

[huwcbjones]

We could also make use of this.

Currently we have a pipeline that builds the arch specific images, then the final job in the pipeline gathers the arch containers and creates the multi-arch manifest. We don't want to use buildx because it emulates the arch (we have arm64 hardware, why would we ever want to emulate). We don't want to have to maintain more deps that do way more than munging some manifests together

[terinjokes]

I tried buildah again but it still requires syscalls commonly blocked by seccomp filters, for what should be an entirely non-privileged operation. See https://github.com/containers/buildah/issues/1901#issuecomment-1674648806

[lysliu]

vote +1

[dekimsey]

I would very much like to see the containers tooling to support being able to read oci-archive tarballs with multiple images. I recognize this more an issues in the underlying libraries but issues tracking those have stalled out (containers/common#1178, containers/image#1381, containers/image#1116). I'm posting this here because it's active and it's not clear where best to follow-up on the issue (it's a cross-repository problem in containers project).

FWIW Docker v25.0.0 has made oci-archive the default output format. And as of v24, when an image is built with docker buildx build it will be produced with multiple images in it's manifest.

I ran into this because I was trying to use the containers libraries to inspect images built with the following command:

docker buildx build --output=type=docker,dest=demo.tar -f Dockerfile -t localhost/demo:purple -t localhost/demo:red .
skopeo inspect --raw oci-archive:demo.tar
FATA[0000] Error parsing image name "oci-archive:demo.tar": more than one image in oci, choose an image 

But the resulting archive's index.json's has multiple layers because multiple tags were specified during build. Given tar -xf ./demo.tar -O index.json | jq .:

{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "annotations": {
        "io.containerd.image.name": "localhost/demo:purple",
        "org.opencontainers.image.created": "2024-02-23T15:35:04Z",
        "org.opencontainers.image.ref.name": "purple"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "annotations": {
        "io.containerd.image.name": "localhost/demo:red",
        "org.opencontainers.image.created": "2024-02-23T15:35:04Z",
        "org.opencontainers.image.ref.name": "red"
      }
    }
  ]
}

Removing the second tag does allow this to work, but at least in our build systems we use docker buildx build with many tags. While I can process the archive as docker-archive: instead, it drops a whole lot of information I need.

[mtrmac]

I would very much like to see the containers tooling to support being able to read oci-archive tarballs with multiple images.

Please file unrelated requests as separate issues. (I’m not responding to the substance here.)