[OCI] Feature to upload container images

Vincinator commented 2 months ago

What would you like to be added

Container images must also be uploaded to the OCI Index via the OCI delivery gear.

Migrating to single package name is a separate task that we first need to verify if this is required.

Container images that will be placed to the OCI Index eventually:

package build container
kernel driver build container
base container for users (current gardenlinux container)
test container

Why is this needed

Sign referenced container digests
Single place to check for completeness
Signature verification of all components before creating open component model for Garden Linux release.
- containers are part of the release, they are re-build for each release
  - kernel driver build container requires the correct kernel headers used for the respective image (mapped via version)
  - package build container requires the correct apt sources configured for the respective image (mapped via version)
  - base container has the same apt packages (mapped via version)

Notes

We do not migrate the OCI delivery target to the package name gardenlinux just yet, and we will keep the current way of publishing container images in place until we want to migrate.

mxmxchere commented 5 days ago

My current idea is the following:

Publish container artifacts to S3 the same way we do it with non-container artifacts (https://github.com/gardenlinux/gardenlinux/pull/2466).

Index-entries that work with docker/podman currently look like this:

{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:64dc94494bb70f8549bff21b09dc19187fe586dee0af8833b26772848f00e4ff",
      "size": 406,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:bf8c5e03c093f3e3c3f7bc6b002fd9ce5700106e11c6fe633db74a47c0ac9e74",
      "size": 406,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    }
  ]
}

An example manifest looks like this:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:99f1d11911c94d959af0806e384eec7e22a37ce8c5843a2c5ab1aa979f2e1cc1",
    "size": 257
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:859caf64c62b76d4c6ccd9e12bb822846df5ed66a4e67b6cd36d9c8dfc66d1ee",
      "size": 34468075
    }
  ]
}

Seems like all relevant entries are packed in the .oci file...

paulphys commented 5 days ago

Container runtimes cannot deal wih additional custom layer media types, e.g. application/io.gardenlinux.release. We need two different manifests in order to support container build artifacts. One that is read by regular container runtimes like docker or podman (includes the actual file system layer) and another one for our custom garden linux tooling (includes logs, test files, ...). We have tried to combine regular layers and for example the .release file

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:99f1d11911c94d959af0806e384eec7e22a37ce8c5843a2c5ab1aa979f2e1cc1",
    "size": 257
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:859caf64c62b76d4c6ccd9e12bb822846df5ed66a4e67b6cd36d9c8dfc66d1ee",
      "size": 34468075
    },
    {
      "mediaType": "application/io.gardenlinux.release",
      "digest": "sha256:56bd950f9cd3f5dc7c4feaf08e50482e43acf3748c413f9c5ae5d7e8189dbdd9",
      "size": 436
    }
  ]
}

which results in the following error with podman

$ podman run -ti ghcr.io/paulphys/gl-oci:1683.0
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:56bd950f9cd3f5dc7c4feaf08e50482e43acf3748c413f9c5ae5d7e8189dbdd9"/""/"sha256:56bd950f9cd3f5dc7c4feaf08e50482e43acf3748c413f9c5ae5d7e8189dbdd9": unpacking failed (error: exit status 1; output: unexpected EOF)
exhausting input failed (error: unexpected EOF)

and with docker

$ docker run -ti ghcr.io/paulphys/gl-oci:1683.0
1683.0: Pulling from paulphys/gl-oci
docker: unsupported media type application/io.gardenlinux.release.

This only makes sense because layers are meant to be combined into one file tree which obviously breaks when you try to combine regular text files and a tarball.

mxmxchere commented 4 days ago

We have tooling in place that create the manifest+index for regular container runtimes (podman) and we have tooling that creates manifest+index for gardenlinux tools (this python tool).

I propose to upload the regular container image with podman and to upload the gardenlinux artifacts with python-gardenlinux-cli. I will check with @paulphys whether this is possible and if the order of execution is important (podman first or vice versa).

mxmxchere commented 4 days ago

Pushing the regular container layers with podman first and afterwards push the additional gardener artifacts with glcli works.

mxmxchere commented 4 days ago

Pushing the gardenlinux artifacts with glcli first and push the regular container artifacts with podman afterwords does not work. Podman replaces the complete index, so that the glcli index entries are gone after a podman push.

mxmxchere commented 3 days ago

So my proposal is to push the regular container images with podman first and then push our special gardenlinux manifests afterwards with glcli.

pnpavlov commented 5 hours ago

@5kt can you please review?

gardenlinux / python-gardenlinux-lib