search

lima-vm / lima

Linux virtual machines, with a focus on running containers
https://lima-vm.io/
Apache License 2.0
15.46k stars 608 forks source link

ORAS and bootable containers - downloading images from registry #2405

Open afbjorklund opened 5 months ago

afbjorklund commented 5 months ago

Description

There are some projects using "OCI Registry As Storage" (ORAS), to download their images.

They add a kernel to a container image, and thus making it into a virtual machine image and bootable.

One such Fedora/CentOS project is "bootc": https://docs.fedoraproject.org/en-US/bootc/

Unfortunately most images are read-only, which means that one needs to use an overlay or rpm-ostree.

The main difference for Lima, is that the actual image and digest is stored in a JSON manifest...

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 19253,
         "digest": "sha256:9be6d762b9cf0d18490eb004f29eb9bdb0ed2a73a4d89cc12bfad25790787e42",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 19289,
         "digest": "sha256:89fc46c0111a9af81db401a9514060779b413f978b4079125df8bc351bc813f1",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

The image itself is stored in a "layer", which means it needs oras rather than curl to download.

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "artifactType": "application/vnd.unknown.artifact.v1",
    "config": {
        "mediaType": "application/vnd.oci.empty.v1+json",
        "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
        "size": 2,
        "data": "e30="
    },
    "layers": [
        {
            "mediaType": "application/x-qemu-qcow2+zstd",
            "digest": "sha256:2331217ee81b4c3a5161a50a93ecee9f1a9f3f1f145c8c019a40b2a5f9029f64",
            "size": 770370809,
            "annotations": {
                "org.opencontainers.image.title": "fedora-bootc-cloud-eln.qcow2.zst"
            }
        }
    ],
    "annotations": {
        "org.opencontainers.image.created": "2024-01-31T12:57:14Z"
    }
}

To be able to separate these images from a normal URL, an oras:// scheme/prefix has been added.

oras pull ghcr.io/centos/fedora-bootc-cloud-disk:eln

Downloading 2331217ee81b fedora-bootc-cloud-eln.qcow2.zst
Downloaded  2331217ee81b fedora-bootc-cloud-eln.qcow2.zst
Pulled [registry] ghcr.io/centos/fedora-bootc-cloud-disk:eln
Digest: sha256:e9ddab1dd369f826eb0905c1fb460d683f57c410ff895d5c89245f4caaa705db

zstd -d fedora-bootc-cloud-eln.qcow2.zst

The base images are docker:// images, that needs to be converted using bootc-image-builder

FROM quay.io/fedora/fedora-bootc:40

FROM quay.io/centos-bootc/centos-bootc:stream9

REPOSITORY                          TAG       IMAGE ID       CREATED        SIZE
quay.io/centos-bootc/centos-bootc   stream9   595883768d0f   33 hours ago   1.34GB

Note: The "oras" scheme is also used by apptainer.org

One could look for quay.io as well, as a special case?

afbjorklund commented 5 months ago

Using mounts needs a special image (with sshfs), since 9p has been disabled in the kernel.

https://github.com/vrothberg/lima-fedora-bootc

One also needs to make sure to use an image with cloud-init included, and with ssh enabled

dnf install -y openssh-server cloud-init sshfs

As a workaround, it is possible to add the fuse-sshfspackage and reboot

https://docs.fedoraproject.org/en-US/bootc/dnf/#_using_dnf_at_runtime

afbjorklund commented 5 months ago

Note, as a special case Fedora CoreOS (and similar) make /usr/local writable with a symlink:

lrwxrwxrwx. 3 root root 15 Jan 31 13:55 /usr/local -> ../var/usrlocal

This enables the regular containerd (nerdctl-full) installation to work, even if /usr is read-only...

Alternatively, one could change the guestInstallPrefix to something else (and convince systemd)

lrwxrwxrwx.   3 root root    8 Jan 31 13:55 home -> var/home
lrwxrwxrwx.   3 root root    7 Jan 31 13:55 mnt -> var/mnt
lrwxrwxrwx.   3 root root    7 Jan 31 13:55 opt -> var/opt
lrwxrwxrwx.   3 root root   12 Jan 31 13:55 root -> var/roothome
afbjorklund commented 5 months ago

The biggest downside is the enormous size and the hard-to-use, so in that sense it's like CoreOS et al...

But one can use ORAS for the storage, without switching over to the rest of the "bootable containers"?

Another related topic would be to support IPFS for storing images, and download them with ipfs get

afbjorklund commented 1 month ago

It would be possible to only support the oras: url, and not the rest of the bloatable containers.

Some other projects are using oci: instead, so should probably adopt that (or rather support both)

Beyond the fancy specifications and the helpers (like oras), it is just json:

https://tech.michaelaltfield.net/2024/09/03/container-download-curl-wget/