search

lima-vm / lima

Linux virtual machines, with a focus on running containers
https://lima-vm.io/
Apache License 2.0
14.58k stars 571 forks source link

ORAS and bootable containers - downloading images from registry #2405

Open afbjorklund opened 4 weeks ago

afbjorklund commented 4 weeks ago

Description

There are some projects using "OCI Registry As Storage" (ORAS), to download their images.

They add a kernel to a container image, and thus making it into a virtual machine image and bootable.

One such Fedora/CentOS project is "bootc": https://docs.fedoraproject.org/en-US/bootc/

Unfortunately most images are read-only, which means that one needs to use an overlay or rpm-ostree.

The main difference for Lima, is that the actual image and digest is stored in a JSON manifest...

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 19253,
         "digest": "sha256:9be6d762b9cf0d18490eb004f29eb9bdb0ed2a73a4d89cc12bfad25790787e42",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 19289,
         "digest": "sha256:89fc46c0111a9af81db401a9514060779b413f978b4079125df8bc351bc813f1",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

The image itself is stored in a "layer", which means it needs oras rather than curl to download.

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "artifactType": "application/vnd.unknown.artifact.v1",
    "config": {
        "mediaType": "application/vnd.oci.empty.v1+json",
        "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
        "size": 2,
        "data": "e30="
    },
    "layers": [
        {
            "mediaType": "application/x-qemu-qcow2+zstd",
            "digest": "sha256:2331217ee81b4c3a5161a50a93ecee9f1a9f3f1f145c8c019a40b2a5f9029f64",
            "size": 770370809,
            "annotations": {
                "org.opencontainers.image.title": "fedora-bootc-cloud-eln.qcow2.zst"
            }
        }
    ],
    "annotations": {
        "org.opencontainers.image.created": "2024-01-31T12:57:14Z"
    }
}

To be able to separate these images from a normal URL, an oras:// scheme/prefix has been added.

oras pull ghcr.io/centos/fedora-bootc-cloud-disk:eln

Downloading 2331217ee81b fedora-bootc-cloud-eln.qcow2.zst
Downloaded  2331217ee81b fedora-bootc-cloud-eln.qcow2.zst
Pulled [registry] ghcr.io/centos/fedora-bootc-cloud-disk:eln
Digest: sha256:e9ddab1dd369f826eb0905c1fb460d683f57c410ff895d5c89245f4caaa705db

zstd -d fedora-bootc-cloud-eln.qcow2.zst

The base images are docker:// images, that needs to be converted using bootc-image-builder

FROM quay.io/fedora/fedora-bootc:40

FROM quay.io/centos-bootc/centos-bootc:stream9

REPOSITORY                          TAG       IMAGE ID       CREATED        SIZE
quay.io/centos-bootc/centos-bootc   stream9   595883768d0f   33 hours ago   1.34GB

Note: The "oras" scheme is also used by apptainer.org

One could look for quay.io as well, as a special case?

afbjorklund commented 4 weeks ago

Using mounts needs a special image (with sshfs), since 9p has been disabled in the kernel.

https://github.com/vrothberg/lima-fedora-bootc

One also needs to make sure to use an image with cloud-init included, and with ssh enabled

dnf install -y openssh-server cloud-init sshfs

As a workaround, it is possible to add the fuse-sshfspackage and reboot

https://docs.fedoraproject.org/en-US/bootc/dnf/#_using_dnf_at_runtime

afbjorklund commented 4 weeks ago

Note, as a special case Fedora CoreOS (and similar) make /usr/local writable with a symlink:

lrwxrwxrwx. 3 root root 15 Jan 31 13:55 /usr/local -> ../var/usrlocal

This enables the regular containerd (nerdctl-full) installation to work, even if /usr is read-only...

Alternatively, one could change the guestInstallPrefix to something else (and convince systemd)

lrwxrwxrwx.   3 root root    8 Jan 31 13:55 home -> var/home
lrwxrwxrwx.   3 root root    7 Jan 31 13:55 mnt -> var/mnt
lrwxrwxrwx.   3 root root    7 Jan 31 13:55 opt -> var/opt
lrwxrwxrwx.   3 root root   12 Jan 31 13:55 root -> var/roothome
afbjorklund commented 4 weeks ago

The biggest downside is the enormous size and the hard-to-use, so in that sense it's like CoreOS et al...

But one can use ORAS for the storage, without switching over to the rest of the "bootable containers"?

Another related topic would be to support IPFS for storing images, and download them with ipfs get