search

lima-vm / alpine-lima

Create an alpine based ISO image for lima
Apache License 2.0
43 stars 28 forks source link

Create edition with nerdctl installed #34

Closed afbjorklund closed 2 years ago

afbjorklund commented 2 years ago

From https://github.com/lima-vm/lima/issues/489

https://github.com/lima-vm/lima/blob/master/examples/alpine.yaml


containerd:
  system: false
  user: false

The lima nerdctl provisioning doesn't work without systemd installed

One approach would be to create a separate ISO, with containerd installed.

And then add nerdctl and buildctl/buildkitd from the nerdctl-full tarball, like lima.

afbjorklund commented 2 years ago

Apparently it is not possible to use containerd from nerdctl, so use apk instead:

alpine-nerdctl-containerd

bin/containerd: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=adf6f2a2a47bd9f2ae142cf08ae6de787138d8bf, for GNU/Linux 3.2.0, stripped

bin/buildkitd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=X2aFQsdVAvRxWnSawzAH/1Hw0V0nkfjHgYf9usyh3/fFM6G3pQuF8I1Z801d1I/7c0ogZwxeBYT3Q_A6vyA, not stripped

afbjorklund commented 2 years ago

Here is a nice soundtrack, while building the image: https://www.youtube.com/watch?v=WKSFmP_16V0

See https://en.wikipedia.org/wiki/MC_Frontalot https://frontalot.bandcamp.com/track/nerdcore-rising

afbjorklund commented 2 years ago 
52M iso/alpine-lima-std-3.14.3-x86_64.iso
157M    iso/alpine-lima-nc-3.14.3-x86_64.iso
173M    iso/alpine-lima-rd-3.14.3-x86_64.iso

EDIT: Had missed the "containerd" package, oops. 100M And the "cni-plugins" package, too 135M

alpine-lima-nerdctl

afbjorklund commented 2 years ago

Contributing buildkit to alpine as an apk would be a nice future improvement.

https://git.alpinelinux.org/aports/tree/community/containerd?h=3.14-stable

I prepared the scripts for it, by making the boolean separate from the nerdctl:

LIMA_INSTALL_BUILDKIT

afbjorklund commented 2 years ago

Think I also need to install cni-plugins, because they're in the wrong path now.

genapkovl-lima.sh:        cp "${tmp}/nerdctl/libexec/cni/${plugin}" "${tmp}/usr/local/libexec/cni/${plugin}"
    "$pkgdir"/usr/bin/containerd config default | sed "s|/opt/cni/bin|/usr/libexec/cni|g" > "$pkgdir"/etc/containerd/config.toml

That is, the containerd package will need the cni-plugins package as well.

afbjorklund commented 2 years ago

@jandubois

I still want to make the "regular" containerd installation work (for "system"), but this "nc" ISO could be a workaround/optimization.

images:
- location: https://github.com/lima-vm/alpine-lima/releases/download/v0.2.2/alpine-lima-std-3.14.3-x86_64.iso
  arch: "x86_64"
  digest: "sha512:573964991fb135aac18e44c444c1c924cd6110d4c823e887451e134adbecd7abb98bb84d22872cec1c9ed5b2cd9d87f664817adb15938ca3a69a9a2c70d66837"
- location: https://github.com/lima-vm/alpine-lima/releases/download/v0.2.2/alpine-lima-std-3.14.3-aarch64.iso
  arch: "aarch64"
  digest: "sha512:6ff651023fbc4ec56c437124392d29cfa8eb8fe6d34c0e797b85b21734a6629aec38226c298f475b9ed63bef7664d49ba1bd5adc667c621efd7aa43e7020cc27"

containerd:
  system: true
  user: false
images:
- location: https://github.com/lima-vm/alpine-lima/releases/download/v0.2.x/alpine-lima-nc-3.14.3-x86_64.iso
  arch: "x86_64"
- location: https://github.com/lima-vm/alpine-lima/releases/download/v0.2.x/alpine-lima-nc-3.14.3-aarch64.iso
  arch: "aarch64"

containerd:
  system: false
  user: false

Will try to learn more about the Alpine distribution (interesting in itself), but no idea how long that will take (for apk etc)

afbjorklund commented 2 years ago

Note: I set buildkitd to use the containerd worker, so I don't think it needs the CNI plugins by itself (for OCI worker) ?

--oci-worker=false --containerd-worker=true

And I don't think nerdctl needs it either, but please correct me if I am wrong - then we could try some other approach.

The plan was to have containerd do everything.

EDIT: nerdctl uses nerdctl CNI* for nerdctl0 network.

* that is: /usr/local/libexec/cni

https://github.com/containerd/nerdctl/blob/master/docs/cni.md

But /etc/cni/net.d is empty, unless installing cri-containerd-cni.

This is so that it is possible to save the images directly to containerd, without having so save and load them first...

https://github.com/moby/buildkit#containerd-image-store

buildctl build ... --output type=image,name=localhost/myimage

ctr --namespace=buildkit images ls

afbjorklund commented 2 years ago

Note: the versions for docker and podman were much easier to do, because of proper apk packaging.

Once nerdctl (and containerd and buildkit) are also available in Alpine, this would become trivial too.

jandubois commented 2 years ago

Maybe we (you?) should submit the packages for upstream first, and we delay this PR until they are available?

Or do you need this image rather "sooner"?

afbjorklund commented 2 years ago

Or do you need this image rather "sooner"?

There is no rush, I think one can use the current (std) alpine image and do the install from there if in a hurry.

I'm not even sure if all these (well, three) images should be built by default, or if they are just "examples" ? Although it will be hard to link to them if they are not built and checksummed and provided automatically.

images:
- location: https://github.com/lima-vm/alpine-lima/releases/download/v0.2.2/alpine-lima-std-3.14.3-x86_64.iso
  arch: "x86_64"
  digest: "sha512:573964991fb135aac18e44c444c1c924cd6110d4c823e887451e134adbecd7abb98bb84d22872cec1c9ed5b2cd9d87f664817adb15938ca3a69a9a2c70d66837"
- location: https://github.com/lima-vm/alpine-lima/releases/download/v0.2.2/alpine-lima-std-3.14.3-aarch64.iso
  arch: "aarch64"
  digest: "sha512:6ff651023fbc4ec56c437124392d29cfa8eb8fe6d34c0e797b85b21734a6629aec38226c298f475b9ed63bef7664d49ba1bd5adc667c621efd7aa43e7020cc27"

It was just that the current "alpine.yaml" example doesn't work (for nerdctl), and that was a bit disappointing...

containerd:
  system: false
  user: false

And with both docker-machine and podman-machine being deprecated, there were no lightweight alternatives. One would have to go full Docker Desktop (LinuxKit) or Podman Desktop (CoreOS), to run them on Mac or Win.

But it (containerd:) can be fixed in limactl, as well.

These lima ISO images were just an "optimization".

52M iso/alpine-lima-std-3.14.3-x86_64.iso

91M iso/alpine-lima-pm-3.14.3-x86_64.iso
125M    iso/alpine-lima-de-3.14.3-x86_64.iso
157M    iso/alpine-lima-nc-3.14.3-x86_64.iso

Initially I even tried to build my own ISO with nerdctl, but I'm not sure that OS effort would be worthwhile anymore...

Would just end up re-creating alpine-lima and lima-init or something very similar, which seems like a waste of time.

afbjorklund commented 2 years ago

I was looking at nerdctl vs. the other container runtimes, and wondering why running "ctr" failed but "nerdctl" worked.

Then I remembered that Rancher Desktop runs nerdctl as set-uid, should make this optional so that it matches the apk.

i.e. when install from apk, you are supposed to run them with sudo nerdctl, sudo docker, sudo podman

You can set up local changes such as root suid or root groups, but that's not the way it comes out-of-the-box.

jandubois commented 2 years ago

Then I remembered that Rancher Desktop runs nerdctl as set-uid, should make this optional so that it matches the apk.

This doesn't sound right; it runs it with sudo:

LIMA_HOME="$HOME/Library/Application Support/rancher-desktop/lima" "${scriptdir}/../lima/bin/limactl" shell 0 sudo --preserve-env=CONTAINERD_ADDRESS nerdctl "$@"
jandubois commented 2 years ago

I see now that we set the binary setuid, but I can't remember why. It looks like it is not necessary, in which case I would rather drop it than make it conditional, but I would have to test first.

afbjorklund commented 2 years ago

If you want it drop-in replacement for docker, then I guess users would not expect to run it with sudo ?

Assuming that they added themselves to the root-equivalent "docker" group earlier, to access the socket.

https://docs.docker.com/engine/install/linux-postinstall/

Some distributions (or people?) prefer the explicit sudo.

afbjorklund commented 2 years ago

Back to "normal" again.

lima-nc:~$ sudo nerdctl version
Client:
 Version:   v0.16.0
 Git commit:    0ddaffd2ce304dd917267470883898feff9463dc

Server:
 containerd:
  Version:  v1.5.8
  GitCommit:    1e5ef943eb76627a6d3b6de8cd1ef6537f393a71
lima-nc:~$ nerdctl version
WARN[0000] environment variable XDG_RUNTIME_DIR is not set, see https://rootlesscontaine.rs/getting-started/common/login/ 
WARN[0000] environment variable XDG_RUNTIME_DIR is not set, see https://rootlesscontaine.rs/getting-started/common/login/ 
FATA[0000] rootless containerd not running? (hint: use `containerd-rootless-setuptool.sh install` to start rootless containerd): environment variable XDG_RUNTIME_DIR is not set, see https://rootlesscontaine.rs/getting-started/common/login/ 
lima-nc:~$ sudo ctr version
Client:
  Version:  v1.5.8
  Revision: 1e5ef943eb76627a6d3b6de8cd1ef6537f393a71
  Go version: go1.16.10

Server:
  Version:  v1.5.8
  Revision: 1e5ef943eb76627a6d3b6de8cd1ef6537f393a71
  UUID: 1b318a9d-50f7-433d-abe2-36a8ecd0e92a
lima-nc:~$ ctr version
Client:
  Version:  v1.5.8
  Revision: 1e5ef943eb76627a6d3b6de8cd1ef6537f393a71
  Go version: go1.16.10

ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied"
afbjorklund commented 2 years ago

Let's do the apk packages first, and then try again with Alpine 3.15 or something.