RFC: support kuasar runtime with containerd

[xiaods]

as pure kubernetes distribution, k8e's feature is almost like k3s. we aimed to fast enterprise adoption with serverless technology.

• [x] k8e containerd support devmapper plugin
• [x] kuasar support
• [ ] Testing feature

Refs:

1. https://github.com/firecracker-microvm/firecracker
2. https://github.com/kuasar-io/kuasar

[xiaods]

use kata to quickstart use vmm on k8s.

[xiaods]

Proposed solution: provide another engine for K8E, Kata provides a one-click reference that can work directly with runc. And firecrack does not apply directly to K8S, this limits our design, so, after consideration, it is recommended to unify the management of VMM containers through Kata

[xiaods]

follow https://github.com/kata-containers/kata-containers/issues/2207 we need let k8e containerd support devmapper plugin.

[root@ip-172-31-28-183 ~]# k8e ctr plugins list
TYPE                                  ID                       PLATFORMS      STATUS    
io.containerd.content.v1              content                  -              ok        
io.containerd.snapshotter.v1          aufs                     linux/amd64    skip      
io.containerd.snapshotter.v1          btrfs                    linux/amd64    skip      
io.containerd.snapshotter.v1          devmapper                linux/amd64    error  

[xiaods]

1. 首先安装Device Mapper，使用命令yum install device-mapper-persistent-data
2. 配置Device Mapper，使用命令systemctl enable devicemapper
3. 打开/etc/lvm/lvm.conf文件，添加以下内容：

   devices {
   dir = "/dev/mapper"
   scan = [ "/dev/mapper" ]
   filter = [ "a|/dev/loop|", "a|/dev/dm-|", "r/.*/" ]
   }

4. 保存并关闭文件
5. 启动Device Mapper，使用命令systemctl start devicemapper
6. 下载containerd的devmapper plugin，使用命令wget https://github.com/containerd/devmapper/releases/download/v1.1.1/devmapper-1.1.1-linux-amd64.tar.gz
7. 解压文件，使用命令tar -xvf devmapper-1.1.1-linux-amd64.tar.gz
8. 将devmapper plugin 安装到containerd，使用命令containerd config default | sudo tee /etc/containerd/config.toml
9. 重启containerd，使用命令systemctl restart containerd
10. 验证devmapper plugin的安装，使用命令containerd plugin ls | grep devmapper

[xiaods]

在/var/lib/k8e/agent/etc/containerd/confg.toml.tmpl 中配置模板, 然后重启服务后k8e就会加载devmapper plugin snapshotter

[plugins."io.containerd.snapshotter.v1.devmapper"]
    root_path = "/var/lib/k8e/agent/containerd"
    pool_name = "devpool"
    base_image_size = "8192MB"
    discard_blocks = true

reference: https://github.com/containerd/containerd/blob/main/docs/snapshotters/devmapper.md

• sudo ctr plugin ls, make sure the devmapper plugin status is ok
• sudo dmsetup ls check devices exist.
• check /var/lib/k8e/agent/etc/containerd/confg.toml configuration is OK

[xiaods]

report issue on kata community: https://github.com/kata-containers/kata-containers/issues/6317

[xiaods]

report issue on kata community: kata-containers/kata-containers#6317

no response from kata community.

[xiaods]

又发现有的库来实现VMM libkrun is a dynamic library that allows programs to easily acquire the ability to run processes in a partially isolated environment using KVM Virtualization.

https://github.com/containers/libkrun

[xiaods]

libkrun的主要目标是在单个动态库中提供所有虚拟化过程隔离所需的功能（包括VMM、模拟设备和Guest内核等），以便可以轻松地在不同命名空间之间切换，并且具有最小可能的内存占用。

我们并不试图成为通用VMM，这使得我们能够减少一些负载，并且专注于此使用案例可以将对外部支持进程和shim的需求降至最低。

我们还正在开发实验性扩展Guest内核，以打破VM和容器之间的障碍，例如透明套接字模仿（Transparent Socket Impersonation），它提供了一定程度的网络命名空间透明度，并会有更多功能推出。

总而言之，libkrun仍处于早期开发阶段，尚未准备好投入生产。因此，如果您需要短期解决方案，则QEMU或Firecracker（与Kata或其他运行时组合）可能是您需要的。

[xiaods]

https://kuasar.io/docs/architecture/why-kuasar/ 今天看到这个，直接替换 runc，提供多 sandbox 方案。可以啊。

[liangyuanpeng]

kuasar 正在申请 cncf sandbox

[xiaods]

看着不错，期待 kuasar

https://github.com/kuasar-io/kuasar/issues/83

[xiaods]

k8e 替换了 containerd , PR https://github.com/xiaods/k8e/pull/331

[xiaods]

Expires, close it