Not able to launch multiple containers with hugepages with CLH

[GabyCT]

I am using the script of https://github.com/kata-containers/tests/blob/main/metrics/storage/webtooling.sh using CLH+containerd and I am enabling hugepages for CLH at the configuration.toml, I am also allocating the hugepages for the host and inside the containers, for the host I am using $ echo 20480 | sudo tee /proc/sys/vm/nr_hugepages and if I do

$ cat /proc/meminfo
HugePages_Total:   20480
HugePages_Free:    20480

The way that I am allocating the images inside the container is doing

        for i in "${containers[@]}"; do
                sudo ctr t exec -d --exec-id "$(random_name)" "$i" sh -c "echo 100 | sudo tee /proc/sys/vm/nr_hugepages"
        done

after line https://github.com/kata-containers/tests/blob/main/metrics/storage/webtooling.sh#L175, however, it seems that when running 20 or more containers we have issues like

+ sudo ctr t exec -d --exec-id kata-gyxFrU kata-n2RmR5 sh -c 'echo 100 | sudo tee /proc/sys/vm/nr_hugepages'
++ sudo ctr t exec -d --exec-id kata-KIpdM7 kata-n2RmR5 sh -c /host/webtooling_start.sh
ctr: cannot enter container kata-JgZFAn, with err ttrpc: closed: unknown

Here it is the environment that I am using

[Kernel]
  Path = "/usr/share/kata-containers/vmlinux-5.10.25-85"
  Parameters = "systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket agent.log=debug agent.log=debug"

[Meta]
  Version = "1.0.25"

[Image]
  Path = "/usr/share/kata-containers/kata-containers.img"

[Initrd]
  Path = ""

[Agent]
  TraceMode = ""
  TraceType = ""
  Debug = true
  Trace = false

[Hypervisor]
  MachineType = "q35"
  Version = "cloud-hypervisor v18.0.0"
  Path = "/usr/bin/cloud-hypervisor"
  BlockDeviceDriver = "virtio-blk"
  EntropySource = "/dev/urandom"
  SharedFS = "virtio-fs"
  VirtioFSDaemon = "/usr/libexec/kata-qemu/virtiofsd"
  Msize9p = 8192
  MemorySlots = 10
  PCIeRootPort = 0
  HotplugVFIOOnRootBus = false
  Debug = true

[Netmon]
  Path = "/usr/libexec/kata-containers/kata-netmon"
  Debug = true
  Enable = false
  [Netmon.Version]
    Semver = "2.3.0-alpha0"
    Commit = "<<unknown>>"
    Major = 2
    Minor = 3
    Patch = 0

[Runtime]
  Path = "/usr/local/bin/kata-runtime"
  Debug = true
  Trace = false
  DisableGuestSeccomp = true
  DisableNewNetNs = false
  SandboxCgroupOnly = false
  [Runtime.Config]
    Path = "/usr/share/defaults/kata-containers/configuration.toml"
  [Runtime.Version]
    OCI = "1.0.2-dev"
    [Runtime.Version.Version]
      Semver = "2.3.0-alpha0"
      Commit = "45d40179c2f1bb6b3bbafa338557e196a9c7f540"
      Major = 2
      Minor = 3
      Patch = 0

And here are the logs

time="2021-09-20T19:56:16.117542041Z" level=debug msg="no loop device" error="open /sys/dev/block/8:2/loop/backing_file: no such file or directory" mount-source=/tmp/webtool.3m2JH8qUhE name=containerd-shim-v2 pid=2904695 sandbox=kata-n2RmR5 source=virtcontainers subsystem=container
time="2021-09-20T19:56:45.950640613Z" level=error msg="Wait for process failed" container=kata-JgZFAn error="Dead agent" name=containerd-shim-v2 pid=kata-JgZFAn sandbox=kata-JgZFAn source=containerd-kata-shim-v2
time="2021-09-20T19:56:45.950653658Z" level=warning msg="sandbox stopped unexpectedly" error="failed to ping hypervisor process: Get http://localhost/api/v1/vmm.ping: context deadline exceeded" name=containerd-shim-v2 pid=2904195 sandbox=kata-JgZFAn source=containerd-kata-shim-v2
time="2021-09-20T19:56:45.95068189Z" level=warning msg="failed to get OOM event from sandbox" error="ttrpc: closed" name=containerd-shim-v2 pid=2904195 sandbox=kata-JgZFAn source=containerd-kata-shim-v2
time="2021-09-20T19:56:46.088108101Z" level=warning msg="Agent did not stop sandbox" error="Dead agent" name=containerd-shim-v2 pid=2904195 sandbox=kata-JgZFAn sandboxid=kata-JgZFAn source=virtcontainers subsystem=sandbox
time="2021-09-20T19:56:46.985189419Z" level=warning msg="failed to get OOM event from sandbox" error="Dead agent" name=containerd-shim-v2 pid=2904195 sandbox=kata-JgZFAn source=containerd-kata-shim-v2
time="2021-09-20T19:56:49.113455215Z" level=error msg="Failed to read guest console logs" console-protocol=pty console-url=/dev/pts/18 error="read /dev/pts/18: file already closed" name=containerd-shim-v2 pid=2904195 sandbox=kata-JgZFAn source=virtcontainers subsystem=sandbox
time="2021-09-20T19:56:49.958865186Z" level=error msg="failed to cleanup cgroups" error="cgroups: cgroup deleted" name=containerd-shim-v2 pid=2904195 sandbox=kata-JgZFAn source=virtcontainers subsystem=sandbox
time="2021-09-20T19:56:49.96403314Z" level=error msg="failed to unmount vm share path /run/kata-containers/shared/sandboxes/kata-JgZFAn/shared" error="no such file or directory" name=containerd-shim-v2 pid=2904195 sandbox=kata-JgZFAn source=virtcontainers subsystem=kata_agent
time="2021-09-20T19:57:00.324407042Z" level=warning msg="removing vm socket failed" error="remove /run/vc/vm/kata-JgZFAn/clh.sock: no such file or directory" name=containerd-shim-v2 path=/run/vc/vm/kata-JgZFAn/clh.sock pid=2904195 sandbox=kata-JgZFAn source=virtcontainers subsystem=cloudHypervisor
time="2021-09-20T19:57:00.325085944Z" level=warning msg="failed to resolve vm path" dir=/run/vc/vm/kata-JgZFAn error="lstat /run/vc/vm/kata-JgZFAn: no such file or directory" name=containerd-shim-v2 pid=2904195 sandbox=kata-JgZFAn source=virtcontainers subsystem=cloudHypervisor

/cc @dborquez @likebreath

[dborquez]

I have replicated the kata setup mentioned by @GabyCT, and tried to set a value in /proc/sys/vm/nr_hugepages inside a single container as follows:

sudo ctr t exec -d --exec-id "$(random_name)" test sh -c 'echo 100 | tee /proc/sys/vm/nr_hugepages

tee: /proc/sys/vm/nr_hugepages: Read-only file system
100

In the second experiment I have set kernel_params = "rw" in the kata configuration.toml, and then ran the container with --privileged parameter and got this result:

sudo ctr run --privileged --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh

ctr: failed to create shim: failed to hotplug block device &{File:/dev/md0 Format:raw ID:drive-7800d0615bee700c MmioAddr: SCSIAddr: NvdimmID: VirtPath:/dev/vdj DevNo: PCIPath: Index:9 ShareRW:false ReadOnly:false Pmem:false Swap:false} error: 500  reason: VmAddDisk(VmAddDisk(DeviceManager(DetectImageType(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })))): unknown

[cmaf]

@likebreath do you think you could take a look?

[GabyCT]

Closing this issue as it has been solved

[likebreath]

For future reference:

The resolution was removing unused block devices under /dev (e.g. snap and loop devices) from the host machine that was experiencing the issue (AWS instance).

It is likely due to the limit of supporting up to 32 devices from Cloud Hypervisor, given the default behavior of Kata when launching privileged container is sharing all block devices under /dev to the guest/container (at least for Kata 1.x with Docker).