search

kata-containers / runtime

Kata Containers version 1.x runtime (for version 2.x see https://github.com/kata-containers/kata-containers).
https://katacontainers.io/
Apache License 2.0
2.1k stars 376 forks source link

fail to run kata with specify memory flag #713

Closed Ace-Tang closed 3 years ago

Ace-Tang commented 6 years ago

Description of problem

Run container without flag can successful,

#pouch run --runtime=kata-runtime -d  reg.docker.alibaba-inc.com/ali/os:7u2
3fb075c98f1d0e1d818da3a12647d4cfb64e060efbc0bcb97f7fd865b6a93245

but if specify memory=2g or specify --cpuset-cpus="0", it will fail, 

#pouch run --runtime=kata-runtime -d  --cpuset-cpus="0" reg.docker.alibaba-inc.com/ali/os:7u2
Error: failed to run container 7cb9d8: {"message":"failed to create container(7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e) on containerd: failed to create task for container(7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e): OCI runtime create failed: rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:367: setting cgroup config for procHooks process caused \\\\\\\"cannot set limits on the cpuset cgroup, as the container has not joined it\\\\\\\"\\\"\": unknown"}

#pouch run --runtime=kata-runtime -d -m 5g reg.docker.alibaba-inc.com/ali/os:7u2
Error: failed to run container ef9911: {"message":"failed to create container(ef99111c83f5a81410f97c5a6bad840806a676d68de21c628c4c3d544308b97f) on containerd: failed to create task for container(ef99111c83f5a81410f97c5a6bad840806a676d68de21c628c4c3d544308b97f): OCI runtime create failed: rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:367: setting cgroup config for procHooks process caused \\\\\\\"failed to write 10737418240 to memory.memsw.limit_in_bytes: open /sys/fs/cgroup/memory/default/ef99111c83f5a81410f97c5a6bad840806a676d68de21c628c4c3d544308b97f/memory.memsw.limit_in_bytes: permission denied\\\\\\\"\\\"\": unknown"}

I found the error come from kata-agent, and kata-agent version is 1.3.0-rc0-e3eb9cec00671706e1e99f349d09995b1f382a33.

Expected result

(replace this text with an explanation of what you thought would happen)

Actual result

(replace this text with details of what actually happened)

logs:

#journalctl -q -o cat -a -t kata-runtime | grep 7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e
time="2018-09-11T13:57:47.639553057+08:00" level=info arch=amd64 arguments="\"create --bundle /home/t4/pouch/containerd/state/io.containerd.runtime.v1.linux/default/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e --pid-file /home/t4/pouch/containerd/state/io.containerd.runtime.v1.linux/default/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/init.pid 7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e\"" command=create commit=8f5fec806461729e089877f3fe9263cf519fb4de name=kata-runtime pid=244815 source=runtime version=1.3.0-rc0
time="2018-09-11T13:57:47.640028627+08:00" level=debug msg="converting /home/t4/pouch/containerd/state/io.containerd.runtime.v1.linux/default/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/config.json" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=oci
time="2018-09-11T13:57:47.641013405+08:00" level=debug arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e default-kernel-parameters="init=/usr/lib/systemd/systemd systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket" name=kata-runtime pid=244815 source=runtime
time="2018-09-11T13:57:47.642233246+08:00" level=debug msg="container rootfs: /home/t4/pouch/containerd/state/io.containerd.runtime.v1.linux/default/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/rootfs" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=oci
time="2018-09-11T13:57:47.642523992+08:00" level=info msg="shm-size detected: 67108864" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=oci
time="2018-09-11T13:57:47.907465343+08:00" level=debug msg="Creating bridges" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=qemu
time="2018-09-11T13:57:47.908137062+08:00" level=debug msg="Creating UUID" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=qemu
time="2018-09-11T13:57:47.908906971+08:00" level=debug msg="Disable nesting environment checks" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e inside-vm=false name=kata-runtime pid=244815 source=virtcontainers subsystem=qemu
time="2018-09-11T13:57:47.909502405+08:00" level=debug msg="agent: Using unix socket form VM socket endpoint" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=kata_agent
time="2018-09-11T13:57:47.909842222+08:00" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=kata_agent
time="2018-09-11T13:57:47.910240545+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e error="open /run/vc/sbs/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/devices.json: no such file or directory" name=kata-runtime pid=244815 sandbox=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e sandboxid=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e source=virtcontainers subsystem=sandbox
time="2018-09-11T13:57:47.982185449+08:00" level=info msg="Attaching endpoint" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e endpoint-type=virtual name=kata-runtime pid=244815 source=virtcontainers subsystem=network
time="2018-09-11T13:57:47.997206761+08:00" level=debug msg="Network added" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=default-network
time="2018-09-11T13:57:47.998452251+08:00" level=info msg="Starting VM" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 sandbox=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e source=virtcontainers subsystem=sandbox
time="2018-09-11T13:57:48.006213297+08:00" level=debug arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=244815 source=virtcontainers subsystem=qemu
time="2018-09-11T13:57:48.007001264+08:00" level=info msg="Adding extra file [0xc4202ca2b0 0xc4202ca2b8 0xc4202ca2c0 0xc4202ca2c8 0xc4202ca2d0 0xc4202ca2d8 0xc4202ca2e0 0xc4202ca2e8 0xc4202ca268 0xc4202ca278 0xc4202ca280 0xc4202ca288 0xc4202ca290 0xc4202ca298 0xc4202ca2a0 0xc4202ca2a8]" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=qmp
time="2018-09-11T13:57:48.007406567+08:00" level=info msg="launching qemu with: [-name sandbox-7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e -uuid 03b28ef9-d837-4842-8bd1-8b69cd234160 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/vm/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/qmp.sock,server,nowait -m 2048M,slots=2,maxmem=49154M -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2 -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/kata-containers/kata-containers.img,size=134217728 -device virtio-scsi-pci,id=scsi0 -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/vm/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/kata.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-kataShared,mount_tag=kataShared -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:c0:a8:05:05,mq=on,vectors=18 -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/kata-containers/vmlinuz.container -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug panic=1 nr_cpus=16 init=/usr/lib/systemd/systemd systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket -smp 1,cores=1,threads=1,sockets=1,maxcpus=16]" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=qmp
time="2018-09-11T13:57:48.077792026+08:00" level=info msg="{\"QMP\": {\"version\": {\"qemu\": {\"micro\": 1, \"minor\": 9, \"major\": 2}, \"package\": \"(kudu-1.0.0-20180710172019gdb07131.alios7)\"}, \"capabilities\": []}}" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=qmp
time="2018-09-11T13:57:48.07847086+08:00" level=info msg="QMP details" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 qmp-capabilities= qmp-major-version=2 qmp-micro-version=1 qmp-minor-version=9 source=virtcontainers subsystem=qemu
time="2018-09-11T13:57:48.078856964+08:00" level=info msg="{\"execute\":\"qmp_capabilities\"}" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=qmp
time="2018-09-11T13:57:48.083789305+08:00" level=info msg="{\"return\": {}}" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=qmp
time="2018-09-11T13:57:48.084406346+08:00" level=info msg="VM started" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 sandbox=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e source=virtcontainers subsystem=sandbox
time="2018-09-11T13:57:48.084713907+08:00" level=info msg="Starting regular Kata proxy rather than built-in" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 sandbox=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e source=virtcontainers subsystem=sandbox
time="2018-09-11T13:57:48.085345222+08:00" level=info msg="proxy started" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 proxy-pid=244909 proxy-url="unix:///run/vc/sbs/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/proxy.sock" sandbox=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e source=virtcontainers subsystem=kata_agent
time="2018-09-11T13:57:48.085635233+08:00" level=info msg="New client" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=kata_agent url="unix:///run/vc/sbs/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/proxy.sock"
time="2018-09-11T13:57:48.088660939+08:00" level=debug msg="sending request" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=grpc.CheckRequest pid=244815 req= source=virtcontainers subsystem=kata_agent
time="2018-09-11T13:57:49.034453371+08:00" level=info msg="New client" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=kata_agent url="unix:///run/vc/sbs/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/proxy.sock"
time="2018-09-11T13:57:49.035816671+08:00" level=debug msg="sending request" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=grpc.UpdateInterfaceRequest pid=244815 req="interface:<device:\"eth0\" name:\"eth0\" IPAddresses:<address:\"192.168.5.5\" mask:\"24\" > mtu:1500 hwAddr:\"02:42:c0:a8:05:05\" > " source=virtcontainers subsystem=kata_agent
time="2018-09-11T13:57:49.045056205+08:00" level=info msg="New client" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=kata_agent url="unix:///run/vc/sbs/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/proxy.sock"
time="2018-09-11T13:57:49.045960323+08:00" level=debug msg="sending request" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=grpc.UpdateRoutesRequest pid=244815 req="routes:<Routes:<gateway:\"192.168.5.1\" device:\"eth0\" > Routes:<dest:\"192.168.5.0/24\" device:\"eth0\" source:\"192.168.5.5\" scope:253 > > " source=virtcontainers subsystem=kata_agent
time="2018-09-11T13:57:49.049180632+08:00" level=info msg="New client" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=kata_agent url="unix:///run/vc/sbs/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/proxy.sock"
time="2018-09-11T13:57:49.049978437+08:00" level=debug msg="sending request" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=grpc.CreateSandboxRequest pid=244815 req="hostname:\"7cb9d893ec00\" storages:<driver:\"9p\" source:\"kataShared\" fstype:\"9p\" options:\"trans=virtio,version=9p2000.L\" options:\"nodev\" options:\"msize=8192\" mount_point:\"/run/kata-containers/shared/containers/\" > storages:<driver:\"ephemeral\" source:\"shm\" fstype:\"tmpfs\" options:\"noexec\" options:\"nosuid\" options:\"nodev\" options:\"mode=1777\" options:\"size=67108864\" mount_point:\"/run/kata-containers/sandbox/shm\" > sandbox_id:\"7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e\" " source=virtcontainers subsystem=kata_agent
time="2018-09-11T13:57:49.064484162+08:00" level=info msg="device details" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e device-major=0 device-minor=86 mount-point=/home/t4/pouch/containerd/state/io.containerd.runtime.v1.linux/default/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/rootfs name=kata-runtime pid=244815 sandbox=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e source=virtcontainers subsystem=container
time="2018-09-11T13:57:49.065910686+08:00" level=debug msg="Replacing OCI mount (/etc/hostname) source /home/t4/pouch/containers/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/hostname with /run/kata-containers/shared/containers/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e-197d7c77d5e9f1a1-hostname" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=kata_agent
time="2018-09-11T13:57:49.066196776+08:00" level=debug msg="Replacing OCI mount (/etc/hosts) source /home/t4/pouch/containers/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/hosts with /run/kata-containers/shared/containers/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e-3dc782a476e61f06-hosts" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=kata_agent
time="2018-09-11T13:57:49.066502229+08:00" level=debug msg="Replacing OCI mount (/etc/resolv.conf) source /home/t4/pouch/containers/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/resolv.conf with /run/kata-containers/shared/containers/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e-eaf7c91cdf046607-resolv.conf" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=kata_agent
time="2018-09-11T13:57:49.067003065+08:00" level=info msg="Using sandbox shm" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 shm-size=67108864 source=virtcontainers subsystem=kata_agent
time="2018-09-11T13:57:49.067280195+08:00" level=info msg="New client" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=kata_agent url="unix:///run/vc/sbs/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/proxy.sock"
time="2018-09-11T13:57:49.068723475+08:00" level=debug msg="sending request" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=grpc.CreateContainerRequest pid=244815 req="container_id:\"7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e\" exec_id:\"7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e\" OCI:<Version:\"1.0.1-dev\" Process:<User:<> Args:\"/sbin/init\" Env:\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\" Env:\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\" Env:\"container=docker\" Cwd:\"/\" Capabilities:<Bounding:\"CAP_CHOWN\" Bounding:\"CAP_DAC_OVERRIDE\" Bounding:\"CAP_FSETID\" Bounding:\"CAP_FOWNER\" Bounding:\"CAP_MKNOD\" Bounding:\"CAP_NET_RAW\" Bounding:\"CAP_SETGID\" Bounding:\"CAP_SETUID\" Bounding:\"CAP_SETFCAP\" Bounding:\"CAP_SETPCAP\" Bounding:\"CAP_NET_BIND_SERVICE\" Bounding:\"CAP_SYS_CHROOT\" Bounding:\"CAP_KILL\" Bounding:\"CAP_AUDIT_WRITE\" Effective:\"CAP_CHOWN\" Effective:\"CAP_DAC_OVERRIDE\" Effective:\"CAP_FSETID\" Effective:\"CAP_FOWNER\" Effective:\"CAP_MKNOD\" Effective:\"CAP_NET_RAW\" Effective:\"CAP_SETGID\" Effective:\"CAP_SETUID\" Effective:\"CAP_SETFCAP\" Effective:\"CAP_SETPCAP\" Effective:\"CAP_NET_BIND_SERVICE\" Effective:\"CAP_SYS_CHROOT\" Effective:\"CAP_KILL\" Effective:\"CAP_AUDIT_WRITE\" Inheritable:\"CAP_CHOWN\" Inheritable:\"CAP_DAC_OVERRIDE\" Inheritable:\"CAP_FSETID\" Inheritable:\"CAP_FOWNER\" Inheritable:\"CAP_MKNOD\" Inheritable:\"CAP_NET_RAW\" Inheritable:\"CAP_SETGID\" Inheritable:\"CAP_SETUID\" Inheritable:\"CAP_SETFCAP\" Inheritable:\"CAP_SETPCAP\" Inheritable:\"CAP_NET_BIND_SERVICE\" Inheritable:\"CAP_SYS_CHROOT\" Inheritable:\"CAP_KILL\" Inheritable:\"CAP_AUDIT_WRITE\" Permitted:\"CAP_CHOWN\" Permitted:\"CAP_DAC_OVERRIDE\" Permitted:\"CAP_FSETID\" Permitted:\"CAP_FOWNER\" Permitted:\"CAP_MKNOD\" Permitted:\"CAP_NET_RAW\" Permitted:\"CAP_SETGID\" Permitted:\"CAP_SETUID\" Permitted:\"CAP_SETFCAP\" Permitted:\"CAP_SETPCAP\" Permitted:\"CAP_NET_BIND_SERVICE\" Permitted:\"CAP_SYS_CHROOT\" Permitted:\"CAP_KILL\" Permitted:\"CAP_AUDIT_WRITE\" > OOMScoreAdj:-500 > Root:<Path:\"/run/kata-containers/shared/containers/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/rootfs\" > Hostname:\"7cb9d893ec00\" Mounts:<destination:\"/proc\" source:\"proc\" type:\"proc\" > Mounts:<destination:\"/dev\" source:\"tmpfs\" type:\"tmpfs\" options:\"nosuid\" options:\"strictatime\" options:\"mode=755\" options:\"size=65536k\" > Mounts:<destination:\"/dev/pts\" source:\"devpts\" type:\"devpts\" options:\"nosuid\" options:\"noexec\" options:\"newinstance\" options:\"ptmxmode=0666\" options:\"mode=0620\" options:\"gid=5\" > Mounts:<destination:\"/dev/shm\" source:\"/run/kata-containers/sandbox/shm\" type:\"bind\" options:\"rbind\" > Mounts:<destination:\"/dev/mqueue\" source:\"mqueue\" type:\"mqueue\" options:\"nosuid\" options:\"noexec\" options:\"nodev\" > Mounts:<destination:\"/sys\" source:\"sysfs\" type:\"sysfs\" options:\"nosuid\" options:\"noexec\" options:\"nodev\" options:\"ro\" > Mounts:<destination:\"/run\" source:\"tmpfs\" type:\"tmpfs\" options:\"nosuid\" options:\"strictatime\" options:\"mode=755\" options:\"size=65536k\" > Mounts:<destination:\"/sys/fs/cgroup\" source:\"cgroup\" type:\"cgroup\" options:\"ro\" options:\"nosuid\" options:\"noexec\" options:\"nodev\" > Mounts:<destination:\"/etc/hostname\" source:\"/run/kata-containers/shared/containers/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e-197d7c77d5e9f1a1-hostname\" type:\"bind\" options:\"rbind\" options:\"rprivate\" > Mounts:<destination:\"/etc/hosts\" source:\"/run/kata-containers/shared/containers/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e-3dc782a476e61f06-hosts\" type:\"bind\" options:\"rbind\" options:\"rprivate\" > Mounts:<destination:\"/etc/resolv.conf\" source:\"/run/kata-containers/shared/containers/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e-eaf7c91cdf046607-resolv.conf\" type:\"bind\" options:\"rbind\" options:\"rprivate\" > Annotations:<key:\"__memory_extra_in_bytes\" value:\"0\" > Annotations:<key:\"__memory_force_empty_ctl\" value:\"0\" > Annotations:<key:\"__memory_wmark_ratio\" value:\"0\" > Annotations:<key:\"__schedule_latency_switch\" value:\"0\" > Linux:<Resources:<Memory:<> CPU:<Cpus:\"0\" > > CgroupsPath:\"/default/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e\" Namespaces:<Type:\"ipc\" > Namespaces:<Type:\"uts\" > Namespaces:<Type:\"mount\" > MaskedPaths:\"/proc/kcore\" MaskedPaths:\"/proc/latency_stats\" MaskedPaths:\"/proc/timer_list\" MaskedPaths:\"/proc/timer_stats\" MaskedPaths:\"/proc/sched_debug\" MaskedPaths:\"/sys/firmware\" MaskedPaths:\"/proc/scsi\" ReadonlyPaths:\"/proc/asound\" ReadonlyPaths:\"/proc/bus\" ReadonlyPaths:\"/proc/fs\" ReadonlyPaths:\"/proc/irq\" ReadonlyPaths:\"/proc/sys\" ReadonlyPaths:\"/proc/sysrq-trigger\" > > " source=virtcontainers subsystem=kata_agent
time="2018-09-11T13:57:49.239813547+08:00" level=info msg="New client" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=kata_agent url="unix:///run/vc/sbs/7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e/proxy.sock"
time="2018-09-11T13:57:49.240695512+08:00" level=debug msg="sending request" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=grpc.DestroySandboxRequest pid=244815 req= source=virtcontainers subsystem=kata_agent
time="2018-09-11T13:57:49.250203066+08:00" level=info msg="Stopping Sandbox" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=qemu
time="2018-09-11T13:57:49.250702232+08:00" level=info msg="{\"QMP\": {\"version\": {\"qemu\": {\"micro\": 1, \"minor\": 9, \"major\": 2}, \"package\": \"(kudu-1.0.0-20180710172019gdb07131.alios7)\"}, \"capabilities\": []}}" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=qmp
time="2018-09-11T13:57:49.251171668+08:00" level=info msg="{\"execute\":\"qmp_capabilities\"}" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=qmp
time="2018-09-11T13:57:49.252024449+08:00" level=info msg="{\"return\": {}}" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=qmp
time="2018-09-11T13:57:49.25244163+08:00" level=info msg="{\"execute\":\"quit\"}" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=qmp
time="2018-09-11T13:57:49.253106203+08:00" level=info msg="{\"return\": {}}" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=qmp
time="2018-09-11T13:57:49.253350718+08:00" level=info msg="{\"timestamp\": {\"seconds\": 1536645469, \"microseconds\": 253096}, \"event\": \"SHUTDOWN\"}" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=qmp
time="2018-09-11T13:57:49.253566126+08:00" level=info msg="Detaching endpoint" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e endpoint-type=virtual name=kata-runtime pid=244815 source=virtcontainers subsystem=network
time="2018-09-11T13:57:49.317136398+08:00" level=debug msg="Network removed" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=default-network
time="2018-09-11T13:57:49.317595757+08:00" level=info msg="Network namespace \"/var/run/netns/cni-17b41c6c-6f11-778b-462d-1f0c50330fc0\" deleted" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=virtcontainers subsystem=default-network
time="2018-09-11T13:57:49.318054729+08:00" level=error msg="rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:367: setting cgroup config for procHooks process caused \\\\\\\"cannot set limits on the cpuset cgroup, as the container has not joined it\\\\\\\"\\\"\"" arch=amd64 command=create container=7cb9d893ec001750d366751714be6c39bf4eaf77e1d264f46acf386b1324e96e name=kata-runtime pid=244815 source=runtime
jodh-intel commented 6 years ago

/cc @devimc.

Ace-Tang commented 6 years ago

I try docker, it also fails,

#docker run -d --runtime=kata-runtime reg.docker.alibaba-inc.com/ali/os:7u2
e2850cf332577e44ac10797d457b977b03d78f46a96b8b34e965374bd2e02f67

[root@t100081180009.et15sqa /home/huamin.thm]
#docker run -d -m 5g --runtime=kata-runtime reg.docker.alibaba-inc.com/ali/os:7u2
50935d902dda4b083fb306efc708db1b7b1512e7b91bdf22b9feff53bed2f85a
docker: Error response from daemon: oci runtime error: rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10737418240 to memory.memsw.limit_in_bytes: open /sys/fs/cgroup/memory/docker/50935d902dda4b083fb306efc708db1b7b1512e7b91bdf22b9feff53bed2f85a/memory.memsw.limit_in_bytes: permission denied\\\"\"".

seems set cgroup also do in vm, thats why it fails.

devimc commented 6 years ago

@Ace-Tang sorry I don't have access to reg.docker.alibaba-inc.com/ali/os:7u2

with debian works for me

$ docker run -d -m 5g --runtime=kata-runtime debian bash
e5ffaa122cf9424366d570c06850ed113b120d5443bf8045c69a89bfbdd21229
Ace-Tang commented 6 years ago

@devimc, does this related guest image. I am not use the image created by the latest osbuilder

devimc commented 6 years ago

@Ace-Tang mmm ok, I guess you are using a custom kernel,

Ace-Tang commented 6 years ago

@devimc , yes, I use 4.9.69-006-3005309+ guest kernel with some custom, but most is base on upstream. failed to write 10737418240 to memory.memsw.limit_in_bytes the log comes from kata-proxy, is proxy also set cgroup in vm? and it need kernel do something? I just guess.

devimc commented 6 years ago

@Ace-Tang seems like your kernel is not up-to-date

see https://github.com/kata-containers/packaging/pull/104 and https://github.com/kata-containers/packaging/blob/master/kernel/configs/x86_64_kata_kvm_4.14.x#L150-L172

bergwolf commented 6 years ago

@devimc I've had it in mind for some time that we should also support the cases that there is no cgroups support in the guest kernel (by adding a kata-runtime config option of course). Then users can choose not to restrict there containers with cgroups in the guest but just add proper limits on the host.

  1. cpu: put qemu vCPU thread in host memcg
  2. memory: limited by qemu guest memory size
  3. network: tc on the host
  4. IO-9pfs: put qemu process in blkio cg
  5. IO-virtio-blk/scsi: add proper qemu virtio parameter to limit the device speed

wdyt about it? cc @sboeuf @gnawux @egernst

Ace-Tang commented 6 years ago

Set memory fail because host not have memory.memsw.limit_in_bytes, but set cpu also fail, I will check where the problem is

#pouch run -d --cpuset-cpus=1 --runtime=kata-runtime f0083c036ad3
Error: failed to run container 9019d1: {"message":"failed to create container(9019d17ee7bc4034af1b032e8edf0dbf756bec0e22f6f90f9757c96f60452f78) on containerd: failed to create task for container(9019d17ee7bc4034af1b032e8edf0dbf756bec0e22f6f90f9757c96f60452f78): OCI runtime create failed: rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"process_linux.go:367: setting cgroup config for procHooks process caused \\\\\\\"cannot set limits on the cpuset cgroup, as the container has not joined it\\\\\\\"\\\"\", 0\n, <nil>: unknown"}
Ace-Tang commented 6 years ago

I found that if specify --cpuset-cpus="0,1" , but set default_vcpus = 1 in /etc/kata-containers/configuration.toml, the container will fail to start, since qemu only get 1 cpu core, I just wonder if default_vcpus less than count of cpuset, we can set default_vcpus=len(cpuset-cpus), @devimc @bergwolf .