Support containerd 1.0 - Githubissues

sameo commented 7 years ago

Docker is planning to switch their stack from the current 0.2.4 containerd to a completely newcontainerd` 1.0 implementation.

There are many architectural changes with this new containerd implementations, and we may be impacted by mostly 2 of them:

A new GRPC interface for the shim. containerd no longer calls directly into containerd-shim but talks to a GRPC socket instead. By default containerd 1.0 ships with a canonical implementation (containerd-shim) for this GRPC interface but we should eventually implement it as part of our own shim implementation and get rid of one layer from our Docker based container stack.
A strong bound to runc. containerd heavily relies on the latest and greatest runc CLI which is not specified anywhere. It takes several runc specific assumptions as well. We will have to implement most of the missing runc options on order to safely work with containerd 1.0.

I put together a quick gist to run CC 3.0 with test the latest containerd code: https://gist.github.com/sameo/fa32cfe4f7a74d9eea74982bb1f0100d.

[x] Have all containerd integration tests running
[x] Docker and Clear Containers work using containerd 1.0
[ ] Be able to run containers using cri-contianred (including integration tests)
[ ] Tests Kubernetes and cri-containerd with Clear Containers

gorozco1 commented 7 years ago

@devimc @jcvenegas @sameo what about support for 2.1 ?

jcvenegas commented 7 years ago

I tested containerd with CC 3.0.3 using ctr, and I see is possible to launch contianers:

 ./bin/containerd --version
containerd github.com/containerd/containerd v1.0.0-beta.2-10-g7f4f4b5e

I replaced runc by cc-runtime ( I am looking to register an new runtime to the contianerd daemon) ln -sf /usr/bin/cc-runtime /usr/bin/runc

Pull busybox image

./bin/ctr --debug pull docker.io/library/busybox:latest

Run container

./bin/ctr --debug run   -t docker.io/library/busybox:latest t14
/ # uname -a
Linux clrcont 4.9.47-77.container #1 SMP Tue Sep 5 18:32:06 UTC 2017 x86_64 GNU/Linux

jcvenegas commented 7 years ago

I just seen that @sameo gist describe how to add a runtime

jcvenegas commented 7 years ago

Running contianerd integration test , I see the runtime hanging randomly at kill command, also cc-shim and qemu are still running.

root@singlevm:/home/jcvenega# pgrep cc-shim
9979
9980
10384
10385
root@singlevm:/home/jcvenega# pgrep cc-runtime
10403
root@singlevm:/home/jcvenega# pgrep qemu
9957
10367
root@singlevm:/home/jcvenega# ps aux | grep cc-runtime
root      1035  0.0  0.5 427500 38312 ?        Ssl  21:22   0:00 /usr/bin/dockerd -D --add-runtime cor=/usr/bin/cc-runtime --default-runtime=cor
root     10403  0.0  0.1  40388  7640 pts/0    Sl   21:44   0:00 /usr/bin/cc-runtime --root /run/containerd/runc/testing --log /run/containerd-test/io.containerd.runtime.v1.linux/testing/TestContainerKillAll/log.json --log-format json kill --all TestContainerKillAll 9
root     10989  0.0  0.0  12944   960 pts/1    S+   21:48   0:00 grep --color=auto cc-runtime

{"arguments":"\"create --bundle /run/containerd-test/io.containerd.runtime.v1.linux/testing/TestContainerKillAll --pid-file /run/containerd-test/io.containerd.runtime.v1.linux/testing/TestContainerKillAll/init.pid TestContainerKillAll\"","commit":"7495928331514ffdaa201d602fcf500066a3d796","level":"info","msg":"","name":"cc-runtime","source":"runtime","time":"2017-10-19T21:44:21Z","version":"3.0.3"} 
{"level":"info","msg":"No sockets from configuration","source":"virtcontainers","time":"2017-10-19T21:44:21Z"} 
{"level":"info","msg":"Device details for container TestContainerKillAll: Major:0, Minor:44, MountPoint:/run/containerd-test/io.containerd.runtime.v1.linux/testing/TestContainerKillAll/rootfs","source":"virtcontainers","time":"2017-10-19T21:44:21Z"} 
{"level":"info","msg":"Starting VM","source":"virtcontainers","time":"2017-10-19T21:44:21Z"} 
{"level":"info","msg":"launching qemu with: [-name pod-TestContainerKillAll -uuid 65436165-4100-0000-0000-000000000000 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/TestContainerKillAll/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/TestContainerKillAll/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=7999M -smp 2,cores=2,threads=1,sockets=1 -device virtio-9p-pci,disable-modern=true,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/run/containerd-test/io.containerd.runtime.v1.linux/testing/TestContainerKillAll/rootfs,security_model=none -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/TestContainerKillAll/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18220-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/TestContainerKillAll/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/TestContainerKillAll/tty.sock,server,nowait -device virtio-9p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/TestContainerKillAll,security_model=none -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.54-78.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::TestContainerKillAll::off::]","source":"virtcontainers","time":"2017-10-19T21:44:21Z"} 
{"level":"info","msg":"{\"QMP\": {\"version\": {\"qemu\": {\"micro\": 1, \"minor\": 7, \"major\": 2}, \"package\": \"(2.7.1+git.d4a337fe91-9.cc)\"}, \"capabilities\": []}}","source":"virtcontainers","time":"2017-10-19T21:44:21Z"} 
{"level":"info","msg":"QMP version 2.7.1","source":"virtcontainers","time":"2017-10-19T21:44:21Z"} 
{"level":"info","msg":"QMP capabilities []","source":"virtcontainers","time":"2017-10-19T21:44:21Z"} 
{"level":"info","msg":"{\"execute\":\"qmp_capabilities\"}","source":"virtcontainers","time":"2017-10-19T21:44:21Z"} 
{"level":"info","msg":"{\"return\": {}}","source":"virtcontainers","time":"2017-10-19T21:44:21Z"} 
{"level":"info","msg":"VM started","source":"virtcontainers","time":"2017-10-19T21:44:21Z"} 
{"level":"info","msg":"Shim(s) started","source":"virtcontainers","time":"2017-10-19T21:44:23Z"} 
{"container":"TestContainerKillAll","level":"info","msg":"Cgroups files not created because cgroupsPath was empty","pid":10384,"source":"runtime","time":"2017-10-19T21:44:23Z"} 
{"arguments":"\"state TestContainerKillAll\"","commit":"7495928331514ffdaa201d602fcf500066a3d796","level":"info","msg":"","name":"cc-runtime","source":"runtime","time":"2017-10-19T21:44:23Z","version":"3.0.3"} 
{"arguments":"\"start TestContainerKillAll\"","commit":"7495928331514ffdaa201d602fcf500066a3d796","level":"info","msg":"","name":"cc-runtime","source":"runtime","time":"2017-10-19T21:44:23Z","version":"3.0.3"} 
{"level":"info","msg":"Started Pod TestContainerKillAll","source":"virtcontainers","time":"2017-10-19T21:44:24Z"} 
{"arguments":"\"state TestContainerKillAll\"","commit":"7495928331514ffdaa201d602fcf500066a3d796","level":"info","msg":"","name":"cc-runtime","source":"runtime","time":"2017-10-19T21:44:24Z","version":"3.0.3"} 
{"arguments":"\"kill --all TestContainerKillAll 9\"","commit":"7495928331514ffdaa201d602fcf500066a3d796","level":"info","msg":"","name":"cc-runtime","source":"runtime","time":"2017-10-19T21:44:24Z","version":"3.0.3"}

jcvenegas commented 7 years ago

Update: Clear Containers Integration test are working: Clear Containers 3.0.6 + docker master using contianerd 1.0 beta 2

containerd integration tests: Most only the following test are failing:

TestTaskUpdate: The test hangs, because is trying to create limited 32MB container and the VM is not able to boot. If the memory is increased it still fails checking if our runtime actually modified the cgroup limit.

TestContainerPids: The command fails trying to call cc-runtime ps

container_test.go:389: /usr/bin/cc-runtime did not terminate sucessfully: Invalid command "ps"

TestContainerCloseIO: The test create a container with workload cat where the stdin is connected to a pipe. Then the pipe is closed (should handle a EOF or something similar) but the container cat command does not exit. Related to #612

clearcontainers / runtime

Support containerd 1.0 #292