search

clearcontainers / runtime

OCI (Open Containers Initiative) compatible runtime using Virtual Machines
Apache License 2.0
589 stars 70 forks source link

docker kill SIGWINCH is not forwarded to container #768

Open jcvenegas opened 7 years ago

jcvenegas commented 7 years ago

Description of problem

https://github.com/clearcontainers/proxy/blob/master/proxy.go#L451

Expected result

Workload will get the signal if docker kill 28 is used. Example: 

docker run --runtime cc-runtime --name P1GjgyJuoQSVthDf1APV4lnjTauxd9 -dt busybox sh -c 'trap "exit 28" 28 ; while : ; do sleep 1; done'
docker kill -s 28 P1GjgyJuoQSVthDf1APV4lnjTauxd9

The contianer should exit after do kill, but looking at docker ps is still up.

$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES fed26d7492f6 busybox "sh -c 'trap \"exit..." 8 minutes ago Up 8 minutes P1GjgyJuoQSVthDf1APV4lnjTauxd9

Actual result

The workload get the signal and in the example exits. (this works with runc)

cc-data.txt

Meta details

Running cc-collect-data.sh version 3.0.5 (commit 40047b5) at 2017-10-26.15:04:14.562059613.

Runtime is /bin/cc-runtime.

cc-env

Output of "/bin/cc-runtime cc-env":

[Meta]
  Version = "1.0.4"

[Runtime]
  [Runtime.Version]
    Semver = "3.0.5"
    Commit = "40047b5"
    OCI = "1.0.0-dev"
  [Runtime.Config]
    Path = "/usr/share/defaults/clear-containers/configuration.toml"
    GlobalLogPath = "/var/lib/clear-containers/runtime/runtime.log"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.7.0, Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"

[Image]
  Path = "/usr/share/clear-containers/clear-18400-containers.img"

[Kernel]
  Path = "/usr/share/clear-containers/vmlinuz-4.9.58-79.1.container"
  Parameters = ""

[Proxy]
  Type = "ccProxy"
  Version = "Version: 3.0.5+git.fb22251"
  URL = "unix:///var/run/clear-containers/proxy.sock"

[Shim]
  Type = "ccShim"
  Version = "shim version: 3.0.5 (commit: 86a7c20)"
  Path = "/usr/libexec/clear-containers/cc-shim"

[Agent]
  Type = "hyperstart"
  Version = "<<unknown>>"
  PauseBinPath = "/var/lib/clear-containers/runtime/bundles/pause_bundle/bin/pause"

[Host]
  Kernel = "4.13.5-100.fc25.x86_64"
  CCCapable = true
  [Host.Distro]
    Name = "Fedora"
    Version = "25"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz"

Runtime config files

Runtime default config files

/usr/share/defaults/clear-containers/configuration.toml
/usr/share/defaults/clear-containers/configuration.toml

Runtime config file contents

Config file /etc/clear-containers/configuration.toml not found Output of "cat "/usr/share/defaults/clear-containers/configuration.toml"":

# XXX: Warning: this file is auto-generated from file "config/configuration.toml.in".

[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
kernel = "/usr/share/clear-containers/vmlinuz.container"
image = "/usr/share/clear-containers/clear-containers.img"
machine_type = "pc"
# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc
kernel_params = ""

# Default number of vCPUs per POD/VM:
# unspecified or 0 --> will be set to 1
# < 0              --> will be set to the actual number of physical cores
# > 0 <= 255       --> will be set to the specified number
# > 255            --> will be set to 255
default_vcpus = -1

# Default memory size in MiB for POD/VM.
# If unspecified then it will be set 2048 MiB.
#default_memory = 2048
disable_block_device_use = false

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically 
# result in memory pre allocation
#enable_hugepages = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# Debug changes the default hypervisor and kernel parameters to
# enable debug output where available.
# Default false
# these logs can be obtained in the cc-proxy logs  when the 
# proxy is set to run in debug mode
# /usr/libexec/clear-containers/cc-proxy -log debug
# or by stopping the cc-proxy service and running the cc-proxy 
# explicitly using the same command line
# 
enable_debug = true

# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
# 
#disable_nesting_checks = true

[proxy.cc]
url = "unix:///var/run/clear-containers/proxy.sock"

[shim.cc]
path = "/usr/libexec/clear-containers/cc-shim"

# If enabled, shim messages will be sent to the system log
# (default: disabled)
enable_debug = true

[agent.hyperstart]
pause_root_path = "/var/lib/clear-containers/runtime/bundles/pause_bundle"

[runtime]
## Uncomment to enable the global logging to the default path.
global_log_path = "/var/lib/clear-containers/runtime/runtime.log"

# If enabled, the runtime will log additional debug messages to the global
# log, assuming that is also enabled.
# (default: disabled)
enable_debug = true

Logfiles

Runtime logs

Recent runtime problems found in global log /var/lib/clear-containers/runtime/runtime.log:

time="2017-10-26T14:34:40.244795313-05:00" pid=19972 name="cc-runtime" level="error" source="runtime" msg="Container ID (28b85bbafe5afa686a7b1a75ccd35e9c76764738efd20651e285fdc931c1917b) does not exist"
time="2017-10-26T14:34:43.552844797-05:00" pid=20037 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-60a2525a7d72e6936ee8844653022ddbd914dbef4ef26a98da9aba6133e524db -uuid 36306132-3532-3561-3764-373265363933 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/60a2525a7d72e6936ee8844653022ddbd914dbef4ef26a98da9aba6133e524db/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/60a2525a7d72e6936ee8844653022ddbd914dbef4ef26a98da9aba6133e524db/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/6afc54161a4e29d6c74e5aaec8edcafd01a9277e3977593eb9c1acbd064e38a1/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/60a2525a7d72e6936ee8844653022ddbd914dbef4ef26a98da9aba6133e524db/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/60a2525a7d72e6936ee8844653022ddbd914dbef4ef26a98da9aba6133e524db/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/60a2525a7d72e6936ee8844653022ddbd914dbef4ef26a98da9aba6133e524db/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/60a2525a7d72e6936ee8844653022ddbd914dbef4ef26a98da9aba6133e524db,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::60a2525a7d72e6936ee8844653022ddbd914dbef4ef26a98da9aba6133e524db::off::]"
time="2017-10-26T14:34:51.158613266-05:00" pid=20094 name="cc-runtime" level="error" source="runtime" msg="Container 60a2525a7d72e6936ee8844653022ddbd914dbef4ef26a98da9aba6133e524db not ready or running, cannot send a signal"
time="2017-10-26T14:34:52.655010411-05:00" pid=20120 name="cc-runtime" level="error" source="runtime" msg="Container ID (60a2525a7d72e6936ee8844653022ddbd914dbef4ef26a98da9aba6133e524db) does not exist"
time="2017-10-26T14:34:58.755128102-05:00" pid=20180 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-4538999b1d7e620f5e729c3ca5858aa8f5ff1061149aa685e7b7fb6a8d06d4b5 -uuid 34353338-3939-3962-3164-376536323066 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/4538999b1d7e620f5e729c3ca5858aa8f5ff1061149aa685e7b7fb6a8d06d4b5/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/4538999b1d7e620f5e729c3ca5858aa8f5ff1061149aa685e7b7fb6a8d06d4b5/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/33d851856706b08e00d277791ec5a2e44b9da0f566232a0982b52839859558f1/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/4538999b1d7e620f5e729c3ca5858aa8f5ff1061149aa685e7b7fb6a8d06d4b5/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/4538999b1d7e620f5e729c3ca5858aa8f5ff1061149aa685e7b7fb6a8d06d4b5/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/4538999b1d7e620f5e729c3ca5858aa8f5ff1061149aa685e7b7fb6a8d06d4b5/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/4538999b1d7e620f5e729c3ca5858aa8f5ff1061149aa685e7b7fb6a8d06d4b5,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::4538999b1d7e620f5e729c3ca5858aa8f5ff1061149aa685e7b7fb6a8d06d4b5::off::]"
time="2017-10-26T14:35:05.892807761-05:00" pid=20254 name="cc-runtime" level="error" source="runtime" msg="Container 4538999b1d7e620f5e729c3ca5858aa8f5ff1061149aa685e7b7fb6a8d06d4b5 not ready or running, cannot send a signal"
time="2017-10-26T14:35:06.132205238-05:00" pid=20266 name="cc-runtime" level="error" source="runtime" msg="Container ID (4538999b1d7e620f5e729c3ca5858aa8f5ff1061149aa685e7b7fb6a8d06d4b5) does not exist"
time="2017-10-26T14:35:10.576431724-05:00" pid=20325 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-11494f60f0f76eacc72d355d41490eecf2da717f50d05ecebe561cb4633d7057 -uuid 31313439-3466-3630-6630-663736656163 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/11494f60f0f76eacc72d355d41490eecf2da717f50d05ecebe561cb4633d7057/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/11494f60f0f76eacc72d355d41490eecf2da717f50d05ecebe561cb4633d7057/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/ecdcbb3a1ea2fbf2a9301d1e11b983bf6f84ef295b94238590dcade408a0ddcd/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/11494f60f0f76eacc72d355d41490eecf2da717f50d05ecebe561cb4633d7057/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/11494f60f0f76eacc72d355d41490eecf2da717f50d05ecebe561cb4633d7057/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/11494f60f0f76eacc72d355d41490eecf2da717f50d05ecebe561cb4633d7057/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/11494f60f0f76eacc72d355d41490eecf2da717f50d05ecebe561cb4633d7057,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::11494f60f0f76eacc72d355d41490eecf2da717f50d05ecebe561cb4633d7057::off::]"
time="2017-10-26T14:35:18.212741274-05:00" pid=20392 name="cc-runtime" level="error" source="runtime" msg="Container 11494f60f0f76eacc72d355d41490eecf2da717f50d05ecebe561cb4633d7057 not ready or running, cannot send a signal"
time="2017-10-26T14:35:19.178801075-05:00" pid=20419 name="cc-runtime" level="error" source="runtime" msg="Container ID (11494f60f0f76eacc72d355d41490eecf2da717f50d05ecebe561cb4633d7057) does not exist"
time="2017-10-26T14:35:23.483815601-05:00" pid=20481 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-a52764920c1905e62eab7d45d973f47d06ccd2ecea6c1e5f071958b83bc23cd8 -uuid 61353237-3634-3932-3063-313930356536 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/a52764920c1905e62eab7d45d973f47d06ccd2ecea6c1e5f071958b83bc23cd8/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/a52764920c1905e62eab7d45d973f47d06ccd2ecea6c1e5f071958b83bc23cd8/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/e3ea1a22a25868a1192767ea37aa72ed68ac647a25ad2504844919f8fc660c12/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/a52764920c1905e62eab7d45d973f47d06ccd2ecea6c1e5f071958b83bc23cd8/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/a52764920c1905e62eab7d45d973f47d06ccd2ecea6c1e5f071958b83bc23cd8/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/a52764920c1905e62eab7d45d973f47d06ccd2ecea6c1e5f071958b83bc23cd8/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/a52764920c1905e62eab7d45d973f47d06ccd2ecea6c1e5f071958b83bc23cd8,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::a52764920c1905e62eab7d45d973f47d06ccd2ecea6c1e5f071958b83bc23cd8::off::]"
time="2017-10-26T14:35:31.618777618-05:00" pid=20545 name="cc-runtime" level="error" source="runtime" msg="Container a52764920c1905e62eab7d45d973f47d06ccd2ecea6c1e5f071958b83bc23cd8 not ready or running, cannot send a signal"
time="2017-10-26T14:35:32.803550239-05:00" pid=20569 name="cc-runtime" level="error" source="runtime" msg="Container ID (a52764920c1905e62eab7d45d973f47d06ccd2ecea6c1e5f071958b83bc23cd8) does not exist"
time="2017-10-26T14:35:35.137388469-05:00" pid=20646 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-b8cc9314dbbf64d67bf332e8d698d247e3850f4d8607f54fa848da29bf953329 -uuid 62386363-3933-3134-6462-626636346436 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/b8cc9314dbbf64d67bf332e8d698d247e3850f4d8607f54fa848da29bf953329/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/b8cc9314dbbf64d67bf332e8d698d247e3850f4d8607f54fa848da29bf953329/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/a89b37a56e3ece4f99b7305b11250fd3383295dd0f6b592e0620bba9c595d070/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/b8cc9314dbbf64d67bf332e8d698d247e3850f4d8607f54fa848da29bf953329/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/b8cc9314dbbf64d67bf332e8d698d247e3850f4d8607f54fa848da29bf953329/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/b8cc9314dbbf64d67bf332e8d698d247e3850f4d8607f54fa848da29bf953329/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/b8cc9314dbbf64d67bf332e8d698d247e3850f4d8607f54fa848da29bf953329,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::b8cc9314dbbf64d67bf332e8d698d247e3850f4d8607f54fa848da29bf953329::off::]"
time="2017-10-26T14:35:42.150827091-05:00" pid=20729 name="cc-runtime" level="error" source="runtime" msg="Container b8cc9314dbbf64d67bf332e8d698d247e3850f4d8607f54fa848da29bf953329 not ready or running, cannot send a signal"
time="2017-10-26T14:35:42.371623955-05:00" pid=20753 name="cc-runtime" level="error" source="runtime" msg="Container ID (b8cc9314dbbf64d67bf332e8d698d247e3850f4d8607f54fa848da29bf953329) does not exist"
time="2017-10-26T14:35:46.959671853-05:00" pid=20841 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-ffaa7a25f854c6762395c36fe2fbf43ac0151ad4a42eed50ac5b488bc4dcf73e -uuid 66666161-3761-3235-6638-353463363736 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/ffaa7a25f854c6762395c36fe2fbf43ac0151ad4a42eed50ac5b488bc4dcf73e/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/ffaa7a25f854c6762395c36fe2fbf43ac0151ad4a42eed50ac5b488bc4dcf73e/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/e594d9eed67eced1c7d66b9d2d34a3870b42cdb21a8a53b0e08afc0f3a99ed2e/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/ffaa7a25f854c6762395c36fe2fbf43ac0151ad4a42eed50ac5b488bc4dcf73e/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/ffaa7a25f854c6762395c36fe2fbf43ac0151ad4a42eed50ac5b488bc4dcf73e/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/ffaa7a25f854c6762395c36fe2fbf43ac0151ad4a42eed50ac5b488bc4dcf73e/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/ffaa7a25f854c6762395c36fe2fbf43ac0151ad4a42eed50ac5b488bc4dcf73e,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::ffaa7a25f854c6762395c36fe2fbf43ac0151ad4a42eed50ac5b488bc4dcf73e::off::]"
time="2017-10-26T14:35:54.665665477-05:00" pid=20920 name="cc-runtime" level="error" source="runtime" msg="Container ffaa7a25f854c6762395c36fe2fbf43ac0151ad4a42eed50ac5b488bc4dcf73e not ready or running, cannot send a signal"
time="2017-10-26T14:35:56.003112556-05:00" pid=20945 name="cc-runtime" level="error" source="runtime" msg="Container ID (ffaa7a25f854c6762395c36fe2fbf43ac0151ad4a42eed50ac5b488bc4dcf73e) does not exist"
time="2017-10-26T14:35:58.09087867-05:00" pid=20999 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-18c750a9a231ff565b9a5303b57b9f08629d36a6d0da49a540bd6d347de545d1 -uuid 31386337-3530-6139-6132-333166663536 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/18c750a9a231ff565b9a5303b57b9f08629d36a6d0da49a540bd6d347de545d1/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/18c750a9a231ff565b9a5303b57b9f08629d36a6d0da49a540bd6d347de545d1/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/8361cd0d5a83a6fe5de82213bd95ae568bb2b1e7c291bb1bb68972a2df981d70/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/18c750a9a231ff565b9a5303b57b9f08629d36a6d0da49a540bd6d347de545d1/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/18c750a9a231ff565b9a5303b57b9f08629d36a6d0da49a540bd6d347de545d1/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/18c750a9a231ff565b9a5303b57b9f08629d36a6d0da49a540bd6d347de545d1/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/18c750a9a231ff565b9a5303b57b9f08629d36a6d0da49a540bd6d347de545d1,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::18c750a9a231ff565b9a5303b57b9f08629d36a6d0da49a540bd6d347de545d1::off::]"
time="2017-10-26T14:36:06.09968451-05:00" pid=21055 name="cc-runtime" level="error" source="runtime" msg="Container 18c750a9a231ff565b9a5303b57b9f08629d36a6d0da49a540bd6d347de545d1 not ready or running, cannot send a signal"
time="2017-10-26T14:36:07.267619927-05:00" pid=21081 name="cc-runtime" level="error" source="runtime" msg="Container ID (18c750a9a231ff565b9a5303b57b9f08629d36a6d0da49a540bd6d347de545d1) does not exist"
time="2017-10-26T14:36:12.554007339-05:00" pid=21138 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-cfd191fd1785276419b1c95335acc08ab415828c265a871597f6aa0f164ec2ce -uuid 63666431-3931-6664-3137-383532373634 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/cfd191fd1785276419b1c95335acc08ab415828c265a871597f6aa0f164ec2ce/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/cfd191fd1785276419b1c95335acc08ab415828c265a871597f6aa0f164ec2ce/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/60c771915e5f2f3c60da55da420ded74b81fb9590d4596c2010d8f427ec29929/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/cfd191fd1785276419b1c95335acc08ab415828c265a871597f6aa0f164ec2ce/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/cfd191fd1785276419b1c95335acc08ab415828c265a871597f6aa0f164ec2ce/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/cfd191fd1785276419b1c95335acc08ab415828c265a871597f6aa0f164ec2ce/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/cfd191fd1785276419b1c95335acc08ab415828c265a871597f6aa0f164ec2ce,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::cfd191fd1785276419b1c95335acc08ab415828c265a871597f6aa0f164ec2ce::off::]"
time="2017-10-26T14:36:23.656541903-05:00" pid=21225 name="cc-runtime" level="error" source="runtime" msg="Container cfd191fd1785276419b1c95335acc08ab415828c265a871597f6aa0f164ec2ce not ready or running, cannot send a signal"
time="2017-10-26T14:36:23.941210036-05:00" pid=21238 name="cc-runtime" level="error" source="runtime" msg="Container ID (cfd191fd1785276419b1c95335acc08ab415828c265a871597f6aa0f164ec2ce) does not exist"
time="2017-10-26T14:36:24.628177505-05:00" pid=21283 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110 -uuid 39613764-3466-3262-6361-333430623437 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/d262fc6b4370a2428cdda4d0e1518069aeceeb48cd16c4eff7e49193764ed5d1/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110::off::]"
time="2017-10-26T14:36:32.405646545-05:00" pid=21343 name="cc-runtime" level="error" source="runtime" msg="Container 9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110 not ready or running, cannot send a signal"
time="2017-10-26T14:36:33.409878125-05:00" pid=21367 name="cc-runtime" level="error" source="runtime" msg="Container ID (9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110) does not exist"
time="2017-10-26T14:36:36.939148594-05:00" pid=21429 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247 -uuid 64666332-3333-3762-3161-396164393765 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/764436c3dd5158267812ba1244bd0343ca0b8a32a01ab243659e671574e8ef41/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247::off::]"
time="2017-10-26T14:36:45.140867103-05:00" pid=21527 name="cc-runtime" level="error" source="runtime" msg="Container dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247 not ready or running, cannot send a signal"
time="2017-10-26T14:36:45.812152044-05:00" pid=21553 name="cc-runtime" level="error" source="runtime" msg="Container ID (dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247) does not exist"
time="2017-10-26T14:36:49.235014349-05:00" pid=21616 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-4a359f5c475b5dad18a64ee46874c6f274e95dcf9bd0075cacad1a666a57de0c -uuid 34613335-3966-3563-3437-356235646164 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/4a359f5c475b5dad18a64ee46874c6f274e95dcf9bd0075cacad1a666a57de0c/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/4a359f5c475b5dad18a64ee46874c6f274e95dcf9bd0075cacad1a666a57de0c/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/a8134748defb84847d542d27b87c51f7fd10cc164e93ccdc51f33bab9b67ca7d/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/4a359f5c475b5dad18a64ee46874c6f274e95dcf9bd0075cacad1a666a57de0c/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/4a359f5c475b5dad18a64ee46874c6f274e95dcf9bd0075cacad1a666a57de0c/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/4a359f5c475b5dad18a64ee46874c6f274e95dcf9bd0075cacad1a666a57de0c/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/4a359f5c475b5dad18a64ee46874c6f274e95dcf9bd0075cacad1a666a57de0c,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::4a359f5c475b5dad18a64ee46874c6f274e95dcf9bd0075cacad1a666a57de0c::off::]"
time="2017-10-26T14:36:56.149631988-05:00" pid=21697 name="cc-runtime" level="error" source="runtime" msg="Container 4a359f5c475b5dad18a64ee46874c6f274e95dcf9bd0075cacad1a666a57de0c not ready or running, cannot send a signal"
time="2017-10-26T14:36:57.531930132-05:00" pid=21721 name="cc-runtime" level="error" source="runtime" msg="Container ID (4a359f5c475b5dad18a64ee46874c6f274e95dcf9bd0075cacad1a666a57de0c) does not exist"
time="2017-10-26T14:40:00.053840765-05:00" pid=21856 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-f8d6217875d662560ca15de7c31b5c459b2f1e3591739ff3bb8b2f79053fc9ab -uuid 66386436-3231-3738-3735-643636323536 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/f8d6217875d662560ca15de7c31b5c459b2f1e3591739ff3bb8b2f79053fc9ab/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/f8d6217875d662560ca15de7c31b5c459b2f1e3591739ff3bb8b2f79053fc9ab/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/1fcad6a1b7aa5561493ec65e2876d8ee7eacf1c0d4df13b47fe9a9f56566938c/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/f8d6217875d662560ca15de7c31b5c459b2f1e3591739ff3bb8b2f79053fc9ab/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/f8d6217875d662560ca15de7c31b5c459b2f1e3591739ff3bb8b2f79053fc9ab/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/f8d6217875d662560ca15de7c31b5c459b2f1e3591739ff3bb8b2f79053fc9ab/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/f8d6217875d662560ca15de7c31b5c459b2f1e3591739ff3bb8b2f79053fc9ab,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::f8d6217875d662560ca15de7c31b5c459b2f1e3591739ff3bb8b2f79053fc9ab::off::]"
time="2017-10-26T14:40:01.696592654-05:00" pid=21915 name="cc-runtime" level="error" source="runtime" msg="Container f8d6217875d662560ca15de7c31b5c459b2f1e3591739ff3bb8b2f79053fc9ab not ready or running, cannot send a signal"
time="2017-10-26T14:40:02.471469505-05:00" pid=21933 name="cc-runtime" level="error" source="runtime" msg="Container ID (f8d6217875d662560ca15de7c31b5c459b2f1e3591739ff3bb8b2f79053fc9ab) does not exist"
time="2017-10-26T14:42:08.717010508-05:00" pid=22533 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-3e9b456872edfbd5008d1cfe86b799b96983135eeb3abe60ab7b5f0408965870 -uuid 33653962-3435-3638-3732-656466626435 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/3e9b456872edfbd5008d1cfe86b799b96983135eeb3abe60ab7b5f0408965870/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/3e9b456872edfbd5008d1cfe86b799b96983135eeb3abe60ab7b5f0408965870/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/4c2e6f703f08121143a5fb35c8593efe7a106be8e91351a4d6efb2458d7f53f5/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/3e9b456872edfbd5008d1cfe86b799b96983135eeb3abe60ab7b5f0408965870/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/3e9b456872edfbd5008d1cfe86b799b96983135eeb3abe60ab7b5f0408965870/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/3e9b456872edfbd5008d1cfe86b799b96983135eeb3abe60ab7b5f0408965870/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/3e9b456872edfbd5008d1cfe86b799b96983135eeb3abe60ab7b5f0408965870,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::3e9b456872edfbd5008d1cfe86b799b96983135eeb3abe60ab7b5f0408965870::off::]"
time="2017-10-26T14:49:15.036740411-05:00" pid=23560 name="cc-runtime" level="error" source="runtime" msg="Container 3e9b456872edfbd5008d1cfe86b799b96983135eeb3abe60ab7b5f0408965870 not ready or running, cannot send a signal"
time="2017-10-26T14:49:15.279488342-05:00" pid=23575 name="cc-runtime" level="error" source="runtime" msg="Container ID (3e9b456872edfbd5008d1cfe86b799b96983135eeb3abe60ab7b5f0408965870) does not exist"
time="2017-10-26T14:50:16.728279878-05:00" pid=23896 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-7d3cbf2f606e51d9d273b5a34af43154e540c295c528a51cf7c19ab8161ae22b -uuid 37643363-6266-3266-3630-366535316439 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/7d3cbf2f606e51d9d273b5a34af43154e540c295c528a51cf7c19ab8161ae22b/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/7d3cbf2f606e51d9d273b5a34af43154e540c295c528a51cf7c19ab8161ae22b/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/1736955ae3469c3d5e0283b7c8296cfcf5abdf7601f871cede31d202211b54ed/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/7d3cbf2f606e51d9d273b5a34af43154e540c295c528a51cf7c19ab8161ae22b/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/7d3cbf2f606e51d9d273b5a34af43154e540c295c528a51cf7c19ab8161ae22b/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/7d3cbf2f606e51d9d273b5a34af43154e540c295c528a51cf7c19ab8161ae22b/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/7d3cbf2f606e51d9d273b5a34af43154e540c295c528a51cf7c19ab8161ae22b,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::7d3cbf2f606e51d9d273b5a34af43154e540c295c528a51cf7c19ab8161ae22b::off::]"
time="2017-10-26T14:50:42.406689124-05:00" pid=24002 name="cc-runtime" level="error" source="runtime" msg="Container 7d3cbf2f606e51d9d273b5a34af43154e540c295c528a51cf7c19ab8161ae22b not ready or running, cannot send a signal"
time="2017-10-26T14:50:43.489850898-05:00" pid=24028 name="cc-runtime" level="error" source="runtime" msg="Container ID (7d3cbf2f606e51d9d273b5a34af43154e540c295c528a51cf7c19ab8161ae22b) does not exist"
time="2017-10-26T14:50:51.251190761-05:00" pid=24136 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-aa308727a8d19a046175fe86a117bcb27ba9ec4408ea6c07b145413f1f688691 -uuid 61613330-3837-3237-6138-643139613034 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/aa308727a8d19a046175fe86a117bcb27ba9ec4408ea6c07b145413f1f688691/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/aa308727a8d19a046175fe86a117bcb27ba9ec4408ea6c07b145413f1f688691/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/b5c6fc2b6d24801c3248459f2c91b8a3b57269e157737836460463133a1c0f63/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/aa308727a8d19a046175fe86a117bcb27ba9ec4408ea6c07b145413f1f688691/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/aa308727a8d19a046175fe86a117bcb27ba9ec4408ea6c07b145413f1f688691/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/aa308727a8d19a046175fe86a117bcb27ba9ec4408ea6c07b145413f1f688691/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/aa308727a8d19a046175fe86a117bcb27ba9ec4408ea6c07b145413f1f688691,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 quiet systemd.show_status=false init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::aa308727a8d19a046175fe86a117bcb27ba9ec4408ea6c07b145413f1f688691::off::]"
time="2017-10-26T14:52:52.754818256-05:00" pid=24586 name="cc-runtime" level="error" source="runtime" msg="Container aa308727a8d19a046175fe86a117bcb27ba9ec4408ea6c07b145413f1f688691 not ready or running, cannot send a signal"
time="2017-10-26T14:52:52.993613744-05:00" pid=24598 name="cc-runtime" level="error" source="runtime" msg="Container ID (aa308727a8d19a046175fe86a117bcb27ba9ec4408ea6c07b145413f1f688691) does not exist"
time="2017-10-26T14:53:03.729032648-05:00" pid=24680 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-c61a6341e6337ca404669a47c8fff5fe0412a130d250788f75ecae8cf389d4ae -uuid 63363161-3633-3431-6536-333337636134 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/c61a6341e6337ca404669a47c8fff5fe0412a130d250788f75ecae8cf389d4ae/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/c61a6341e6337ca404669a47c8fff5fe0412a130d250788f75ecae8cf389d4ae/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/7add7e0fc81bff3795f73e33d1720ecb91fd08a4a482ecfd26e838bfabd305f6/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/c61a6341e6337ca404669a47c8fff5fe0412a130d250788f75ecae8cf389d4ae/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/c61a6341e6337ca404669a47c8fff5fe0412a130d250788f75ecae8cf389d4ae/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/c61a6341e6337ca404669a47c8fff5fe0412a130d250788f75ecae8cf389d4ae/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/c61a6341e6337ca404669a47c8fff5fe0412a130d250788f75ecae8cf389d4ae,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::c61a6341e6337ca404669a47c8fff5fe0412a130d250788f75ecae8cf389d4ae::off::]"
time="2017-10-26T14:54:22.189683922-05:00" pid=24883 name="cc-runtime" level="error" source="runtime" msg="Container c61a6341e6337ca404669a47c8fff5fe0412a130d250788f75ecae8cf389d4ae not ready or running, cannot send a signal"
time="2017-10-26T14:54:22.460205301-05:00" pid=24897 name="cc-runtime" level="error" source="runtime" msg="Container ID (c61a6341e6337ca404669a47c8fff5fe0412a130d250788f75ecae8cf389d4ae) does not exist"
time="2017-10-26T14:54:32.09023956-05:00" pid=24987 name="cc-runtime" level="info" source="virtcontainers" subsystem="qmp" msg="launching qemu with: [-name pod-fed26d7492f60dddc6adeeb624e7b1f1b89d21afc284e7a6ee0d98f04d8b0a0d -uuid 66656432-3664-3734-3932-663630646464 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/fed26d7492f60dddc6adeeb624e7b1f1b89d21afc284e7a6ee0d98f04d8b0a0d/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/fed26d7492f60dddc6adeeb624e7b1f1b89d21afc284e7a6ee0d98f04d8b0a0d/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=16946M -smp 8,cores=8,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/overlay2/696e2b9326f77e31283f05706000547051cf087da7dcb1cc5f00308d27beb59c/merged,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/fed26d7492f60dddc6adeeb624e7b1f1b89d21afc284e7a6ee0d98f04d8b0a0d/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-18400-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/fed26d7492f60dddc6adeeb624e7b1f1b89d21afc284e7a6ee0d98f04d8b0a0d/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/fed26d7492f60dddc6adeeb624e7b1f1b89d21afc284e7a6ee0d98f04d8b0a0d/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/fed26d7492f60dddc6adeeb624e7b1f1b89d21afc284e7a6ee0d98f04d8b0a0d,security_model=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.58-79.1.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::fed26d7492f60dddc6adeeb624e7b1f1b89d21afc284e7a6ee0d98f04d8b0a0d::off::]"

Proxy logs

Recent proxy problems found in system journal:


time="2017-10-26T14:36:31.474990364-05:00" level=debug msg="{\\\"container\\\":\\\"9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110\\\",\\\"container-pid\\\":\\\"3\\\",\\\"level\\\":\\\"warning\\\",\\\"msg\\\":\\\"Container process no longer exists\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":190,\\\"time\\\":\\\"2017-10-26T19:36:31.45877852Z\\\"}" name=cc-proxy pid=29499 source=qemu vm=9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110
time="2017-10-26T14:36:31.475103337-05:00" level=debug msg="{\\\"container\\\":\\\"9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110\\\",\\\"container-pid\\\":\\\"3\\\",\\\"level\\\":\\\"warning\\\",\\\"msg\\\":\\\"Container process no longer exists\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":190,\\\"time\\\":\\\"2017-10-26T19:36:31.458949603Z\\\"}" name=cc-proxy pid=29499 source=qemu vm=9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110
time="2017-10-26T14:36:31.475530723-05:00" level=error msg="error serving client: write unix /var/run/clear-containers/proxy.sock->@: use of closed network connection" client=1598 name=cc-proxy pid=29499 source=proxy
time="2017-10-26T14:36:32.475605786-05:00" level=debug msg="{\\\"container-pid\\\":\\\"1\\\",\\\"error\\\":\\\"signal: killed\\\",\\\"level\\\":\\\"error\\\",\\\"msg\\\":\\\"Process wait failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":190,\\\"time\\\":\\\"2017-10-26T19:36:32.459355493Z\\\"}" name=cc-proxy pid=29499 source=qemu vm=9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110
time="2017-10-26T14:36:32.475802213-05:00" level=debug msg="{\\\"error\\\":\\\"EOF\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"Sequence has been closed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":190,\\\"sequence\\\":2,\\\"time\\\":\\\"2017-10-26T19:36:32.459587917Z\\\"}" name=cc-proxy pid=29499 source=qemu vm=9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110
time="2017-10-26T14:36:32.475924383-05:00" level=debug msg="{\\\"error\\\":\\\"EOF\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"Sequence has been closed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":190,\\\"sequence\\\":1,\\\"time\\\":\\\"2017-10-26T19:36:32.459773639Z\\\"}" name=cc-proxy pid=29499 source=qemu vm=9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110
time="2017-10-26T14:36:32.478377524-05:00" level=debug msg="{\\\"container\\\":\\\"pause-container\\\",\\\"container-pid\\\":\\\"1\\\",\\\"level\\\":\\\"warning\\\",\\\"msg\\\":\\\"Container process no longer exists\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":190,\\\"time\\\":\\\"2017-10-26T19:36:32.462238241Z\\\"}" name=cc-proxy pid=29499 source=qemu vm=9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110
time="2017-10-26T14:36:32.478398528-05:00" level=debug msg="{\\\"container\\\":\\\"pause-container\\\",\\\"container-pid\\\":\\\"1\\\",\\\"level\\\":\\\"warning\\\",\\\"msg\\\":\\\"Container process no longer exists\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":190,\\\"time\\\":\\\"2017-10-26T19:36:32.462286417Z\\\"}" name=cc-proxy pid=29499 source=qemu vm=9a7d4f2bca340b476a6da991e6a93ab403ae3c3cda3431c83a42245a5acab110
time="2017-10-26T14:36:43.189145612-05:00" level=info msg="Signal(power failure,0,0)" client=1605 name=cc-proxy pid=29499 source=proxy
time="2017-10-26T14:36:44.188830482-05:00" level=debug msg="{\\\"error\\\":\\\"read /dev/ptmx: input/output error\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"Sequence has been closed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":193,\\\"sequence\\\":3,\\\"time\\\":\\\"2017-10-26T19:36:44.172771832Z\\\"}" name=cc-proxy pid=29499 source=qemu vm=dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247
time="2017-10-26T14:36:44.189394556-05:00" level=debug msg="{\\\"container-pid\\\":\\\"3\\\",\\\"error\\\":\\\"exit status 30\\\",\\\"level\\\":\\\"error\\\",\\\"msg\\\":\\\"Process wait failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":193,\\\"time\\\":\\\"2017-10-26T19:36:44.173029012Z\\\"}" name=cc-proxy pid=29499 source=qemu vm=dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247
time="2017-10-26T14:36:44.190019477-05:00" level=debug msg="{\\\"container\\\":\\\"dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247\\\",\\\"container-pid\\\":\\\"3\\\",\\\"level\\\":\\\"warning\\\",\\\"msg\\\":\\\"Container process no longer exists\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":193,\\\"time\\\":\\\"2017-10-26T19:36:44.174115065Z\\\"}" name=cc-proxy pid=29499 source=qemu vm=dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247
time="2017-10-26T14:36:44.190490592-05:00" level=debug msg="{\\\"container\\\":\\\"dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247\\\",\\\"container-pid\\\":\\\"3\\\",\\\"level\\\":\\\"warning\\\",\\\"msg\\\":\\\"Container process no longer exists\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":193,\\\"time\\\":\\\"2017-10-26T19:36:44.174379652Z\\\"}" name=cc-proxy pid=29499 source=qemu vm=dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247
time="2017-10-26T14:36:44.19058423-05:00" level=error msg="error serving client: write unix /var/run/clear-containers/proxy.sock->@: use of closed network connection" client=1605 name=cc-proxy pid=29499 source=proxy
time="2017-10-26T14:36:45.223761196-05:00" level=debug msg="{\\\"container-pid\\\":\\\"1\\\",\\\"error\\\":\\\"signal: killed\\\",\\\"level\\\":\\\"error\\\",\\\"msg\\\":\\\"Process wait failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":193,\\\"time\\\":\\\"2017-10-26T19:36:45.207902546Z\\\"}" name=cc-proxy pid=29499 source=qemu vm=dfc2337b1a9ad97eab5bd9fdc339738bce978247e0a9cf4cc9aaa4f078a21247
time="2017-10-26T14:36:45.22420
jcvenegas commented 7 years ago

cc shim logs shows that proxy return an error.

time="2017-10-26T14:55:07.392943628-0500" level="debug" pid=25034 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:6, payload:{\"signalNumber\": 28, \"rows\":0, \"columns\":0}"
time="2017-10-26T14:55:07.393403655-0500" level="debug" pid=25034 function="read_frame" line=495 source="shim" name="cc-shim" msg="Frame read - HeaderLength: 3, Version: 2, Payloadlen: 62, Type:1, Opcode: 6, Err:16"
time="2017-10-26T14:55:07.393418323-0500" level="debug" pid=25034 function="handle_proxy_message" line=737 source="shim" name="cc-shim" msg="Received frame with type: 1, opcode: 6"
time="2017-10-26T14:55:07.393442110-0500" level="error" pid=25034 function="handle_proxy_response" line=616 source="shim" name="cc-shim" msg="Error response received from proxy at /var/run/clear-containers/proxy.sock: {\"msg\":\"received SIGWINCH but terminal size is invalid (0,0)\"}"

cc-proxy is not allowing to send that signal because of its values:

https://github.com/clearcontainers/proxy/blob/master/proxy.go#L451