search

clearcontainers / runtime

OCI (Open Containers Initiative) compatible runtime using Virtual Machines
Apache License 2.0
590 stars 70 forks source link

fedora: cc-runtime list stops working and qemu and CC processes are left behind after running swarm #902

Open chavafg opened 6 years ago

chavafg commented 6 years ago

Description of problem

After swarm tests are executed in the CI, the cc-runtime list command does not work and there are qemu, cc-proxy and cc-shim processes left behind.

Expected result

cc-runtime list should work no processes should left behind.

Actual result

Swarm tests executed: 

[fuentess@ci-failure tests]$ sudo -E PATH=$PATH make swarm            
bats integration/swarm/swarm.bats                                                                                                                                                                                  
1..4                                                                                                                         
ok 1 check_replicas_interfaces                                                                                                                                                                                     
ok 2 # skip (This is not working (https://github.com/clearcontainers/tests/issues/694)) check_service_ip_among_the_replicas  
ok 3 # skip (This is not working (https://github.com/clearcontainers/runtime/issues/771)) obtain hostname of the replicas
ok 4 # skip (This is not working (https://github.com/clearcontainers/tests/issues/714)) check mtu values in different interfaces

cc-runtime list failure:

[fuentess@ci-failure tests]$ sudo -E PATH=$PATH /usr/local/bin/cc-runtime list                                                                                                                                     
stat /var/lib/docker/overlay2/892032c0d059fbb32ba07497cfe5c74d6a16506955400d8532e3c150f8f80723/merged: no such file or directory

docker ps -a:

[fuentess@ci-failure tests]$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[fuentess@ci-failure tests]$ sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

processes left:

root     115728 115365  1 15:48 ?        00:00:06 /usr/bin/qemu-lite-system-x86_64 -name pod-e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b -uuid 284021fc-558a-4e31-af5a-8ccbeefadd9f -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b/284021fc-558a-4e3,server,nowait -qmp unix:/run/virtcontainers/pods/e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b/284021fc-558a-4e3,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b/tty.sock,server,nowait -device virtio-9p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:12:00:05,disable-modern=true,mq=on,vectors=18 -netdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:0a:ff:00:06,disable-modern=true,mq=on,vectors=18 -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b::off:: -smp 2,cores=2,threads=1,sockets=1
root     115752 115365  0 15:48 ?        00:00:00 /usr/libexec/clear-containers/cc-proxy -uri unix:///run/virtcontainers/pods/e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b/proxy.sock -log debug
root     115860 115365  0 15:49 ?        00:00:00 /usr/libexec/clear-containers/cc-shim -c e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b -t Zp4jZ_zQuAhgX6rPiL3FMInse2vRU3_aMAWdsqKljfc= -u unix:///run/virtcontainers/pods/e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b/proxy.sock -d
root     115861 115860  0 15:49 ?        00:00:00 /usr/libexec/clear-containers/cc-shim -c e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b -t Zp4jZ_zQuAhgX6rPiL3FMInse2vRU3_aMAWdsqKljfc= -u unix:///run/virtcontainers/pods/e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b/proxy.sock -d

[fuentess@ci-failure tests]$ sudo -E PATH=$PATH cc-collect-data.sh

Meta details

Running cc-collect-data.sh version 3.0.12 (commit be7d1cebb5e8afbb2a9329264d4163adc70b0362) at 2018-01-08.16:00:02.709520618+0000.

Runtime is /usr/local/bin/cc-runtime.

cc-env

Output of "/usr/local/bin/cc-runtime cc-env": 

[Meta]                                                                                                                                                                                                             
  Version = "1.0.6"                                                                                                                                                                                                

[Runtime]                                                                                                                                                                                                          
  Debug = false                                                                                                                                                                                                    
  [Runtime.Version]                                                                                                                                                                                                
    Semver = "3.0.12"                                                                                                                                                                                              
    Commit = "be7d1cebb5e8afbb2a9329264d4163adc70b0362"                                                                                                                                                            
    OCI = "1.0.0-dev"                                                                                                                                                                                              
  [Runtime.Config]                                                                                                                                                                                                 
    Path = "/usr/share/defaults/clear-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.7.0, Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"
  Debug = false

[Image]
  Path = "/usr/share/clear-containers/clear-19790-containers.img"

[Kernel]
  Path = "/usr/share/clear-containers/vmlinuz-4.9.60-83.container"
  Parameters = ""

[Proxy]
  Type = "ccProxy"
  Version = "Version: 3.0.12-3c6daa636ebb7c981f9449f185a003f7cf9b748e"
  Path = "/usr/libexec/clear-containers/cc-proxy"
  Debug = true

[Shim]
  Type = "ccShim"
  Version = "shim version: 3.0.12 (commit: d01f9a71056fc005a98fbaeba733c7b3b07874f6)"
  Path = "/usr/libexec/clear-containers/cc-shim"
  Debug = true

[Agent]
  Type = "hyperstart"
  Version = "<<unknown>>"

[Host]
  Kernel = "4.11.8-300.fc26.x86_64"
  CCCapable = true
  [Host.Distro]
    Name = "Fedora"
    Version = "26"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz"

Runtime config files

Runtime default config files

/etc/clear-containers/configuration.toml
/usr/share/defaults/clear-containers/configuration.toml

Runtime config file contents

Config file /etc/clear-containers/configuration.toml not found Output of "cat "/usr/share/defaults/clear-containers/configuration.toml"":

# XXX: Warning: this file is auto-generated from file "config/configuration.toml.in".

[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
kernel = "/usr/share/clear-containers/vmlinuz.container"
image = "/usr/share/clear-containers/clear-containers.img"
machine_type = "pc"
# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc
kernel_params = ""

# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""

# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""

# Default number of vCPUs per POD/VM:
# unspecified or 0 --> will be set to 1
# < 0              --> will be set to the actual number of physical cores
# > 0 <= 255       --> will be set to the specified number
# > 255            --> will be set to 255
default_vcpus = -1

# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
#   This limitation could be a bug in qemu or in the kernel
# Default number of bridges per POD/VM:
# unspecified or 0   --> will be set to 1
# > 1 <= 5           --> will be set to the specified number
# > 5                --> will be set to 5
default_bridges = 1

# Default memory size in MiB for POD/VM.
# If unspecified then it will be set 2048 MiB.
#default_memory = 2048
disable_block_device_use = false

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
#enable_hugepages = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# Debug changes the default hypervisor and kernel parameters to
# enable debug output where available.
# Default false
# these logs can be obtained in the cc-proxy logs  when the
# proxy is set to run in debug mode
# /usr/libexec/clear-containers/cc-proxy -log debug
# or by stopping the cc-proxy service and running the cc-proxy
# explicitly using the same command line
#
enable_debug = true

# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true

[proxy.cc]
path = "/usr/libexec/clear-containers/cc-proxy"

# If enabled, proxy messages will be sent to the system log
# (default: disabled)
enable_debug = true

[shim.cc]
path = "/usr/libexec/clear-containers/cc-shim"

# If enabled, shim messages will be sent to the system log
# (default: disabled)
enable_debug = true

[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
enable_debug = true

Logfiles

Runtime logs

Recent runtime problems found in system journal:

time="2018-01-08T15:47:33Z" level=info msg="launching qemu with: [-name pod-7daaf956cd029a297fb3f57d0c06fb9b5c9f27e05b57e6953bbbf0932e8ba0cf -uuid 1941e57d-4070-4e48-a542-5b2b1d516070 -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/7daaf956cd029a297fb3f57d0c06fb9b5c9f27e05b57e6953bbbf0932e8ba0cf/1941e57d-4070-4e4,server,nowait -qmp unix:/run/virtcontainers/pods/7daaf956c
d029a297fb3f57d0c06fb9b5c9f27e05b57e6953bbbf0932e8ba0cf/1941e57d-4070-4e4,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/7daaf956cd029a297fb3f57d0c06fb9b5c9f27e05b57e6953bbbf0932e8ba0cf/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/7daaf956cd029a297fb3f57d0c06fb9b5c9f27e05b57e6953bbbf0932e8ba0cf/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/7daaf956cd029a297fb3f57d0c06fb9b5c9f27e05b57e6953bbbf0932e8ba0cf/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/7daaf956cd029a297fb3f57d0c06fb9b5c9f27e05b57e6953bbbf0932e8ba0cf,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:07,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:04,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::7daaf956cd029a297fb3f57d0c06fb9b5c9f27e05b57e6953bbbf0932e8ba0cf::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:47:34Z" level=info msg="launching qemu with: [-name pod-02e96d1d2b5bab93bd1da47df44dc311367819b349254b524511a45d32c18cdb -uuid ae4a99a3-efaa-4804-a1c4-1265ababf6f2 -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/02e96d1d2b5bab93bd1da47df44dc311367819b349254b524511a45d32c18cdb/ae4a99a3-efaa-480,server,nowait -qmp unix:/run/virtcontainers/pods/02e96d1d2
b5bab93bd1da47df44dc311367819b349254b524511a45d32c18cdb/ae4a99a3-efaa-480,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/02e96d1d2b5bab93bd1da47df44dc311367819b349254b524511a45d32c18cdb/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/02e96d1d2b5bab93bd1da47df44dc311367819b349254b524511a45d32c18cdb/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/02e96d1d2b5bab93bd1da47df44dc311367819b349254b524511a45d32c18cdb/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/02e96d1d2b5bab93bd1da47df44dc311367819b349254b524511a45d32c18cdb,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:05,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:06,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::02e96d1d2b5bab93bd1da47df44dc311367819b349254b524511a45d32c18cdb::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:47:35Z" level=info msg="launching qemu with: [-name pod-41b4f7a2a2b8409025c1de1e5338ec3a731d20be998b176b6855c5117153bab7 -uuid 86613c84-7ac0-4009-bbe8-8fcc2bffb27a -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/41b4f7a2a2b8409025c1de1e5338ec3a731d20be998b176b6855c5117153bab7/86613c84-7ac0-400,server,nowait -qmp unix:/run/virtcontainers/pods/41b4f7a2a
2b8409025c1de1e5338ec3a731d20be998b176b6855c5117153bab7/86613c84-7ac0-400,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/41b4f7a2a2b8409025c1de1e5338ec3a731d20be998b176b6855c5117153bab7/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/41b4f7a2a2b8409025c1de1e5338ec3a731d20be998b176b6855c5117153bab7/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/41b4f7a2a2b8409025c1de1e5338ec3a731d20be998b176b6855c5117153bab7/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/41b4f7a2a2b8409025c1de1e5338ec3a731d20be998b176b6855c5117153bab7,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:06,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:05,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::41b4f7a2a2b8409025c1de1e5338ec3a731d20be998b176b6855c5117153bab7::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:47:51Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:47:51Z" level=error msg="Container 02e96d1d2b5bab93bd1da47df44dc311367819b349254b524511a45d32c18cdb not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:47:52Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:47:52Z" level=error msg="Container 41b4f7a2a2b8409025c1de1e5338ec3a731d20be998b176b6855c5117153bab7 not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:48:03Z" level=error msg="Container 8231fa023f3cd96a8ce9f2dd1c81def5fa19e4b2883deb8f9996513b5d72f10a not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:48:03Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:48:03Z" level=error msg="Container 7daaf956cd029a297fb3f57d0c06fb9b5c9f27e05b57e6953bbbf0932e8ba0cf not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:48:03Z" level=error msg="Container 8231fa023f3cd96a8ce9f2dd1c81def5fa19e4b2883deb8f9996513b5d72f10a not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:48:15Z" level=info msg="launching qemu with: [-name pod-714e47bf80d8ad39ab720f2bcb40a6cab5eb943221bfbd6e0148a47ca5c44006 -uuid 7f723a36-f5f3-40bc-82ad-33edda084a99 -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/714e47bf80d8ad39ab720f2bcb40a6cab5eb943221bfbd6e0148a47ca5c44006/7f723a36-f5f3-40b,server,nowait -qmp unix:/run/virtcontainers/pods/714e47bf8
0d8ad39ab720f2bcb40a6cab5eb943221bfbd6e0148a47ca5c44006/7f723a36-f5f3-40b,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/714e47bf80d8ad39ab720f2bcb40a6cab5eb943221bfbd6e0148a47ca5c44006/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/714e47bf80d8ad39ab720f2bcb40a6cab5eb943221bfbd6e0148a47ca5c44006/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/714e47bf80d8ad39ab720f2bcb40a6cab5eb943221bfbd6e0148a47ca5c44006/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/714e47bf80d8ad39ab720f2bcb40a6cab5eb943221bfbd6e0148a47ca5c44006,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:07,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:04,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::714e47bf80d8ad39ab720f2bcb40a6cab5eb943221bfbd6e0148a47ca5c44006::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:48:17Z" level=info msg="launching qemu with: [-name pod-f2dab6836bbc473114aa33404c294f7760d4dba9783dfda7ca6d5674fd33e594 -uuid 17150882-7fe1-4de3-9ca6-38ac6c9f69fc -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/f2dab6836bbc473114aa33404c294f7760d4dba9783dfda7ca6d5674fd33e594/17150882-7fe1-4de,server,nowait -qmp unix:/run/virtcontainers/pods/f2dab6836
bbc473114aa33404c294f7760d4dba9783dfda7ca6d5674fd33e594/17150882-7fe1-4de,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/f2dab6836bbc473114aa33404c294f7760d4dba9783dfda7ca6d5674fd33e594/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/f2dab6836bbc473114aa33404c294f7760d4dba9783dfda7ca6d5674fd33e594/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/f2dab6836bbc473114aa33404c294f7760d4dba9783dfda7ca6d5674fd33e594/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/f2dab6836bbc473114aa33404c294f7760d4dba9783dfda7ca6d5674fd33e594,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:05,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:05,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::f2dab6836bbc473114aa33404c294f7760d4dba9783dfda7ca6d5674fd33e594::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:48:17Z" level=info msg="launching qemu with: [-name pod-73031f80f579e97f0d5ae64f424d935f7901d787a160409994b5c41453cb41a0 -uuid 42e3ac78-649e-4d50-9e7a-e462895f8a48 -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/73031f80f579e97f0d5ae64f424d935f7901d787a160409994b5c41453cb41a0/42e3ac78-649e-4d5,server,nowait -qmp unix:/run/virtcontainers/pods/73031f80f
579e97f0d5ae64f424d935f7901d787a160409994b5c41453cb41a0/42e3ac78-649e-4d5,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/73031f80f579e97f0d5ae64f424d935f7901d787a160409994b5c41453cb41a0/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/73031f80f579e97f0d5ae64f424d935f7901d787a160409994b5c41453cb41a0/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/73031f80f579e97f0d5ae64f424d935f7901d787a160409994b5c41453cb41a0/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/73031f80f579e97f0d5ae64f424d935f7901d787a160409994b5c41453cb41a0,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:06,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:03,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::73031f80f579e97f0d5ae64f424d935f7901d787a160409994b5c41453cb41a0::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:48:18Z" level=info msg="launching qemu with: [-name pod-45866763e60d361ed7c27bdb769971a7eba907b471b20c5b4b28361a343e156f -uuid 06acc96f-1071-4f7d-b1fb-aedef1870a18 -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/45866763e60d361ed7c27bdb769971a7eba907b471b20c5b4b28361a343e156f/06acc96f-1071-4f7,server,nowait -qmp unix:/run/virtcontainers/pods/45866763e
60d361ed7c27bdb769971a7eba907b471b20c5b4b28361a343e156f/06acc96f-1071-4f7,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/45866763e60d361ed7c27bdb769971a7eba907b471b20c5b4b28361a343e156f/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/45866763e60d361ed7c27bdb769971a7eba907b471b20c5b4b28361a343e156f/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/45866763e60d361ed7c27bdb769971a7eba907b471b20c5b4b28361a343e156f/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/45866763e60d361ed7c27bdb769971a7eba907b471b20c5b4b28361a343e156f,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:04,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:06,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::45866763e60d361ed7c27bdb769971a7eba907b471b20c5b4b28361a343e156f::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:48:34Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:48:34Z" level=error msg="Container f2dab6836bbc473114aa33404c294f7760d4dba9783dfda7ca6d5674fd33e594 not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:48:36Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:48:36Z" level=error msg="Container 45866763e60d361ed7c27bdb769971a7eba907b471b20c5b4b28361a343e156f not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:48:37Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:48:37Z" level=error msg="Container 714e47bf80d8ad39ab720f2bcb40a6cab5eb943221bfbd6e0148a47ca5c44006 not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:48:38Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:48:38Z" level=error msg="Container 73031f80f579e97f0d5ae64f424d935f7901d787a160409994b5c41453cb41a0 not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:48:49Z" level=info msg="launching qemu with: [-name pod-761e8712618b6720569c460f73cb3a833e96be64e33f99a8f09184a2a7046be1 -uuid baeefc22-e650-472e-9bea-bfccfa637bdd -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/761e8712618b6720569c460f73cb3a833e96be64e33f99a8f09184a2a7046be1/baeefc22-e650-472,server,nowait -qmp unix:/run/virtcontainers/pods/761e87126
18b6720569c460f73cb3a833e96be64e33f99a8f09184a2a7046be1/baeefc22-e650-472,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/761e8712618b6720569c460f73cb3a833e96be64e33f99a8f09184a2a7046be1/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/761e8712618b6720569c460f73cb3a833e96be64e33f99a8f09184a2a7046be1/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/761e8712618b6720569c460f73cb3a833e96be64e33f99a8f09184a2a7046be1/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/761e8712618b6720569c460f73cb3a833e96be64e33f99a8f09184a2a7046be1,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:05,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:04,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::761e8712618b6720569c460f73cb3a833e96be64e33f99a8f09184a2a7046be1::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:48:49Z" level=info msg="launching qemu with: [-name pod-e5e1e4cded53aa744f442bb43f03488b32b2c200597443491461d1a2543d99d8 -uuid ad4eb5d7-397f-4cda-b3c6-52639104f000 -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/e5e1e4cded53aa744f442bb43f03488b32b2c200597443491461d1a2543d99d8/ad4eb5d7-397f-4cd,server,nowait -qmp unix:/run/virtcontainers/pods/e5e1e4cde
d53aa744f442bb43f03488b32b2c200597443491461d1a2543d99d8/ad4eb5d7-397f-4cd,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/e5e1e4cded53aa744f442bb43f03488b32b2c200597443491461d1a2543d99d8/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/e5e1e4cded53aa744f442bb43f03488b32b2c200597443491461d1a2543d99d8/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/e5e1e4cded53aa744f442bb43f03488b32b2c200597443491461d1a2543d99d8/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/e5e1e4cded53aa744f442bb43f03488b32b2c200597443491461d1a2543d99d8,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:04,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:06,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::e5e1e4cded53aa744f442bb43f03488b32b2c200597443491461d1a2543d99d8::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:48:50Z" level=info msg="launching qemu with: [-name pod-70004f0e82930784590e630aae54d5ed57f2054ad6fde0f96e985c084ae28450 -uuid 700384d2-bf96-46c8-bfa7-7a5bf53b1f50 -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/70004f0e82930784590e630aae54d5ed57f2054ad6fde0f96e985c084ae28450/700384d2-bf96-46c,server,nowait -qmp unix:/run/virtcontainers/pods/70004f0e8
2930784590e630aae54d5ed57f2054ad6fde0f96e985c084ae28450/700384d2-bf96-46c,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/70004f0e82930784590e630aae54d5ed57f2054ad6fde0f96e985c084ae28450/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/70004f0e82930784590e630aae54d5ed57f2054ad6fde0f96e985c084ae28450/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/70004f0e82930784590e630aae54d5ed57f2054ad6fde0f96e985c084ae28450/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/70004f0e82930784590e630aae54d5ed57f2054ad6fde0f96e985c084ae28450,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:07,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:03,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::70004f0e82930784590e630aae54d5ed57f2054ad6fde0f96e985c084ae28450::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:48:52Z" level=info msg="launching qemu with: [-name pod-e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b -uuid 284021fc-558a-4e31-af5a-8ccbeefadd9f -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b/284021fc-558a-4e3,server,nowait -qmp unix:/run/virtcontainers/pods/e09effab7
0bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b/284021fc-558a-4e3,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:12:00:05,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:0a:ff:00:06,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::e09effab70bf0787bcb237de5d07cbd8342266f5ae2246f3eee4eb2fadd1496b::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:49:02Z" level=error msg="ERROR received from VM agent, control msg received : Could not setup the network: Could not setup network interfaces: file exists" source=runtime
time="2018-01-08T15:49:10Z" level=info msg="launching qemu with: [-name pod-260709dd46a6b67edad92e77846c813c28d0190ed89b76042f9d422bce18bb28 -uuid 056401b7-fc8f-4c52-953d-62d8e2642d38 -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/260709dd46a6b67edad92e77846c813c28d0190ed89b76042f9d422bce18bb28/056401b7-fc8f-4c5,server,nowait -qmp unix:/run/virtcontainers/pods/260709dd4
6a6b67edad92e77846c813c28d0190ed89b76042f9d422bce18bb28/056401b7-fc8f-4c5,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/260709dd46a6b67edad92e77846c813c28d0190ed89b76042f9d422bce18bb28/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/260709dd46a6b67edad92e77846c813c28d0190ed89b76042f9d422bce18bb28/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/260709dd46a6b67edad92e77846c813c28d0190ed89b76042f9d422bce18bb28/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/260709dd46a6b67edad92e77846c813c28d0190ed89b76042f9d422bce18bb28,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:08,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:05,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::260709dd46a6b67edad92e77846c813c28d0190ed89b76042f9d422bce18bb28::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:49:18Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:49:18Z" level=error msg="Container 260709dd46a6b67edad92e77846c813c28d0190ed89b76042f9d422bce18bb28 not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:49:19Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:49:20Z" level=error msg="Container e5e1e4cded53aa744f442bb43f03488b32b2c200597443491461d1a2543d99d8 not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:49:21Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:49:21Z" level=error msg="Container 761e8712618b6720569c460f73cb3a833e96be64e33f99a8f09184a2a7046be1 not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:49:24Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:49:24Z" level=error msg="Container 70004f0e82930784590e630aae54d5ed57f2054ad6fde0f96e985c084ae28450 not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:49:35Z" level=info msg="launching qemu with: [-name pod-dbefcd9c91e90e2b16bfa1238ea6a427ddfcc2440d2121ff9874bfb7c871d6d9 -uuid 908a9a83-1b51-4a1f-bece-885dfe87f857 -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/dbefcd9c91e90e2b16bfa1238ea6a427ddfcc2440d2121ff9874bfb7c871d6d9/908a9a83-1b51-4a1,server,nowait -qmp unix:/run/virtcontainers/pods/dbefcd9c9
1e90e2b16bfa1238ea6a427ddfcc2440d2121ff9874bfb7c871d6d9/908a9a83-1b51-4a1,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/dbefcd9c91e90e2b16bfa1238ea6a427ddfcc2440d2121ff9874bfb7c871d6d9/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/dbefcd9c91e90e2b16bfa1238ea6a427ddfcc2440d2121ff9874bfb7c871d6d9/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/dbefcd9c91e90e2b16bfa1238ea6a427ddfcc2440d2121ff9874bfb7c871d6d9/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/dbefcd9c91e90e2b16bfa1238ea6a427ddfcc2440d2121ff9874bfb7c871d6d9,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:07,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:03,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::dbefcd9c91e90e2b16bfa1238ea6a427ddfcc2440d2121ff9874bfb7c871d6d9::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:49:36Z" level=info msg="launching qemu with: [-name pod-4a6cdd022a982a7e07bf61cc199f2fff37a31ba639918671fc4b64739dd4f88f -uuid 77f9ce83-9a87-4f96-adc5-e4fe3458af92 -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/4a6cdd022a982a7e07bf61cc199f2fff37a31ba639918671fc4b64739dd4f88f/77f9ce83-9a87-4f9,server,nowait -qmp unix:/run/virtcontainers/pods/4a6cdd022
a982a7e07bf61cc199f2fff37a31ba639918671fc4b64739dd4f88f/77f9ce83-9a87-4f9,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/4a6cdd022a982a7e07bf61cc199f2fff37a31ba639918671fc4b64739dd4f88f/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/4a6cdd022a982a7e07bf61cc199f2fff37a31ba639918671fc4b64739dd4f88f/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/4a6cdd022a982a7e07bf61cc199f2fff37a31ba639918671fc4b64739dd4f88f/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/4a6cdd022a982a7e07bf61cc199f2fff37a31ba639918671fc4b64739dd4f88f,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:04,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:04,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::4a6cdd022a982a7e07bf61cc199f2fff37a31ba639918671fc4b64739dd4f88f::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:49:37Z" level=info msg="launching qemu with: [-name pod-49c70e05bfc112d29198472ef1d0ff4d5cc11257015e74e2dd86c47e82ed3aa8 -uuid 7d3b6c1d-d829-476a-836b-118f9fac14ba -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/49c70e05bfc112d29198472ef1d0ff4d5cc11257015e74e2dd86c47e82ed3aa8/7d3b6c1d-d829-476,server,nowait -qmp unix:/run/virtcontainers/pods/49c70e05b
fc112d29198472ef1d0ff4d5cc11257015e74e2dd86c47e82ed3aa8/7d3b6c1d-d829-476,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/49c70e05bfc112d29198472ef1d0ff4d5cc11257015e74e2dd86c47e82ed3aa8/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/49c70e05bfc112d29198472ef1d0ff4d5cc11257015e74e2dd86c47e82ed3aa8/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/49c70e05bfc112d29198472ef1d0ff4d5cc11257015e74e2dd86c47e82ed3aa8/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/49c70e05bfc112d29198472ef1d0ff4d5cc11257015e74e2dd86c47e82ed3aa8,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:06,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:05,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::49c70e05bfc112d29198472ef1d0ff4d5cc11257015e74e2dd86c47e82ed3aa8::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:49:38Z" level=info msg="launching qemu with: [-name pod-699f5cc51b188a422cb385d0264f736cc2c34d36572631f97a9ee0b744804fb8 -uuid 3028b683-6495-4c14-a33b-8fe31e2163e6 -machine pc,accel=kvm,kerne
l_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/virtcontainers/pods/699f5cc51b188a422cb385d0264f736cc2c34d36572631f97a9ee0b744804fb8/3028b683-6495-4c1,server,nowait -qmp unix:/run/virtcontainers/pods/699f5cc51
b188a422cb385d0264f736cc2c34d36572631f97a9ee0b744804fb8/3028b683-6495-4c1,server,nowait -m 2048M,slots=2,maxmem=8988M -device virtio-serial-pci,disable-modern=true,id=serial0 -device virtconsole,chardev=charcons
ole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/699f5cc51b188a422cb385d0264f736cc2c34d36572631f97a9ee0b744804fb8/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -objec
t memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-19790-containers.img,size=235929600 -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on -device virtserialport,chardev=charc
h0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/699f5cc51b188a422cb385d0264f736cc2c34d36572631f97a9ee0b744804fb8/hyper.sock,server,nowait -device virtserialport,ch
ardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/699f5cc51b188a422cb385d0264f736cc2c34d36572631f97a9ee0b744804fb8/tty.sock,server,nowait -device virtio-9
p-pci,disable-modern=true,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/run/hyper/shared/pods/699f5cc51b188a422cb385d0264f736cc2c34d36572631f97a9ee0b744804fb8,securi
ty_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:0a:ff:00:05,disable-modern=true,mq=on,vectors=18 -ne
tdev tap,id=network-1,vhost=on,vhostfds=19:20:21:22:23:24:25:26,fds=27:28:29:30:31:32:33:34 -device driver=virtio-net-pci,netdev=network-1,mac=02:42:ac:12:00:06,disable-modern=true,mq=on,vectors=18 -rtc base=utc
,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz-4.9.60-83.container -append root=/dev/pmem0p1 rootf
lags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=h
vc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=sy
stemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::699f5cc51b188a422cb385d0264f736cc2c34d36572631f97a9ee0b744804fb8::off:: -smp 2,cores=2,threads=1,sockets=1]" source=virtcontainers subsystem=q
mp
time="2018-01-08T15:49:53Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:49:53Z" level=error msg="Container 699f5cc51b188a422cb385d0264f736cc2c34d36572631f97a9ee0b744804fb8 not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:49:56Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:49:56Z" level=error msg="Container 49c70e05bfc112d29198472ef1d0ff4d5cc11257015e74e2dd86c47e82ed3aa8 not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:49:58Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:49:58Z" level=error msg="Container 4a6cdd022a982a7e07bf61cc199f2fff37a31ba639918671fc4b64739dd4f88f not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:49:59Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:49:59Z" level=error msg="Container dbefcd9c91e90e2b16bfa1238ea6a427ddfcc2440d2121ff9874bfb7c871d6d9 not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:50:07Z" level=error msg="stat /var/lib/docker/overlay2/892032c0d059fbb32ba07497cfe5c74d6a16506955400d8532e3c150f8f80723/merged: no such file or directory" source=runtime

Proxy logs

No recent proxy problems found in system journal.

Shim logs

Recent shim problems found in system journal:

time="2018-01-08T15:33:27.251245169+0000" level="debug" pid=105216 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:33:32.136866325+0000" level="debug" pid=105388 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:33:37.155699610+0000" level="debug" pid=105549 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:33:52.013310056+0000" level="debug" pid=106023 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:33:57.121438120+0000" level="debug" pid=106189 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:34:02.187767396+0000" level="debug" pid=106352 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:34:07.466832148+0000" level="debug" pid=106514 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:34:23.247067973+0000" level="debug" pid=106989 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:34:28.017834885+0000" level="debug" pid=107140 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:34:33.186263722+0000" level="debug" pid=107293 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:34:38.323853184+0000" level="debug" pid=107455 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:34:54.757623073+0000" level="debug" pid=107936 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:34:59.215842211+0000" level="debug" pid=108093 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:35:04.195285268+0000" level="debug" pid=108256 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:35:08.991319950+0000" level="debug" pid=108418 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:35:40.464930180+0000" level="debug" pid=108888 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:35:44.471933806+0000" level="debug" pid=109014 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:35:48.720182621+0000" level="debug" pid=109143 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:35:53.131810152+0000" level="debug" pid=109279 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:35:57.179384387+0000" level="debug" pid=109466 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:36:00.345312344+0000" level="debug" pid=109576 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:36:03.981818082+0000" level="debug" pid=109726 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:36:23.209254672+0000" level="debug" pid=110449 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:36:26.824886066+0000" level="debug" pid=110569 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:36:30.512234738+0000" level="debug" pid=110687 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:36:34.198649726+0000" level="debug" pid=110806 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:36:52.145528135+0000" level="debug" pid=111383 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:36:56.611888734+0000" level="debug" pid=111534 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:37:00.296050760+0000" level="debug" pid=111661 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:37:03.695687135+0000" level="debug" pid=111784 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:47:50.591763892+0000" level="debug" pid=113157 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:47:50.767529000+0000" level="debug" pid=113191 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:47:50.953869627+0000" level="debug" pid=113224 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:47:51.133952461+0000" level="debug" pid=113254 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:47:51.346454042+0000" level="debug" pid=113049 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:47:52.660105675+0000" level="debug" pid=113050 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:48:03.402869060+0000" level="debug" pid=112988 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:48:03.435063212+0000" level="debug" pid=112968 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:48:34.552020388+0000" level="debug" pid=114521 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:48:35.870201044+0000" level="debug" pid=114527 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:48:37.322521315+0000" level="debug" pid=114467 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:48:38.543824320+0000" level="debug" pid=114509 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:49:18.699069003+0000" level="debug" pid=116092 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:49:19.868168097+0000" level="debug" pid=115782 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:49:21.323255722+0000" level="debug" pid=115821 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:49:24.586463160+0000" level="debug" pid=115840 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:49:32.499351728+0000" level="warning" pid=115860 function="reconnect_to_proxy" line=1020 source="shim" name="cc-shim" msg="Reconnecting to cc-proxy (timeout 10 s)"
time="2018-01-08T15:49:53.399632062+0000" level="debug" pid=117411 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:49:56.735034071+0000" level="debug" pid=117394 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-01-08T15:49:57.968999461+0000" level="debug" pid=117382 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"

Container manager details

Have docker

Docker

Output of "docker version":

Client:
 Version:       17.12.0-ce
 API version:   1.35
 Go version:    go1.9.2
 Git commit:    c97c6d6
 Built: Wed Dec 27 20:12:20 2017
 OS/Arch:       linux/amd64

Server:
 Engine:
  Version:      17.12.0-ce
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.9.2
  Git commit:   c97c6d6
  Built:        Wed Dec 27 20:14:56 2017
  OS/Arch:      linux/amd64
  Experimental: false

Output of "docker info":

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 6
Server Version: 17.12.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: cc-runtime runc
Default Runtime: cc-runtime
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: be7d1cebb5e8afbb2a9329264d4163adc70b0362 (expected: b2567b37d7b75eb4cf325b77297b140ea686ce8f)
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.11.8-300.fc26.x86_64
Operating System: Fedora 26 (Cloud Edition)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.778GiB
Name: ci-failure.localdomain
ID: C4LH:LOVD:5XNV:SJ4Y:33GE:6HFW:4KLY:BK77:YDMH:MM7I:UYJX:KWWG
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 20
 Goroutines: 34
 System Time: 2018-01-08T16:00:03.02933017Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Output of "systemctl show docker":

Type=notify
Restart=on-failure
NotifyAccess=main
RestartUSec=100ms
TimeoutStartUSec=infinity
TimeoutStopUSec=1min 30s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestamp=Mon 2018-01-08 14:35:22 UTC
WatchdogTimestampMonotonic=450603331
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=16385
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
UID=4294967295
GID=4294967295
ExecMainStartTimestamp=Mon 2018-01-08 14:35:21 UTC
ExecMainStartTimestampMonotonic=449538635
ExecMainExitTimestampMonotonic=0
ExecMainPID=16385
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -D --add-runtime cc-runtime=/usr/local/bin/cc-runtime --default-runtime=cc-runtime ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; co
de=(null) ; status=0/0 }
ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/docker.service
MemoryCurrent=1050226688
CPUUsageNSec=474323198111
TasksCurrent=63
Delegate=yes
CPUAccounting=no
CPUWeight=18446744073709551615
StartupCPUWeight=18446744073709551615
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
IOAccounting=no
IOWeight=18446744073709551615
StartupIOWeight=18446744073709551615
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLow=0
MemoryHigh=18446744073709551615
MemoryMax=18446744073709551615
MemorySwapMax=18446744073709551615
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=yes
TasksMax=19660
UMask=0022
LimitCPU=18446744073709551615
LimitCPUSoft=18446744073709551615
LimitFSIZE=18446744073709551615
LimitFSIZESoft=18446744073709551615
LimitDATA=18446744073709551615
LimitDATASoft=18446744073709551615
LimitSTACK=18446744073709551615
LimitSTACKSoft=8388608
LimitCORE=18446744073709551615
LimitCORESoft=18446744073709551615
LimitRSS=18446744073709551615
LimitRSSSoft=18446744073709551615
LimitNOFILE=18446744073709551615
LimitNOFILESoft=18446744073709551615
LimitAS=18446744073709551615
LimitASSoft=18446744073709551615
LimitNPROC=18446744073709551615
LimitNPROCSoft=18446744073709551615
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=18446744073709551615
LimitLOCKSSoft=18446744073709551615
LimitSIGPENDING=31773
LimitSIGPENDINGSoft=31773
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=18446744073709551615
LimitRTTIMESoft=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
DynamicUser=no
RemoveIPC=no
MountFlags=0
PrivateTmp=no
PrivateDevices=no
ProtectKernelTunables=no
ProtectKernelModules=no
ProtectControlGroups=no
PrivateNetwork=no
PrivateUsers=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
MemoryDenyWriteExecute=no
RestrictRealtime=no
RestrictNamespaces=no
MountAPIVFS=no
KillMode=process
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=docker.service
Names=docker.service
Requires=system.slice sysinit.target
Wants=network-online.target
Conflicts=shutdown.target
Before=shutdown.target
After=firewalld.service basic.target sysinit.target network-online.target systemd-journald.socket system.slice
Documentation=https://docs.docker.com
Description=Docker Application Container Engine
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/usr/lib/systemd/system/docker.service
DropInPaths=/etc/systemd/system/docker.service.d/clear-containers.conf
UnitFileState=disabled
UnitFilePreset=disabled
StateChangeTimestamp=Mon 2018-01-08 14:35:22 UTC
StateChangeTimestampMonotonic=450603334
InactiveExitTimestamp=Mon 2018-01-08 14:35:21 UTC
InactiveExitTimestampMonotonic=449538703
ActiveEnterTimestamp=Mon 2018-01-08 14:35:22 UTC
ActiveEnterTimestampMonotonic=450603334
ActiveExitTimestampMonotonic=0
InactiveEnterTimestampMonotonic=0
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Mon 2018-01-08 14:35:21 UTC
ConditionTimestampMonotonic=449535662
AssertTimestamp=Mon 2018-01-08 14:35:21 UTC
AssertTimestampMonotonic=449535663
Transient=no
Perpetual=no
StartLimitIntervalSec=60000000
StartLimitBurst=3
StartLimitAction=none
InvocationID=3dc3b8cab36b4e98b3aa85e48e5441b0

No kubectl

Packages

No dpkg Have rpm Output of "rpm -qa|egrep "(cc-proxy|cc-runtime|cc-shim|clear-containers-image|linux-container|qemu-lite|qemu-system-x86|cc-oci-runtime)"":

qemu-lite-data-741f430a960b5b67745670e8270db91aeb083c5f-29.x86_64
qemu-lite-bin-741f430a960b5b67745670e8270db91aeb083c5f-29.x86_64
chavafg commented 6 years ago

This is mainly why we have the Openshift test failing on our jenkins CI. Openshift seems to work well, but when checking if there is a CC launched with Openshift using cc-runtime list, we got this failure.

jodh-intel commented 6 years ago

Hi @chavafg - this is pretty weird. list calls stat in the default scenario on the rootfs provided by the config.json config file. And the error you are seeing shows that directory doesn't exist. So either the directory hasn't yet been created by the time list is called, or else it's already been deleted. It seems totally wrong if the runtime is provided with a config with an invalid rootfs so I'm assuming it's the latter scenario even though that too seems odd.

Could you get any further insight into which scenario we might be hitting if you add some debug to hello_world.bats?

chavafg commented 6 years ago

From what I can see is that we are seeing the scenario where the directory has already been deleted (from the swarm execution) because if Openshift is executed alone (no swarm tests executed prior Openshift), this error does not appear.

jodh-intel commented 6 years ago

Thanks @chavafg. It does sound like your error handling for list is a little aggressive. I've raised #906 to mellow it a bit! Running the CI on the test PR I'm about to raise should give us a useful datapoint...

I also noticed that we're not doing exactly what runc is doing wrt user handling for list so also raised #905.

jodh-intel commented 6 years ago

907 raised to be a bit more lenient if list isn't able to query all details.

jcvenegas commented 6 years ago

@chavafg can you confirm this happend with all distros or just in fedora. I tried to reproduce it in ubuntu 16.04 and cc-runtime list is still working and not qemu instances are left.

chavafg commented 6 years ago

@jcvenegas I confirm that this only happens when running with Fedora

jcvenegas commented 6 years ago

@chavafg , Great that narrows the search space. So by looking at 

time="2018-01-08T15:49:53Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:49:53Z" level=error msg="Container 699f5cc51b188a422cb385d0264f736cc2c34d36572631f97a9ee0b744804fb8 not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:49:56Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:49:56Z" level=error msg="Container 49c70e05bfc112d29198472ef1d0ff4d5cc11257015e74e2dd86c47e82ed3aa8 not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:49:58Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:49:58Z" level=error msg="Container 4a6cdd022a982a7e07bf61cc199f2fff37a31ba639918671fc4b64739dd4f88f not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:49:59Z" level=error msg="Container not running, impossible to signal the container" source=runtime
time="2018-01-08T15:49:59Z" level=error msg="Container dbefcd9c91e90e2b16bfa1238ea6a427ddfcc2440d2121ff9874bfb7c871d6d9 not ready or running, cannot send a signal" source=runtime
time="2018-01-08T15:50:07Z" level=error msg="stat /var/lib/docker/overlay2/892032c0d059fbb32ba07497cfe5c74d6a16506955400d8532e3c150f8f80723/merged: no such file or directory" source=runtime

We have some issues trying to stop a container (and probably delete) , lets try to get a complete runtime log from a swarm testsuite to know what is going on.

jodh-intel commented 6 years ago

Hi @chavafg - is this still a problem?