Closed t3hmrman closed 3 years ago
jodh-intel
commented
6 years ago Assigning to @egernst but I think the problem is pretty clear from the log fields - you're using IPv6 and we don't support that fully yet:
msg="unsupported address"
unsupported-address-type=ipv6
t3hmrman
commented
6 years ago I thought that maybe that wasn't the issue do to the error that was showing up in the events log, in particular:
Could not add route dest(10.244.0.0/24)/gw()/dev(eth1):Also, those messages are warnings, the error is the one at the bottom (with the content that's pasted above), I think the real error is just the last line, sorry for the confusing paste
jodh-intel
commented
6 years ago Hi @t3hmrman - could you paste the full output of sudo kata-collect-data.sh into this issue (after reviewing to make sure it doesn't expose any sensitive information)?
t3hmrman
commented
6 years ago Can do, here it is:
Running kata-collect-data.sh version 1.2.0 (commit 0bcb32f7042625ea0ecee8f37b99b4459ebd5af8) at 2018-09-18.03:24:45.814366296+0200.
Runtime is /opt/kata/bin/kata-runtime.
kata-envOutput of "/opt/kata/bin/kata-runtime kata-env":
[Meta]
Version = "1.0.13"
[Runtime]
Debug = false
[Runtime.Version]
Semver = "1.2.0"
Commit = "0bcb32f7042625ea0ecee8f37b99b4459ebd5af8"
OCI = "1.0.1"
[Runtime.Config]
Path = "/opt/kata/share/defaults/kata-containers/configuration.toml"
[Hypervisor]
MachineType = "pc"
Version = "QEMU emulator version 2.11.2(kata-static)\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
Path = "/opt/kata/bin/qemu-system-x86_64"
BlockDeviceDriver = "virtio-scsi"
Msize9p = 8192
Debug = false
UseVSock = false
[Image]
Path = "/opt/kata/share/kata-containers/kata-containers-image_clearlinux_1.2.0_agent_fcfa054a757.img"
[Kernel]
Path = "/opt/kata/share/kata-containers/vmlinuz-4.14.51-7"
Parameters = ""
[Initrd]
Path = ""
[Proxy]
Type = "kataProxy"
Version = "kata-proxy version 1.2.0-17962180fc61c66066905546bfd5d8933bf73df1"
Path = "/opt/kata/libexec/kata-containers/kata-proxy"
Debug = false
[Shim]
Type = "kataShim"
Version = "kata-shim version 1.2.0-0a37760c0224167143cb3cc920c78f5147f52e70"
Path = "/opt/kata/libexec/kata-containers/kata-shim"
Debug = false
[Agent]
Type = "kata"
[Host]
Kernel = "4.15.0-29-generic"
Architecture = "amd64"
VMContainerCapable = true
[Host.Distro]
Name = "Ubuntu"
Version = "18.04"
[Host.CPU]
Vendor = "GenuineIntel"
Model = "Intel(R) Core(TM) i7 CPU X 990 @ 3.47GHz"/etc/kata-containers/configuration.toml
/opt/kata/share/defaults/kata-containers/configuration.tomlConfig file /etc/kata-containers/configuration.toml not found
Output of "cat "/opt/kata/share/defaults/kata-containers/configuration.toml"":
# Copyright (c) 2017-2018 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#
# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration.toml.in"
# XXX: Project:
# XXX: Name: Kata Containers
# XXX: Type: kata
[hypervisor.qemu]
path = "/opt/kata/bin/qemu-system-x86_64"
kernel = "/opt/kata/share/kata-containers/vmlinuz.container"
image = "/opt/kata/share/kata-containers/kata-containers.img"
machine_type = "pc"
# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = ""
# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""
# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""
# Default number of vCPUs per SB/VM:
# unspecified or 0 --> will be set to 1
# < 0 --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores --> will be set to the actual number of physical cores
default_vcpus = 1
# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number
# of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores --> will be set to the actual number of physical cores or to the maximum number
# of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0
# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
# This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0 --> will be set to 1
# > 1 <= 5 --> will be set to the specified number
# > 5 --> will be set to 5
default_bridges = 1
# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
#default_memory = 2048
# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons.
# This flag prevents the block device from being passed to the hypervisor,
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false
# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is either virtio-scsi or
# virtio-blk.
block_device_driver = "virtio-scsi"
# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false
# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true
# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
#enable_hugepages = true
# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true
# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
#
# Default false
#enable_debug = true
# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true
# This is the msize used for 9p shares. It is the number of bytes
# used for 9p packet payload.
#msize_9p = 8192
# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true
[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Default false
#enable_template = true
[proxy.kata]
path = "/opt/kata/libexec/kata-containers/kata-proxy"
# If enabled, proxy messages will be sent to the system log
# (default: disabled)
#enable_debug = true
[shim.kata]
path = "/opt/kata/libexec/kata-containers/kata-shim"
# If enabled, shim messages will be sent to the system log
# (default: disabled)
#enable_debug = true
[agent.kata]
# There is no field for this section. The goal is only to be able to
# specify which type of agent the user wants to use.
[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
#enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
# - bridged
# Uses a linux bridge to interconnect the container interface to
# the VM. Works for most cases except macvlan and ipvlan.
#
# - macvtap
# Used when the Container network interface can be bridged using
# macvtap.
internetworking_model="macvtap"Config file /usr/share/defaults/kata-containers/configuration.toml not found
---
osbuilder:
url: "https://github.com/kata-containers/osbuilder"
version: "1.2.0-ae14163ca2fad2ad41154cda31554a3075e8f5b8"
rootfs-creation-time: "2018-08-10T14:15:04.923100607-0500Z"
description: "osbuilder rootfs"
file-format-version: "0.0.2"
architecture: "x86_64"
base-distro:
name: "Clear"
version: "24330"
packages:
default:
- "iptables-bin"
- "libudev0-shim"
- "systemd"
extra:
agent:
url: "https://github.com/kata-containers/agent"
name: "kata-agent"
version: "1.2.0-fcfa054a757e7c17afba47b0b4d7e91cbb8688ed"
agent-is-init-daemon: "no"No initrd
Recent runtime problems found in system journal:
time="2018-09-16T14:13:38.423943095+02:00" level=error msg="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" arch=amd64 command=create container=8853236b9ceeb9501465168dc8a9481e8dff6400a44875df61ebdeb7acde8623 name=kata-runtime pid=32576 source=runtime
time="2018-09-16T14:13:52.232846835+02:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=b2c581d0bde8b7e549f04a3488119cc00550de904be721d69829b6d2b74fcad1 error="open /run/vc/sbs/b2c581d0bde8b7e549f04a3488119cc00550de904be721d69829b6d2b74fcad1/devices.json: no such file or directory" name=kata-runtime pid=483 sandbox=b2c581d0bde8b7e549f04a3488119cc00550de904be721d69829b6d2b74fcad1 sandboxid=b2c581d0bde8b7e549f04a3488119cc00550de904be721d69829b6d2b74fcad1 source=virtcontainers subsystem=sandbox
time="2018-09-16T14:13:53.342223382+02:00" level=warning msg="unsupported address" address="fe80::245a:32ff:fe8a:9277/64" arch=amd64 command=create container=b2c581d0bde8b7e549f04a3488119cc00550de904be721d69829b6d2b74fcad1 name=kata-runtime pid=483 source=virtcontainers subsystem=kata_agent unsupported-address-type=ipv6
time="2018-09-16T14:13:53.342354246+02:00" level=warning msg="unsupported route" arch=amd64 command=create container=b2c581d0bde8b7e549f04a3488119cc00550de904be721d69829b6d2b74fcad1 destination="fe80::/64" name=kata-runtime pid=483 source=virtcontainers subsystem=kata_agent unsupported-route-type=ipv6
time="2018-09-16T14:13:53.342424909+02:00" level=warning msg="unsupported address" address="fe80::78f7:7aff:fe76:165a/64" arch=amd64 command=create container=b2c581d0bde8b7e549f04a3488119cc00550de904be721d69829b6d2b74fcad1 name=kata-runtime pid=483 source=virtcontainers subsystem=kata_agent unsupported-address-type=ipv6
time="2018-09-16T14:13:53.34249248+02:00" level=warning msg="unsupported route" arch=amd64 command=create container=b2c581d0bde8b7e549f04a3488119cc00550de904be721d69829b6d2b74fcad1 destination="fe80::/64" name=kata-runtime pid=483 source=virtcontainers subsystem=kata_agent unsupported-route-type=ipv6
time="2018-09-16T14:13:53.351603955+02:00" level=error msg="update routes request failed" arch=amd64 command=create container=b2c581d0bde8b7e549f04a3488119cc00550de904be721d69829b6d2b74fcad1 error="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" name=kata-runtime pid=483 resulting-routes="<nil>" routes-requested="[gateway:\"10.244.0.1\" device:\"eth0\" dest:\"10.244.0.0/24\" device:\"eth0\" source:\"10.244.0.137\" scope:253 dest:\"10.244.0.0/24\" device:\"eth1\" source:\"10.244.0.138\" scope:253 ]" source=virtcontainers subsystem=kata_agent
time="2018-09-16T14:13:53.353396114+02:00" level=error msg="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" arch=amd64 command=create container=b2c581d0bde8b7e549f04a3488119cc00550de904be721d69829b6d2b74fcad1 name=kata-runtime pid=483 source=runtime
time="2018-09-16T14:14:10.204683092+02:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=904091f693bd0754aa06e3d2236b4f2373c0fe81e6992733888be0363b63bff2 error="open /run/vc/sbs/904091f693bd0754aa06e3d2236b4f2373c0fe81e6992733888be0363b63bff2/devices.json: no such file or directory" name=kata-runtime pid=914 sandbox=904091f693bd0754aa06e3d2236b4f2373c0fe81e6992733888be0363b63bff2 sandboxid=904091f693bd0754aa06e3d2236b4f2373c0fe81e6992733888be0363b63bff2 source=virtcontainers subsystem=sandbox
time="2018-09-16T14:14:11.306382834+02:00" level=warning msg="unsupported address" address="fe80::287d:daff:fe7f:8310/64" arch=amd64 command=create container=904091f693bd0754aa06e3d2236b4f2373c0fe81e6992733888be0363b63bff2 name=kata-runtime pid=914 source=virtcontainers subsystem=kata_agent unsupported-address-type=ipv6
time="2018-09-16T14:14:11.30655573+02:00" level=warning msg="unsupported route" arch=amd64 command=create container=904091f693bd0754aa06e3d2236b4f2373c0fe81e6992733888be0363b63bff2 destination="fe80::/64" name=kata-runtime pid=914 source=virtcontainers subsystem=kata_agent unsupported-route-type=ipv6
time="2018-09-16T14:14:11.306618698+02:00" level=warning msg="unsupported address" address="fe80::4055:80ff:fed2:9629/64" arch=amd64 command=create container=904091f693bd0754aa06e3d2236b4f2373c0fe81e6992733888be0363b63bff2 name=kata-runtime pid=914 source=virtcontainers subsystem=kata_agent unsupported-address-type=ipv6
time="2018-09-16T14:14:11.306689777+02:00" level=warning msg="unsupported route" arch=amd64 command=create container=904091f693bd0754aa06e3d2236b4f2373c0fe81e6992733888be0363b63bff2 destination="fe80::/64" name=kata-runtime pid=914 source=virtcontainers subsystem=kata_agent unsupported-route-type=ipv6
time="2018-09-16T14:14:11.317922289+02:00" level=error msg="update routes request failed" arch=amd64 command=create container=904091f693bd0754aa06e3d2236b4f2373c0fe81e6992733888be0363b63bff2 error="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" name=kata-runtime pid=914 resulting-routes="<nil>" routes-requested="[gateway:\"10.244.0.1\" device:\"eth0\" dest:\"10.244.0.0/24\" device:\"eth0\" source:\"10.244.0.139\" scope:253 dest:\"10.244.0.0/24\" device:\"eth1\" source:\"10.244.0.140\" scope:253 ]" source=virtcontainers subsystem=kata_agent
time="2018-09-16T14:14:11.319740025+02:00" level=error msg="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" arch=amd64 command=create container=904091f693bd0754aa06e3d2236b4f2373c0fe81e6992733888be0363b63bff2 name=kata-runtime pid=914 source=runtime
time="2018-09-16T14:14:25.316692361+02:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=377e5787bd92db9357992d12e29c202d875e72cab51868d7630234867a3f7a16 error="open /run/vc/sbs/377e5787bd92db9357992d12e29c202d875e72cab51868d7630234867a3f7a16/devices.json: no such file or directory" name=kata-runtime pid=1216 sandbox=377e5787bd92db9357992d12e29c202d875e72cab51868d7630234867a3f7a16 sandboxid=377e5787bd92db9357992d12e29c202d875e72cab51868d7630234867a3f7a16 source=virtcontainers subsystem=sandbox
time="2018-09-16T14:14:26.418838208+02:00" level=warning msg="unsupported address" address="fe80::ecf7:c9ff:fe34:a001/64" arch=amd64 command=create container=377e5787bd92db9357992d12e29c202d875e72cab51868d7630234867a3f7a16 name=kata-runtime pid=1216 source=virtcontainers subsystem=kata_agent unsupported-address-type=ipv6
time="2018-09-16T14:14:26.419043227+02:00" level=warning msg="unsupported route" arch=amd64 command=create container=377e5787bd92db9357992d12e29c202d875e72cab51868d7630234867a3f7a16 destination="fe80::/64" name=kata-runtime pid=1216 source=virtcontainers subsystem=kata_agent unsupported-route-type=ipv6
time="2018-09-16T14:14:26.419127636+02:00" level=warning msg="unsupported address" address="fe80::7007:e1ff:fe23:d8ee/64" arch=amd64 command=create container=377e5787bd92db9357992d12e29c202d875e72cab51868d7630234867a3f7a16 name=kata-runtime pid=1216 source=virtcontainers subsystem=kata_agent unsupported-address-type=ipv6
time="2018-09-16T14:14:26.419222592+02:00" level=warning msg="unsupported route" arch=amd64 command=create container=377e5787bd92db9357992d12e29c202d875e72cab51868d7630234867a3f7a16 destination="fe80::/64" name=kata-runtime pid=1216 source=virtcontainers subsystem=kata_agent unsupported-route-type=ipv6
time="2018-09-16T14:14:26.429897652+02:00" level=error msg="update routes request failed" arch=amd64 command=create container=377e5787bd92db9357992d12e29c202d875e72cab51868d7630234867a3f7a16 error="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" name=kata-runtime pid=1216 resulting-routes="<nil>" routes-requested="[gateway:\"10.244.0.1\" device:\"eth0\" dest:\"10.244.0.0/24\" device:\"eth0\" source:\"10.244.0.141\" scope:253 dest:\"10.244.0.0/24\" device:\"eth1\" source:\"10.244.0.142\" scope:253 ]" source=virtcontainers subsystem=kata_agent
time="2018-09-16T14:14:26.431671532+02:00" level=error msg="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" arch=amd64 command=create container=377e5787bd92db9357992d12e29c202d875e72cab51868d7630234867a3f7a16 name=kata-runtime pid=1216 source=runtime
time="2018-09-16T14:14:42.236969757+02:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=7a81ece27a61e10b78669503f641ed729e6d8a0f3152d58ea438eab641c35842 error="open /run/vc/sbs/7a81ece27a61e10b78669503f641ed729e6d8a0f3152d58ea438eab641c35842/devices.json: no such file or directory" name=kata-runtime pid=1648 sandbox=7a81ece27a61e10b78669503f641ed729e6d8a0f3152d58ea438eab641c35842 sandboxid=7a81ece27a61e10b78669503f641ed729e6d8a0f3152d58ea438eab641c35842 source=virtcontainers subsystem=sandbox
time="2018-09-16T14:14:43.342429468+02:00" level=warning msg="unsupported address" address="fe80::ae:e6ff:fedb:4853/64" arch=amd64 command=create container=7a81ece27a61e10b78669503f641ed729e6d8a0f3152d58ea438eab641c35842 name=kata-runtime pid=1648 source=virtcontainers subsystem=kata_agent unsupported-address-type=ipv6
time="2018-09-16T14:14:43.342655418+02:00" level=warning msg="unsupported route" arch=amd64 command=create container=7a81ece27a61e10b78669503f641ed729e6d8a0f3152d58ea438eab641c35842 destination="fe80::/64" name=kata-runtime pid=1648 source=virtcontainers subsystem=kata_agent unsupported-route-type=ipv6
time="2018-09-16T14:14:43.342725068+02:00" level=warning msg="unsupported address" address="fe80::ec54:e4ff:fee0:7191/64" arch=amd64 command=create container=7a81ece27a61e10b78669503f641ed729e6d8a0f3152d58ea438eab641c35842 name=kata-runtime pid=1648 source=virtcontainers subsystem=kata_agent unsupported-address-type=ipv6
time="2018-09-16T14:14:43.3427994+02:00" level=warning msg="unsupported route" arch=amd64 command=create container=7a81ece27a61e10b78669503f641ed729e6d8a0f3152d58ea438eab641c35842 destination="fe80::/64" name=kata-runtime pid=1648 source=virtcontainers subsystem=kata_agent unsupported-route-type=ipv6
time="2018-09-16T14:14:43.353028558+02:00" level=error msg="update routes request failed" arch=amd64 command=create container=7a81ece27a61e10b78669503f641ed729e6d8a0f3152d58ea438eab641c35842 error="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" name=kata-runtime pid=1648 resulting-routes="<nil>" routes-requested="[gateway:\"10.244.0.1\" device:\"eth0\" dest:\"10.244.0.0/24\" device:\"eth0\" source:\"10.244.0.143\" scope:253 dest:\"10.244.0.0/24\" device:\"eth1\" source:\"10.244.0.144\" scope:253 ]" source=virtcontainers subsystem=kata_agent
time="2018-09-16T14:14:43.355477407+02:00" level=error msg="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" arch=amd64 command=create container=7a81ece27a61e10b78669503f641ed729e6d8a0f3152d58ea438eab641c35842 name=kata-runtime pid=1648 source=runtime
time="2018-09-16T14:14:58.216688153+02:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=3f1858377540f0c1da84c4e076a1d0664956a7ecab845b869888dca63670166e error="open /run/vc/sbs/3f1858377540f0c1da84c4e076a1d0664956a7ecab845b869888dca63670166e/devices.json: no such file or directory" name=kata-runtime pid=1930 sandbox=3f1858377540f0c1da84c4e076a1d0664956a7ecab845b869888dca63670166e sandboxid=3f1858377540f0c1da84c4e076a1d0664956a7ecab845b869888dca63670166e source=virtcontainers subsystem=sandbox
time="2018-09-16T14:14:59.330311752+02:00" level=warning msg="unsupported address" address="fe80::786d:e7ff:fe1f:c2b6/64" arch=amd64 command=create container=3f1858377540f0c1da84c4e076a1d0664956a7ecab845b869888dca63670166e name=kata-runtime pid=1930 source=virtcontainers subsystem=kata_agent unsupported-address-type=ipv6
time="2018-09-16T14:14:59.330524243+02:00" level=warning msg="unsupported route" arch=amd64 command=create container=3f1858377540f0c1da84c4e076a1d0664956a7ecab845b869888dca63670166e destination="fe80::/64" name=kata-runtime pid=1930 source=virtcontainers subsystem=kata_agent unsupported-route-type=ipv6
time="2018-09-16T14:14:59.330606489+02:00" level=warning msg="unsupported address" address="fe80::3430:a5ff:fe72:c5db/64" arch=amd64 command=create container=3f1858377540f0c1da84c4e076a1d0664956a7ecab845b869888dca63670166e name=kata-runtime pid=1930 source=virtcontainers subsystem=kata_agent unsupported-address-type=ipv6
time="2018-09-16T14:14:59.330688207+02:00" level=warning msg="unsupported route" arch=amd64 command=create container=3f1858377540f0c1da84c4e076a1d0664956a7ecab845b869888dca63670166e destination="fe80::/64" name=kata-runtime pid=1930 source=virtcontainers subsystem=kata_agent unsupported-route-type=ipv6
time="2018-09-16T14:14:59.340764369+02:00" level=error msg="update routes request failed" arch=amd64 command=create container=3f1858377540f0c1da84c4e076a1d0664956a7ecab845b869888dca63670166e error="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" name=kata-runtime pid=1930 resulting-routes="<nil>" routes-requested="[gateway:\"10.244.0.1\" device:\"eth0\" dest:\"10.244.0.0/24\" device:\"eth0\" source:\"10.244.0.145\" scope:253 dest:\"10.244.0.0/24\" device:\"eth1\" source:\"10.244.0.146\" scope:253 ]" source=virtcontainers subsystem=kata_agent
time="2018-09-16T14:14:59.342315768+02:00" level=error msg="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" arch=amd64 command=create container=3f1858377540f0c1da84c4e076a1d0664956a7ecab845b869888dca63670166e name=kata-runtime pid=1930 source=runtime
time="2018-09-16T14:15:12.204867312+02:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=977eabb12b1d9499ef867a088eea6b86dd5f168d6db73a8330306bae7cfa8980 error="open /run/vc/sbs/977eabb12b1d9499ef867a088eea6b86dd5f168d6db73a8330306bae7cfa8980/devices.json: no such file or directory" name=kata-runtime pid=2414 sandbox=977eabb12b1d9499ef867a088eea6b86dd5f168d6db73a8330306bae7cfa8980 sandboxid=977eabb12b1d9499ef867a088eea6b86dd5f168d6db73a8330306bae7cfa8980 source=virtcontainers subsystem=sandbox
time="2018-09-16T14:15:13.302582654+02:00" level=warning msg="unsupported address" address="fe80::ac81:19ff:fe5d:9ba7/64" arch=amd64 command=create container=977eabb12b1d9499ef867a088eea6b86dd5f168d6db73a8330306bae7cfa8980 name=kata-runtime pid=2414 source=virtcontainers subsystem=kata_agent unsupported-address-type=ipv6
time="2018-09-16T14:15:13.302722095+02:00" level=warning msg="unsupported route" arch=amd64 command=create container=977eabb12b1d9499ef867a088eea6b86dd5f168d6db73a8330306bae7cfa8980 destination="fe80::/64" name=kata-runtime pid=2414 source=virtcontainers subsystem=kata_agent unsupported-route-type=ipv6
time="2018-09-16T14:15:13.302800911+02:00" level=warning msg="unsupported address" address="fe80::7c7d:bff:fe1d:b976/64" arch=amd64 command=create container=977eabb12b1d9499ef867a088eea6b86dd5f168d6db73a8330306bae7cfa8980 name=kata-runtime pid=2414 source=virtcontainers subsystem=kata_agent unsupported-address-type=ipv6
time="2018-09-16T14:15:13.302869915+02:00" level=warning msg="unsupported route" arch=amd64 command=create container=977eabb12b1d9499ef867a088eea6b86dd5f168d6db73a8330306bae7cfa8980 destination="fe80::/64" name=kata-runtime pid=2414 source=virtcontainers subsystem=kata_agent unsupported-route-type=ipv6
time="2018-09-16T14:15:13.313567568+02:00" level=error msg="update routes request failed" arch=amd64 command=create container=977eabb12b1d9499ef867a088eea6b86dd5f168d6db73a8330306bae7cfa8980 error="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" name=kata-runtime pid=2414 resulting-routes="<nil>" routes-requested="[gateway:\"10.244.0.1\" device:\"eth0\" dest:\"10.244.0.0/24\" device:\"eth0\" source:\"10.244.0.147\" scope:253 dest:\"10.244.0.0/24\" device:\"eth1\" source:\"10.244.0.148\" scope:253 ]" source=virtcontainers subsystem=kata_agent
time="2018-09-16T14:15:13.315594296+02:00" level=error msg="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" arch=amd64 command=create container=977eabb12b1d9499ef867a088eea6b86dd5f168d6db73a8330306bae7cfa8980 name=kata-runtime pid=2414 source=runtime
time="2018-09-16T14:15:27.320817727+02:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=8119366b57296ec3ac0b4b612851469b0a6d265733085640edea6ad49141f84c error="open /run/vc/sbs/8119366b57296ec3ac0b4b612851469b0a6d265733085640edea6ad49141f84c/devices.json: no such file or directory" name=kata-runtime pid=2780 sandbox=8119366b57296ec3ac0b4b612851469b0a6d265733085640edea6ad49141f84c sandboxid=8119366b57296ec3ac0b4b612851469b0a6d265733085640edea6ad49141f84c source=virtcontainers subsystem=sandbox
time="2018-09-16T14:15:28.425651336+02:00" level=warning msg="unsupported address" address="fe80::e4fa:c2ff:fe98:375c/64" arch=amd64 command=create container=8119366b57296ec3ac0b4b612851469b0a6d265733085640edea6ad49141f84c name=kata-runtime pid=2780 source=virtcontainers subsystem=kata_agent unsupported-address-type=ipv6
time="2018-09-16T14:15:28.42582518+02:00" level=warning msg="unsupported route" arch=amd64 command=create container=8119366b57296ec3ac0b4b612851469b0a6d265733085640edea6ad49141f84c destination="fe80::/64" name=kata-runtime pid=2780 source=virtcontainers subsystem=kata_agent unsupported-route-type=ipv6
time="2018-09-16T14:15:28.425892132+02:00" level=warning msg="unsupported address" address="fe80::873:97ff:fe61:b142/64" arch=amd64 command=create container=8119366b57296ec3ac0b4b612851469b0a6d265733085640edea6ad49141f84c name=kata-runtime pid=2780 source=virtcontainers subsystem=kata_agent unsupported-address-type=ipv6
time="2018-09-16T14:15:28.425972892+02:00" level=warning msg="unsupported route" arch=amd64 command=create container=8119366b57296ec3ac0b4b612851469b0a6d265733085640edea6ad49141f84c destination="fe80::/64" name=kata-runtime pid=2780 source=virtcontainers subsystem=kata_agent unsupported-route-type=ipv6
time="2018-09-16T14:15:28.437689021+02:00" level=error msg="update routes request failed" arch=amd64 command=create container=8119366b57296ec3ac0b4b612851469b0a6d265733085640edea6ad49141f84c error="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" name=kata-runtime pid=2780 resulting-routes="<nil>" routes-requested="[gateway:\"10.244.0.1\" device:\"eth0\" dest:\"10.244.0.0/24\" device:\"eth0\" source:\"10.244.0.149\" scope:253 dest:\"10.244.0.0/24\" device:\"eth1\" source:\"10.244.0.150\" scope:253 ]" source=virtcontainers subsystem=kata_agent
time="2018-09-16T14:15:28.439558264+02:00" level=error msg="rpc error: code = Internal desc = Could not add route dest(10.244.0.0/24)/gw()/dev(eth1): file exists" arch=amd64 command=create container=8119366b57296ec3ac0b4b612851469b0a6d265733085640edea6ad49141f84c name=kata-runtime pid=2780 source=runtimeNo recent proxy problems found in system journal.
No recent shim problems found in system journal.
No docker
Have kubectl
Output of "kubectl version":
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}Output of "kubectl config view":
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://XXX.XXX.XXX.XXX:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data: REDACTED
client-key-data: REDACTEDOutput of "systemctl show kubelet":
Type=simple
Restart=always
NotifyAccess=none
RestartUSec=10s
TimeoutStartUSec=1min 30s
TimeoutStopUSec=1min 30s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestamp=Sun 2018-09-16 14:02:37 CEST
WatchdogTimestampMonotonic=60447579
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=5360
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
UID=[not set]
GID=[not set]
NRestarts=0
ExecMainStartTimestamp=Sun 2018-09-16 14:02:37 CEST
ExecMainStartTimestampMonotonic=60447524
ExecMainExitTimestampMonotonic=0
ExecMainPID=5360
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/bin/kubelet ; argv[]=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/kubelet.service
MemoryCurrent=[not set]
CPUUsageNSec=[not set]
TasksCurrent=43
IPIngressBytes=18446744073709551615
IPIngressPackets=18446744073709551615
IPEgressBytes=18446744073709551615
IPEgressPackets=18446744073709551615
Delegate=no
CPUAccounting=no
CPUWeight=[not set]
StartupCPUWeight=[not set]
CPUShares=[not set]
StartupCPUShares=[not set]
CPUQuotaPerSecUSec=infinity
IOAccounting=no
IOWeight=[not set]
StartupIOWeight=[not set]
BlockIOAccounting=no
BlockIOWeight=[not set]
StartupBlockIOWeight=[not set]
MemoryAccounting=no
MemoryLow=0
MemoryHigh=infinity
MemoryMax=infinity
MemorySwapMax=infinity
MemoryLimit=infinity
DevicePolicy=auto
TasksAccounting=yes
TasksMax=4915
IPAccounting=no
Environment=[unprintable] KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml [unprintable]
EnvironmentFile=/var/lib/kubelet/kubeadm-flags.env (ignore_errors=yes)
EnvironmentFile=/etc/default/kubelet (ignore_errors=yes)
UMask=0022
LimitCPU=infinity
LimitCPUSoft=infinity
LimitFSIZE=infinity
LimitFSIZESoft=infinity
LimitDATA=infinity
LimitDATASoft=infinity
LimitSTACK=infinity
LimitSTACKSoft=8388608
LimitCORE=infinity
LimitCORESoft=0
LimitRSS=infinity
LimitRSSSoft=infinity
LimitNOFILE=4096
LimitNOFILESoft=1024
LimitAS=infinity
LimitASSoft=infinity
LimitNPROC=96143
LimitNPROCSoft=96143
LimitMEMLOCK=16777216
LimitMEMLOCKSoft=16777216
LimitLOCKS=infinity
LimitLOCKSSoft=infinity
LimitSIGPENDING=96143
LimitSIGPENDINGSoft=96143
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=infinity
LimitRTTIMESoft=infinity
OOMScoreAdjust=0
Nice=0
IOSchedulingClass=0
IOSchedulingPriority=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardInputData=
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
LogLevelMax=-1
SecureBits=0
CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend
AmbientCapabilities=
DynamicUser=no
RemoveIPC=no
MountFlags=
PrivateTmp=no
PrivateDevices=no
ProtectKernelTunables=no
ProtectKernelModules=no
ProtectControlGroups=no
PrivateNetwork=no
PrivateUsers=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
LockPersonality=no
RuntimeDirectoryPreserve=no
RuntimeDirectoryMode=0755
StateDirectoryMode=0755
CacheDirectoryMode=0755
LogsDirectoryMode=0755
ConfigurationDirectoryMode=0755
MemoryDenyWriteExecute=no
RestrictRealtime=no
RestrictNamespaces=no
MountAPIVFS=no
KeyringMode=private
KillMode=control-group
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=kubelet.service
Names=kubelet.service
Requires=sysinit.target system.slice
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=shutdown.target multi-user.target
After=basic.target system.slice systemd-journald.socket sysinit.target
Documentation=https://kubernetes.io/docs/home/
Description=kubelet: The Kubernetes Node Agent
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/lib/systemd/system/kubelet.service
DropInPaths=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf /etc/systemd/system/kubelet.service.d/20-cri.conf
UnitFileState=enabled
UnitFilePreset=enabled
StateChangeTimestamp=Sun 2018-09-16 14:02:37 CEST
StateChangeTimestampMonotonic=60447581
InactiveExitTimestamp=Sun 2018-09-16 14:02:37 CEST
InactiveExitTimestampMonotonic=60447581
ActiveEnterTimestamp=Sun 2018-09-16 14:02:37 CEST
ActiveEnterTimestampMonotonic=60447581
ActiveExitTimestamp=Sun 2018-09-16 14:02:37 CEST
ActiveExitTimestampMonotonic=60394698
InactiveEnterTimestamp=Sun 2018-09-16 14:02:37 CEST
InactiveEnterTimestampMonotonic=60440324
CanStart=yes
CanStop=yes
CanReload=no
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobRunningTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Sun 2018-09-16 14:02:37 CEST
ConditionTimestampMonotonic=60441506
AssertTimestamp=Sun 2018-09-16 14:02:37 CEST
AssertTimestampMonotonic=60441507
Transient=no
Perpetual=no
StartLimitIntervalUSec=0
StartLimitBurst=5
StartLimitAction=none
FailureAction=none
SuccessAction=none
InvocationID=95a5d49dfec2450a8929698cc3cbdf98
CollectMode=inactiveNo crio
Have dpkg
Output of "dpkg -l|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-containers-image|linux-container|qemu-)"":
No rpm
jodh-intel
commented
5 years ago
Description of problem
kata-containersviakata-deployExpected result
Successful pod creation
Actual result
Pod stuck in
ContainerCreatingPod events:
relevant
containerdwarnings/errors (journal -xef -u containerd):NOTE I run
kube-router, which I'm pretty sure is properly CNI compliant, I can try switching to canal instead if that's more common