kata-containers / runtime

Kata Containers version 1.x runtime (for version 2.x see https://github.com/kata-containers/kata-containers).
https://katacontainers.io/
Apache License 2.0
2.1k stars 375 forks source link

rpc error: code = Internal desc = Could not set name eth2 for interface file exists: unknown #785

Closed fredbcode closed 3 years ago

fredbcode commented 6 years ago

Description of problem

docker-compose up stop with an error message (works without kataruntime)

Expected result

All dockers image runs

Actual result

ERROR: for app_apache-rp_1 Cannot start service apache-rp: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth2 for interface &{{3 1500 1000 eth1 02:42:ac:14:00:03 broadcast|multicast 4098 0 0 0xc4202940e0 0 0xc420017760 ether down 0 0 0 []}}: file exists: unknown

ERROR: for apache-rp Cannot start service apache-rp: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth2 for interface &{{3 1500 1000 eth1 02:42:ac:14:00:03 broadcast|multicast 4098 0 0 0xc4202940e0 0 0xc420017760 ether down 0 0 0 []}}: file exists: unknown ERROR: Encountered errors while bringing up the project.


Meta details

Running kata-collect-data.sh version 1.3.0-rc1 (commit 22aedc4) at 2018-09-25.12:56:13.961551045+0000.


Runtime is /usr/bin/kata-runtime.

kata-env

Output of "/usr/bin/kata-runtime kata-env":

[Meta]
  Version = "1.0.15"

[Runtime]
  Debug = true
  Path = "/usr/bin/kata-runtime"
  [Runtime.Version]
    Semver = "1.3.0-rc1"
    Commit = "22aedc4"
    OCI = "1.0.1"
  [Runtime.Config]
    Path = "/usr/share/defaults/kata-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.11.0\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  Msize9p = 8192
  Debug = true
  UseVSock = false

[Image]
  Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.3.0-rc1_agent_1ee972176ae.img"

[Kernel]
  Path = "/usr/share/kata-containers/vmlinuz-4.14.67.11-137.container"
  Parameters = ""

[Initrd]
  Path = ""

[Proxy]
  Type = "kataProxy"
  Version = "kata-proxy version 1.3.0-rc1-981fef4"
  Path = "/usr/libexec/kata-containers/kata-proxy"
  Debug = true

[Shim]
  Type = "kataShim"
  Version = "kata-shim version 1.3.0-rc1-9b2891c"
  Path = "/usr/libexec/kata-containers/kata-shim"
  Debug = true

[Agent]
  Type = "kata"

[Host]
  Kernel = "4.15.0-34-generic"
  Architecture = "amd64"
  VMContainerCapable = true
  SupportVSocks = false
  [Host.Distro]
    Name = "Ubuntu"
    Version = "18.04"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz"

Runtime config files

Runtime default config files

/etc/kata-containers/configuration.toml
/usr/share/defaults/kata-containers/configuration.toml

Runtime config file contents

Config file /etc/kata-containers/configuration.toml not found Output of "cat "/usr/share/defaults/kata-containers/configuration.toml"":

# Copyright (c) 2017-2018 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#

# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration.toml.in"
# XXX: Project:
# XXX:   Name: Kata Containers
# XXX:   Type: kata

[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
kernel = "/usr/share/kata-containers/vmlinuz.container"
image = "/usr/share/kata-containers/kata-containers.img"
machine_type = "pc"

# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = ""

# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""

# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""

# Default number of vCPUs per SB/VM:
# unspecified or 0                --> will be set to 1
# < 0                             --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores
default_vcpus = 1

# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0             --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0

# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
#   This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0   --> will be set to 1
# > 1 <= 5           --> will be set to the specified number
# > 5                --> will be set to 5
default_bridges = 1

# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
#default_memory = 2048

# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's 
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons. 
# This flag prevents the block device from being passed to the hypervisor, 
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false

# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is either virtio-scsi or 
# virtio-blk.
block_device_driver = "virtio-scsi"

# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically 
# result in memory pre allocation
#enable_hugepages = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
# 
# Default false
enable_debug = true

# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
# 
#disable_nesting_checks = true

# This is the msize used for 9p shares. It is the number of bytes 
# used for 9p packet payload.
#msize_9p = 8192

# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true

# VFIO devices are hotplugged on a bridge by default. 
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on 
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true

[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Default false
#enable_template = true

[proxy.kata]
path = "/usr/libexec/kata-containers/kata-proxy"

# If enabled, proxy messages will be sent to the system log
# (default: disabled)
enable_debug = true

[shim.kata]
path = "/usr/libexec/kata-containers/kata-shim"

# If enabled, shim messages will be sent to the system log
# (default: disabled)
enable_debug = true

[agent.kata]
# There is no field for this section. The goal is only to be able to
# specify which type of agent the user wants to use.

[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
#   - bridged
#     Uses a linux bridge to interconnect the container interface to
#     the VM. Works for most cases except macvlan and ipvlan.
#
#   - macvtap
#     Used when the Container network interface can be bridged using
#     macvtap.
internetworking_model="macvtap"

# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true

Image details

---
osbuilder:
  url: "https://github.com/kata-containers/osbuilder"
  version: "unknown"
rootfs-creation-time: "2018-09-13T22:26:12.472554196+0000Z"
description: "osbuilder rootfs"
file-format-version: "0.0.2"
architecture: "x86_64"
base-distro:
  name: "Clear"
  version: "25000"
  packages:
    default:
      - "iptables-bin"
      - "libudev0-shim"
      - "systemd"
    extra:

agent:
  url: "https://github.com/kata-containers/agent"
  name: "kata-agent"
  version: "1.3.0-rc1-1ee972176ae437bcace0a37227818c506bb64ba9"
  agent-is-init-daemon: "no"

Initrd details

No initrd


Logfiles

Runtime logs

Recent runtime problems found in system journal:

time="2018-09-25T12:52:13.081180913Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b name=kata-runtime pid=18282 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:13.081273761Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b error="open /run/vc/sbs/3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b/devices.json: no such file or directory" name=kata-runtime pid=18282 sandbox=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b sandboxid=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:13.112850265Z" level=debug arch=amd64 command=create container=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=18282 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:15.786608462Z" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b name=kata-runtime pid=18416 sandbox=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b source=runtime
time="2018-09-25T12:52:15.851309108Z" level=error msg="Container 3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b name=kata-runtime pid=18451 sandbox=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b source=runtime
time="2018-09-25T12:52:23.93719488Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=ed5f28d13e88c9833daf76b49a22bd05f934f4620c9023cfc98ca74cd48118bc name=kata-runtime pid=18776 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:23.937275076Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=ed5f28d13e88c9833daf76b49a22bd05f934f4620c9023cfc98ca74cd48118bc error="open /run/vc/sbs/ed5f28d13e88c9833daf76b49a22bd05f934f4620c9023cfc98ca74cd48118bc/devices.json: no such file or directory" name=kata-runtime pid=18776 sandbox=ed5f28d13e88c9833daf76b49a22bd05f934f4620c9023cfc98ca74cd48118bc sandboxid=ed5f28d13e88c9833daf76b49a22bd05f934f4620c9023cfc98ca74cd48118bc source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:23.940444629Z" level=debug arch=amd64 command=create container=ed5f28d13e88c9833daf76b49a22bd05f934f4620c9023cfc98ca74cd48118bc default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=18776 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:25.041218059Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=6b9fbbaf54e53b051a38b342e3957456e0de0397b7106eeb891531da07fe0737 name=kata-runtime pid=18850 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:25.041306973Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=6b9fbbaf54e53b051a38b342e3957456e0de0397b7106eeb891531da07fe0737 error="open /run/vc/sbs/6b9fbbaf54e53b051a38b342e3957456e0de0397b7106eeb891531da07fe0737/devices.json: no such file or directory" name=kata-runtime pid=18850 sandbox=6b9fbbaf54e53b051a38b342e3957456e0de0397b7106eeb891531da07fe0737 sandboxid=6b9fbbaf54e53b051a38b342e3957456e0de0397b7106eeb891531da07fe0737 source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:25.044666392Z" level=debug arch=amd64 command=create container=6b9fbbaf54e53b051a38b342e3957456e0de0397b7106eeb891531da07fe0737 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=18850 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:26.146000833Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=ec110f10065d4f2b0c613c50fc126ffe088d126ab75e39c9f40e18f02bc2a53b name=kata-runtime pid=18937 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:26.146090795Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=ec110f10065d4f2b0c613c50fc126ffe088d126ab75e39c9f40e18f02bc2a53b error="open /run/vc/sbs/ec110f10065d4f2b0c613c50fc126ffe088d126ab75e39c9f40e18f02bc2a53b/devices.json: no such file or directory" name=kata-runtime pid=18937 sandbox=ec110f10065d4f2b0c613c50fc126ffe088d126ab75e39c9f40e18f02bc2a53b sandboxid=ec110f10065d4f2b0c613c50fc126ffe088d126ab75e39c9f40e18f02bc2a53b source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:26.149340329Z" level=debug arch=amd64 command=create container=ec110f10065d4f2b0c613c50fc126ffe088d126ab75e39c9f40e18f02bc2a53b default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=18937 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:27.234578298Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=025d882ad02621fd22420efd9eb152a4cba3c467a93e9aeb95123894765bf8d1 name=kata-runtime pid=18936 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:27.234668383Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=025d882ad02621fd22420efd9eb152a4cba3c467a93e9aeb95123894765bf8d1 error="open /run/vc/sbs/025d882ad02621fd22420efd9eb152a4cba3c467a93e9aeb95123894765bf8d1/devices.json: no such file or directory" name=kata-runtime pid=18936 sandbox=025d882ad02621fd22420efd9eb152a4cba3c467a93e9aeb95123894765bf8d1 sandboxid=025d882ad02621fd22420efd9eb152a4cba3c467a93e9aeb95123894765bf8d1 source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:27.237834807Z" level=debug arch=amd64 command=create container=025d882ad02621fd22420efd9eb152a4cba3c467a93e9aeb95123894765bf8d1 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=18936 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:28.2194947Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb name=kata-runtime pid=19005 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:28.219589461Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb error="open /run/vc/sbs/7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb/devices.json: no such file or directory" name=kata-runtime pid=19005 sandbox=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb sandboxid=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:28.222680814Z" level=debug arch=amd64 command=create container=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=19005 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:29.301948351Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=19580 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:29.302020019Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="open /run/vc/sbs/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/devices.json: no such file or directory" name=kata-runtime pid=19580 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 sandboxid=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:29.306542004Z" level=debug arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=19580 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:30.45857333Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd name=kata-runtime pid=19208 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:30.45867002Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd error="open /run/vc/sbs/1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd/devices.json: no such file or directory" name=kata-runtime pid=19208 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd sandboxid=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=virtcontainers subsystem=sandbox
time="2018-09-25T12:52:30.465720646Z" level=debug arch=amd64 command=create container=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=19208 source=virtcontainers subsystem=qemu
time="2018-09-25T12:52:32.049678014Z" level=error msg="update interface request failed" arch=amd64 command=create container=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd error="rpc error: code = Internal desc = Could not set name eth2 for interface &{{3 1500 1000 eth1 02:42:ac:14:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4202940e0 0 0xc420017760 ether <nil> down 0 0 0 []}}: file exists" interface-requested="device:\"eth2\" name:\"eth2\" IPAddresses:<address:\"172.20.0.3\" mask:\"16\" > mtu:1500 hwAddr:\"02:42:ac:14:00:03\" " name=kata-runtime pid=19208 resulting-interface="<nil>" source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:52:32.284653706Z" level=error msg="rpc error: code = Internal desc = Could not set name eth2 for interface &{{3 1500 1000 eth1 02:42:ac:14:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4202940e0 0 0xc420017760 ether <nil> down 0 0 0 []}}: file exists" arch=amd64 command=create container=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd name=kata-runtime pid=19208 source=runtime
time="2018-09-25T12:53:35.291267859Z" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20048 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=runtime
time="2018-09-25T12:53:35.357743544Z" level=error msg="Container d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20083 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=runtime
time="2018-09-25T12:53:36.805211646Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20162 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:53:36.805307288Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="open /run/vc/sbs/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/devices.json: no such file or directory" name=kata-runtime pid=20162 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 sandboxid=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=virtcontainers subsystem=sandbox
time="2018-09-25T12:53:36.809959751Z" level=debug arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=20162 source=virtcontainers subsystem=qemu
time="2018-09-25T12:54:42.635201905Z" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20323 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=runtime
time="2018-09-25T12:54:42.701442185Z" level=error msg="Container d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20358 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=runtime
time="2018-09-25T12:54:44.157224658Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20435 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:54:44.157319126Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="open /run/vc/sbs/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/devices.json: no such file or directory" name=kata-runtime pid=20435 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 sandboxid=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=virtcontainers subsystem=sandbox
time="2018-09-25T12:54:44.161909512Z" level=debug arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=20435 source=virtcontainers subsystem=qemu
time="2018-09-25T12:55:49.999109408Z" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20581 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=runtime
time="2018-09-25T12:55:50.065967266Z" level=error msg="Container d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20618 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=runtime
time="2018-09-25T12:55:51.501234794Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20695 source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:55:51.501331066Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="open /run/vc/sbs/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/devices.json: no such file or directory" name=kata-runtime pid=20695 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 sandboxid=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=virtcontainers subsystem=sandbox
time="2018-09-25T12:55:51.505721051Z" level=debug arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=20695 source=virtcontainers subsystem=qemu
time="2018-09-25T12:55:52.881076812Z" level=error msg="update interface request failed" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:15:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201d45d4 0 0xc420017220 ether <nil> down 0 0 0 []}}: file exists" interface-requested="device:\"eth1\" name:\"eth1\" IPAddresses:<address:\"172.21.0.3\" mask:\"16\" > mtu:1500 hwAddr:\"02:42:ac:15:00:03\" " name=kata-runtime pid=20695 resulting-interface="<nil>" source=virtcontainers subsystem=kata_agent
time="2018-09-25T12:55:53.024631795Z" level=error msg="rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:15:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201d45d4 0 0xc420017220 ether <nil> down 0 0 0 []}}: file exists" arch=amd64 command=create container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-runtime pid=20695 source=runtime

Proxy logs

Recent proxy problems found in system journal:

time="2018-09-25T12:52:29.346550913Z" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=19539 sandbox=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb source=agent
time="2018-09-25T12:52:29.347968943Z" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=19539 sandbox=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb source=agent
time="2018-09-25T12:52:29.721355457Z" level=info msg="time=\"2018-09-25T12:52:29.69625668Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=106 sandbox=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb signal=\"child exited\" source=agent\n" name=kata-proxy pid=19539 sandbox=7a4cb6531c18de6d196d621703136720944c9e7ebc75b66fa19815c8f96a3cbb source=agent
time="2018-09-25T12:52:30.41325028Z" level=info msg="[    0.555245] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:30.413968208Z" level=info msg="[    0.555811] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:30.46014495Z" level=info msg="[    0.602045] systemd-gpt-auto-generator[68]: Failed to chase block device '/', ignoring: No such file or directory\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:30.532208493Z" level=info msg="[    0.674066] systemd-journald[78]: Failed to open configuration file '/etc/systemd/journald.conf': No such file or directory\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:30.543204241Z" level=info msg="[    0.685121] systemd[79]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:30.565093226Z" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:30.568146089Z" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:30.945781594Z" level=info msg="time=\"2018-09-25T12:52:30.917286105Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=111 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 signal=\"child exited\" source=agent\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:52:31.656728874Z" level=info msg="[    0.597650] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=19795 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=agent
time="2018-09-25T12:52:31.657552474Z" level=info msg="[    0.598450] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=19795 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=agent
time="2018-09-25T12:52:31.703661711Z" level=info msg="[    0.644517] systemd-gpt-auto-generator[68]: Failed to chase block device '/', ignoring: No such file or directory\n" name=kata-proxy pid=19795 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=agent
time="2018-09-25T12:52:31.749604961Z" level=info msg="[    0.690513] systemd[75]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=19795 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=agent
time="2018-09-25T12:52:31.76842037Z" level=info msg="[    0.709265] systemd-journald[76]: Failed to open configuration file '/etc/systemd/journald.conf': No such file or directory\n" name=kata-proxy pid=19795 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=agent
time="2018-09-25T12:52:31.79259753Z" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=19795 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=agent
time="2018-09-25T12:52:31.793060201Z" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=19795 sandbox=1c0b5aa212d0b44d3d62622ec87692c6d1a0489372bc079611cd05e71c43c5dd source=agent
time="2018-09-25T12:53:35.123132286Z" level=info msg="time=\"2018-09-25T12:53:35.093430369Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=111 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 signal=\"child exited\" source=agent\n" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:35.125151466Z" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/proxy.sock->@: read: connection reset by peer" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=proxy
time="2018-09-25T12:53:35.39996404Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/kata.sock: use of closed network connection" name=kata-proxy pid=19706 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=proxy
time="2018-09-25T12:53:37.791036025Z" level=info msg="[    0.430956] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:37.791736675Z" level=info msg="[    0.431591] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:37.836782316Z" level=info msg="[    0.476577] systemd-gpt-auto-generator[68]: Failed to chase block device '/', ignoring: No such file or directory\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:37.878559988Z" level=info msg="[    0.518363] systemd-journald[70]: Failed to open configuration file '/etc/systemd/journald.conf': No such file or directory\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:37.882172747Z" level=info msg="[    0.522104] systemd[72]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:37.928259714Z" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:37.930315784Z" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:53:38.329717686Z" level=info msg="time=\"2018-09-25T12:53:38.299431836Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=107 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 signal=\"child exited\" source=agent\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:42.494115853Z" level=info msg="time=\"2018-09-25T12:54:42.462991419Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=107 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 signal=\"child exited\" source=agent\n" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:42.495302946Z" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/proxy.sock->@: read: connection reset by peer" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=proxy
time="2018-09-25T12:54:42.738042246Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/kata.sock: use of closed network connection" name=kata-proxy pid=20242 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=proxy
time="2018-09-25T12:54:45.149484374Z" level=info msg="[    0.435340] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:45.150155549Z" level=info msg="[    0.435947] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:45.193234336Z" level=info msg="[    0.478985] systemd-gpt-auto-generator[68]: Failed to chase block device '/', ignoring: No such file or directory\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:45.239850619Z" level=info msg="[    0.525611] systemd-journald[71]: Failed to open configuration file '/etc/systemd/journald.conf': No such file or directory\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:45.2449347Z" level=info msg="[    0.530747] systemd[72]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:45.280246221Z" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:45.283244161Z" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:54:45.690288717Z" level=info msg="time=\"2018-09-25T12:54:45.657952596Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=106 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 signal=\"child exited\" source=agent\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:49.851166455Z" level=info msg="time=\"2018-09-25T12:55:49.817549873Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=106 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 signal=\"child exited\" source=agent\n" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:49.8526652Z" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/proxy.sock->@: read: connection reset by peer" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=proxy
time="2018-09-25T12:55:50.106968285Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67/kata.sock: use of closed network connection" name=kata-proxy pid=20512 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=proxy
time="2018-09-25T12:55:52.49255434Z" level=info msg="[    0.434797] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=20772 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:52.493227222Z" level=info msg="[    0.435424] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=20772 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:52.535750235Z" level=info msg="[    0.477910] systemd-gpt-auto-generator[68]: Failed to chase block device '/', ignoring: No such file or directory\n" name=kata-proxy pid=20772 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:52.579513501Z" level=info msg="[    0.521724] systemd[71]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=20772 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:52.611329036Z" level=info msg="[    0.553487] systemd-journald[80]: Failed to open configuration file '/etc/systemd/journald.conf': No such file or directory\n" name=kata-proxy pid=20772 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:52.621840806Z" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=20772 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent
time="2018-09-25T12:55:52.62393186Z" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=20772 sandbox=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 source=agent

Shim logs

Recent shim problems found in system journal:

time="2018-09-25T12:52:15.667560942Z" level=info msg="copy stderr failed" container=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b error="rpc error: code = Unknown desc = EOF" exec-id=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b name=kata-shim pid=1 source=shim
time="2018-09-25T12:52:15.667588664Z" level=info msg="copy stdout failed" container=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b error="rpc error: code = Unknown desc = EOF" exec-id=3e7e9abde319b527e31098303268cab3d31d650380902ed1112e4a8d55cdee8b name=kata-shim pid=1 source=shim
time="2018-09-25T12:53:35.120234685Z" level=info msg="copy stdout failed" container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="rpc error: code = Unknown desc = EOF" exec-id=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-shim pid=1 source=shim
time="2018-09-25T12:53:35.120241713Z" level=info msg="copy stderr failed" container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="rpc error: code = Unknown desc = EOF" exec-id=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-shim pid=1 source=shim
time="2018-09-25T12:54:42.492389169Z" level=info msg="copy stdout failed" container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="rpc error: code = Unknown desc = EOF" exec-id=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-shim pid=1 source=shim
time="2018-09-25T12:54:42.492392873Z" level=info msg="copy stderr failed" container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="rpc error: code = Unknown desc = EOF" exec-id=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-shim pid=1 source=shim
time="2018-09-25T12:55:49.848505925Z" level=info msg="copy stdout failed" container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="rpc error: code = Unknown desc = EOF" exec-id=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-shim pid=1 source=shim
time="2018-09-25T12:55:49.848538129Z" level=info msg="copy stderr failed" container=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 error="rpc error: code = Unknown desc = EOF" exec-id=d52b689c536af1a5fcd717a2a31c7b0482e2026d8588731312cdd803fa483a67 name=kata-shim pid=1 source=shim

Container manager details

Have docker

Docker

Output of "docker version":

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:51 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:15 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of "docker info":

Containers: 7
 Running: 5
 Paused: 0
 Stopped: 2
Images: 13
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: syslog
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: kata-runtime runc
Default Runtime: kata-runtime
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 22aedc4 (expected: 69663f0bd4b60df09991c08812a60108003fa340)
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-34-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 62.82GiB
Name: RNS-SV-KATA-001
ID: UWGY:MJA5:6VGJ:YAQY:X6F3:QRQL:JEE3:HWYM:23HC:MJHV:XSJR:J4UQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Output of "systemctl show docker":

Type=notify
Restart=on-failure
NotifyAccess=main
RestartUSec=100ms
TimeoutStartUSec=infinity
TimeoutStopUSec=1min 30s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestamp=Tue 2018-09-25 09:49:43 UTC
WatchdogTimestampMonotonic=2971349640
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=6456
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
UID=[not set]
GID=[not set]
NRestarts=0
ExecMainStartTimestamp=Tue 2018-09-25 09:49:42 UTC
ExecMainStartTimestampMonotonic=2970822338
ExecMainExitTimestampMonotonic=0
ExecMainPID=6456
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/docker.service
MemoryCurrent=[not set]
CPUUsageNSec=[not set]
TasksCurrent=245
IPIngressBytes=18446744073709551615
IPIngressPackets=18446744073709551615
IPEgressBytes=18446744073709551615
IPEgressPackets=18446744073709551615
Delegate=yes
DelegateControllers=cpu cpuacct io blkio memory devices pids
CPUAccounting=no
CPUWeight=[not set]
StartupCPUWeight=[not set]
CPUShares=[not set]
StartupCPUShares=[not set]
CPUQuotaPerSecUSec=infinity
IOAccounting=no
IOWeight=[not set]
StartupIOWeight=[not set]
BlockIOAccounting=no
BlockIOWeight=[not set]
StartupBlockIOWeight=[not set]
MemoryAccounting=no
MemoryLow=0
MemoryHigh=infinity
MemoryMax=infinity
MemorySwapMax=infinity
MemoryLimit=infinity
DevicePolicy=auto
TasksAccounting=yes
TasksMax=infinity
IPAccounting=no
UMask=0022
LimitCPU=infinity
LimitCPUSoft=infinity
LimitFSIZE=infinity
LimitFSIZESoft=infinity
LimitDATA=infinity
LimitDATASoft=infinity
LimitSTACK=infinity
LimitSTACKSoft=8388608
LimitCORE=infinity
LimitCORESoft=infinity
LimitRSS=infinity
LimitRSSSoft=infinity
LimitNOFILE=1048576
LimitNOFILESoft=1048576
LimitAS=infinity
LimitASSoft=infinity
LimitNPROC=infinity
LimitNPROCSoft=infinity
LimitMEMLOCK=16777216
LimitMEMLOCKSoft=16777216
LimitLOCKS=infinity
LimitLOCKSSoft=infinity
LimitSIGPENDING=257088
LimitSIGPENDINGSoft=257088
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=infinity
LimitRTTIMESoft=infinity
OOMScoreAdjust=0
Nice=0
IOSchedulingClass=0
IOSchedulingPriority=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardInputData=
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
LogLevelMax=-1
SecureBits=0
CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend
AmbientCapabilities=
DynamicUser=no
RemoveIPC=no
MountFlags=
PrivateTmp=no
PrivateDevices=no
ProtectKernelTunables=no
ProtectKernelModules=no
ProtectControlGroups=no
PrivateNetwork=no
PrivateUsers=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
LockPersonality=no
RuntimeDirectoryPreserve=no
RuntimeDirectoryMode=0755
StateDirectoryMode=0755
CacheDirectoryMode=0755
LogsDirectoryMode=0755
ConfigurationDirectoryMode=0755
MemoryDenyWriteExecute=no
RestrictRealtime=no
RestrictNamespaces=no
MountAPIVFS=no
KeyringMode=private
KillMode=process
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=docker.service
Names=docker.service
Requires=docker.socket system.slice sysinit.target
Wants=network-online.target
WantedBy=multi-user.target
ConsistsOf=docker.socket
Conflicts=shutdown.target
Before=shutdown.target multi-user.target
After=network-online.target system.slice basic.target firewalld.service systemd-journald.socket sysinit.target docker.socket
TriggeredBy=docker.socket
Documentation=https://docs.docker.com
Description=Docker Application Container Engine
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/lib/systemd/system/docker.service
UnitFileState=enabled
UnitFilePreset=enabled
StateChangeTimestamp=Tue 2018-09-25 09:49:43 UTC
StateChangeTimestampMonotonic=2971349643
InactiveExitTimestamp=Tue 2018-09-25 09:49:42 UTC
InactiveExitTimestampMonotonic=2970822385
ActiveEnterTimestamp=Tue 2018-09-25 09:49:43 UTC
ActiveEnterTimestampMonotonic=2971349643
ActiveExitTimestamp=Tue 2018-09-25 09:48:34 UTC
ActiveExitTimestampMonotonic=2902640442
InactiveEnterTimestamp=Tue 2018-09-25 09:48:35 UTC
InactiveEnterTimestampMonotonic=2903649196
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobRunningTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Tue 2018-09-25 09:49:42 UTC
ConditionTimestampMonotonic=2970821279
AssertTimestamp=Tue 2018-09-25 09:49:42 UTC
AssertTimestampMonotonic=2970821280
Transient=no
Perpetual=no
StartLimitIntervalUSec=1min
StartLimitBurst=3
StartLimitAction=none
FailureAction=none
SuccessAction=none
InvocationID=1c51818194e7435ba5a2b7b6aa7cb9e7
CollectMode=inactive

No kubectl


Packages

Have dpkg Output of "dpkg -l|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-containers-image|linux-container|qemu-)"":

ii  kata-containers-image                 1.3.0~rc1-34                      amd64        Kata containers image
ii  kata-linux-container                  4.14.67.11-137                    amd64        linux kernel optimised for container-like workloads.
ii  kata-proxy                            1.3.0~rc1+git.981fef4-34          amd64        
ii  kata-runtime                          1.3.0~rc1+git.22aedc4-49          amd64        
ii  kata-shim                             1.3.0~rc1+git.9b2891c-35          amd64        
ii  qemu-lite                             2.11.0+git.f886228056-50          amd64        linux kernel optimised for container-like workloads.
ii  qemu-vanilla                          2.11.2+git.a39e0b3e82-44          amd64        linux kernel optimised for container-like workloads.

No rpm


Perhaps related with #702

proxy.log runtime.log shim.log

fredbcode commented 6 years ago

No not related with #702

jodh-intel commented 6 years ago

Hi @fredbcode - thanks for reporting. However, it's difficult to see what is happening because the proxy log errors captured when you ran kata-collect-data.sh show that agent debug was not enabled and the attached logs also do not have agent debug enabled.

If you are running on a dev / test box and are happy to lose your journal log, it would be useful if you could clear the journal and re-run:

caoruidong commented 6 years ago

What are interfaces in all containers? Seems there is already a "eth2" interface?

fredbcode commented 6 years ago

There is no ethX at all but enspX, yes network is ready and the same docker-compose works well without kataruntime (on same machine of course)

I don't known where kata find this eth2

fredbcode commented 6 years ago

@jodh-intel I will provide logs tomorrow

jodh-intel commented 6 years ago

@fredbcode - thanks ;)

fredbcode commented 6 years ago

Haveged installed (see #702)

cat /proc/sys/kernel/random/entropy_avail 4066

Full log enabled and purged

ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d0:94:66:16:91:ff brd ff:ff:ff:ff:ff:ff
    inet x.x.x.x/26 brd x.x.x.x scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::d294:66ff:fe16:91ff/64 scope link 
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d0:94:66:16:92:00 brd ff:ff:ff:ff:ff:ff
    inet x.x.x.x/30 brd x.x.x.x scope global eno2
       valid_lft forever preferred_lft forever
    inet6 fe80::d294:66ff:fe16:9200/64 scope link 
       valid_lft forever preferred_lft forever
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d0:94:66:16:92:01 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d0:94:66:16:92:02 brd ff:ff:ff:ff:ff:ff
6: idrac: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d0:94:66:16:92:04 brd ff:ff:ff:ff:ff:ff 

Sorry I can't use kata-log-parser on this sever.

kata.txt proxy.log runtime.log shim.log


ERROR: for app_viewbox_1  Cannot start service viewbox: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:15Creating app_apache_1 ... error

ERROR: for app_apache_1  Cannot start service apache: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201dc5d4 0 0xc420017440 ether <nil> down 0 0 0 []}}: file exists: unknown

ERROR: for viewbox  Cannot start service viewbox: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:15:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201dc5d4 0 0xc420017440 ether <nil> down 0 0 0 []}}: file exists: unknown

ERROR: for apache  Cannot start service apache: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201dc5d4 0 0xc420017440 ether <nil> down 0 0 0 []}}: file exists: unknown
fredbcode commented 6 years ago
version: '2.1'
services:
  apache:
    build:
      context: "apache-rp"
    networks:
      - office
      - talkmost
      - viewbox
      - dokuwiki
    ports:
      - "x.x.x.x:443:443"
  office:
    image: registry.test/images/squid:current
    networks:
      - office
    restart: unless-stopped

  talkmost:
    image: registry.test/images/ubuntu
    networks:
      - talkmost
    restart: unless-stopped

  viewbox:
    image: registry.test/images/nginx
    networks:
      - viewbox
    restart: unless-stopped

networks:
  office:
  talkmost:
  viewbox:
  dokuwiki:
docker-compose up 
Starting app_apache_1 ... 
app_viewbox_1 is up-to-date
Starting app_apache_1
app_office_1 is up-to-date
Starting app_apache_1 ... error

ERROR: for app_apache_1  Cannot start service apache: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:13:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201e65d4 0 0xc420017460 ether <nil> down 0 0 0 []}}: file exists: unknown

ERROR: for apache  Cannot start service apache: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:13:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201e65d4 0 0xc420017460 ether <nil> down 0 0 0 []}}: file exists: unknown
ERROR: Encountered errors while bringing up the project.

docker-compose up without kata

docker-compose up
Creating network "app_dokuwiki" with the default driver
Creating network "app_viewbox" with the default driver
Creating network "app_office" with the default driver
Creating network "app_talkmost" with the default driver
Creating app_apache_1 ... 
Creating app_viewbox_1 ... 
Creating app_office_1 ... 
Creating app_talkmost_1 ... 
Creating app_viewbox_1
Creating app_office_1
Creating app_talkmost_1
Creating app_apache_1 ... done
Attaching to app_viewbox_1, app_talkmost_1, app_office_1, app_apache_1
office_1    | WARNING: Cannot write log file: stdio:/dev/stdout
office_1    | stdio:/dev/stdout: No such file or directory
office_1    |          messages will be sent to 'stderr'.
office_1    | 2018/09/27 08:28:30| Set Current Directory to /var/spool/squid
office_1    | WARNING: Cannot write log file: stdio:/dev/stdout
office_1    | stdio:/dev/stdout: No such file or directory
office_1    |          messages will be sent to 'stderr'.
office_1    | 2018/09/27 08:28:30| WARNING: Closing open FD    2
office_1    | 2018/09/27 08:28:30| Starting Squid Cache version 3.5.12 for x86_64-pc-linux-gnu...
office_1    | 2018/09/27 08:28:30| Service Name: squid
office_1    | 2018/09/27 08:28:30| Process ID 9
office_1    | 2018/09/27 08:28:30| Process Roles: master worker
office_1    | 2018/09/27 08:28:30| With 1048576 file descriptors available
office_1    | 2018/09/27 08:28:30| Initializing IP Cache...
office_1    | 2018/09/27 08:28:30| DNS Socket created at [::], FD 7
office_1    | 2018/09/27 08:28:30| DNS Socket created at 0.0.0.0, FD 8
office_1    | 2018/09/27 08:28:30| Adding domain rie from /etc/resolv.conf
office_1    | 2018/09/27 08:28:30| Adding nameserver 127.0.0.11 from /etc/resolv.conf
office_1    | 2018/09/27 08:28:30| Adding ndots 1 from /etc/resolv.conf
office_1    | 2018/09/27 08:28:30| Logfile: opening log stdio:/dev/stdout
office_1    | 2018/09/27 08:28:30| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
office_1    | 2018/09/27 08:28:30| Store logging disabled
office_1    | 2018/09/27 08:28:30| Swap maxSize 0 + 262144 KB, estimated 20164 objects
office_1    | 2018/09/27 08:28:30| Target number of buckets: 1008
office_1    | 2018/09/27 08:28:30| Using 8192 Store buckets
office_1    | 2018/09/27 08:28:30| Max Mem  size: 262144 KB
office_1    | 2018/09/27 08:28:30| Max Swap size: 0 KB
office_1    | 2018/09/27 08:28:30| Using Least Load store dir selection
office_1    | 2018/09/27 08:28:30| Set Current Directory to /var/spool/squid
office_1    | 2018/09/27 08:28:30| Finished loading MIME types and icons.
office_1    | 2018/09/27 08:28:30| HTCP Disabled.
office_1    | 2018/09/27 08:28:30| commBind: Cannot bind socket FD 11 to [::1]: (99) Cannot assign requested address
office_1    | 2018/09/27 08:28:30| commBind: Cannot bind socket FD 12 to [::1]: (99) Cannot assign requested address
office_1    | 2018/09/27 08:28:30| ERROR: Failed to create helper child read FD: UDP[::1]
office_1    | 2018/09/27 08:28:30| Squid plugin modules loaded: 0
office_1    | 2018/09/27 08:28:30| Adaptation support is off.
office_1    | 2018/09/27 08:28:30| Accepting HTTP Socket connections at local=[::]:3128 remote=[::] FD 10 flags=9
office_1    | 2018/09/27 08:28:31| storeLateRelease: released 0 objects
apache_1    | [Thu Sep 27 08:28:32.487388 2018] [core:error] [pid 10:tid 139667229403008] (EAI 2)Name or service not known: AH00547: Could not resolve host name block.scn.rie.gouv.fr -- ignoring!
app_talkmost_1 exited with code 0
app_talkmost_1 exited with code 0
app_talkmost_1 exited with code 0
app_talkmost_1 exited with code 0
app_talkmost_1 exited with code 0

This message is useless: office_1 | 2018/09/27 08:28:30| commBind: Cannot bind socket FD 12 to [::1]: (99) Cannot assign requested address It's about squid and ipv6, same problem with another image without network needed ...

fredbcode commented 6 years ago

Without ports:

grahamwhaley commented 6 years ago

@amshinde any thoughts from the network side on the eth1/eth2 info here?

caoruidong commented 6 years ago

Maybe related to different nic naming "ethXX" vs "enoXX".

fredbcode commented 6 years ago

I guess I found something very interesting !

version: '2.1'
services:
  fred:
    build:
      context: "ubuntu"
    networks:
      - test
      - test1
      - test2
  office:
    image: registry.test/images/ubuntu
    networks:
      - test
    restart: unless-stopped

  talkmost:
    image: registry.test/images/ubuntu
    networks:
      - test1
    restart: unless-stopped

  talkmost1:
    image: registry.test/images/ubuntu
    networks:
      - test2
    restart: unless-stopped

networks:
  test:
  test1:
  test2:
docker-compose up
Creating network "app_test1" with the default driver
Creating network "app_test" with the default driver
Creating network "app_test2" with the default driver
Creating app_talkmost1_1 ... 
Creating app_fred_1 ... 
Creating app_talkmost_1 ... 
Creating app_office_1 ... 
Creating app_talkmost1_1
Creating app_talkmost_1
Creating app_office_1
Creating app_fred_1 ... error

ERROR: for app_fred_1  Cannot start service fred: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth2 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:Creating app_talkmost1_1 ... done

ERROR: for fred  Cannot start service fred: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth2 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201e05d4 0 0xc420017440 ether <nil> down 0 0 0 []}}: file exists: unknown
ERROR: Encountered errors while bringing up the project.

But with only two internal networks:

docker-compose up
Creating network "app_test1" with the default driver
Creating network "app_test" with the default driver
Creating network "app_test2" with the default driver
Creating app_talkmost1_1 ... 
Creating app_talkmost_1 ... 
Creating app_fred_1 ... 
Creating app_office_1 ... 
Creating app_talkmost1_1
Creating app_fred_1
Creating app_talkmost_1
Creating app_office_1 ... done
Attaching to app_talkmost1_1, app_fred_1, app_talkmost_1, app_office_1
talkmost1_1  | WARNING: no logs are available with the 'syslog' log driver
fred_1       | WARNING: no logs are available with the 'syslog' log driver
talkmost_1   | WARNING: no logs are available with the 'syslog' log driver
office_1     | WARNING: no logs are available with the 'syslog' log driver
app_fred_1 exited with code 0
app_office_1 exited with code 0
app_talkmost1_1 exited with code 0
app_talkmost_1 exited with code 0
version: '2.1'
services:
  fred:
    build:
      context: "ubuntu"
    networks:
      - test
      - test1
#      - test2 **-> HERE**
  office:
    image: registry.test/images/ubuntu
    networks:
      - test
    restart: unless-stopped

  talkmost:
    image: registry.test/images/ubuntu
    networks:
      - test1
    restart: unless-stopped

  talkmost1:
    image: registry.test/images/ubuntu
    networks:
      - test2
    restart: unless-stopped

networks:
  test:
  test1:
  test2:

I'm exactly in the same situation that before with issue #702 with Kata only one or two containers works well (but here my entropy is 4066)

My Dockerfile contains only a FROM to registry - ubuntu image -

fredbcode commented 6 years ago

Of course I mean, max two containers works well -> containers with network in this case

fredbcode commented 6 years ago

Syslog:

Sep 27 09:58:03 TEST kernel: [ 2720.132900] veth1d5f6d8: renamed from eth0
Sep 27 09:58:03 TEST systemd-udevd[8922]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Sep 27 09:58:03 TEST kernel: [ 2720.202228] br-bc4629d62f80: port 1(veth9fd8479) entered disabled state
Sep 27 09:58:03 TEST networkd-dispatcher[935]: WARNING:Unknown index 380 seen, reloading interface list
Sep 27 09:58:03 TEST kernel: [ 2720.220081] device veth9fd8479 left promiscuous mode
Sep 27 09:58:03 TEST kernel: [ 2720.220086] br-bc4629d62f80: port 1(veth9fd8479) entered disabled state
Sep 27 09:58:03 TEST systemd-udevd[8922]: link_config: could not get ethtool features for veth1d5f6d8
Sep 27 09:58:03 TEST systemd-udevd[8922]: Could not set offload features of veth1d5f6d8: No such device
Sep 27 09:58:03 TEST networkd-dispatcher[935]: ERROR:Unknown interface index 380 seen even after reload
Sep 27 09:58:04 TEST systemd-networkd[971]: br-bc4629d62f80: Lost carrier
jodh-intel commented 6 years ago

/cc @sboeuf, @amshinde, @mcastelino.

sboeuf commented 6 years ago

@amshinde @mcastelino correct me if I'm wrong but I think docker-compose is relying on dockerd to setup things inside the network namespace of the container usually. And IIRC, this does not work well with our VM setup as we don't/can't propagate the same kind of setup inside the VM.

grahamwhaley commented 6 years ago

Good question @sboeuf - and that is not going to surprise me. I think we have not tested Kata with docker compose much/enough. If this is the answer, can we check we have this in our Limitations do, and if not, add it.

jodh-intel commented 6 years ago

+1. But also I wonder if there is a way for us to detect this scenario and fail elegantly?

fredbcode commented 6 years ago

Tried without ipv6 (I have a lot of warnings about) without success, someone can try to reproduce with this simple docker-compose ?

1) docker system prune -a and reboot 2) docker-compose up

Comment and uncomment a network and sometimes image works, sometimes not When I have the error message a process kata-proxy still running (even after docker-compose down)

root 11013 1 0 09:25 ? 00:00:00 /usr/libexec/kata-containers/kata-proxy -listen-socket unix:///run/vc/sbs/c231fb2cab62baedfda31a7876ef7413c3270615a09ee243e1640c49cc66882a/proxy.sock -mux-socket /run/vc/vm/c231fb2cab62baedfda31a7876ef7413c3270615a09ee243e1640c49cc66882a/kata.sock -sandbox c231fb2cab62baedfda31a7876ef7413c3270615a09ee243e1640c49cc66882a -log debug -agent-logs-socket /run/vc/vm/c231fb2cab62baedfda31a7876ef7413c3270615a09ee243e1640c49cc66882a/console.sock

A kill command is needed

I'm pretty sure that there is also a timeout/exhausted resources issue here, because right after the same docker-compose can works (if kata-proxy is well dead)

version: '2.1'
services:
  fred:
    build:
      context: "ubuntu"
    networks:
      - test
      - test1
      - test2
      - test3
  fred1:
    image: registry.test/images/ubuntu
    networks:
      - test1

  fred2:
    image: registry.test/images/ubuntu
    networks:
      - test2

  fred3:
    image: registry.test/images/ubuntu
    networks:
      - test3

networks:
  test:
  test1:
  test2:
  test3:

A complete sequence:

docker-compose up  
Creating network "app_test1" with the default driver
Creating network "app_test" with the default driver
Creating network "app_test3" with the default driver
Creating network "app_test2" with the default driver
Creating app_fred1_1 ... 
Creating app_fred3_1 ... 
Creating app_fred2_1 ... 
Creating app_fred_1 ... 
Creating app_fred1_1
Creating app_fred_1
Creating app_fred3_1
Creating app_fred_1 ... error

ERROR: for app_fred_1  Cannot start service fred: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201e65d4 0 0xc420017400 ether <nil> down 0 0 0 []}}: file exists: unknown

ERROR: for fred  Cannot start service fred: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil>  0xc4201e65d4 0 0xc420017400 ether <nil> down 0 0 0 []}}: file exists: unknown
ERROR: Encountered errors while bringing up the project.
root@TEST:/opt/app# killall kata-proxy
root@TEST:/opt/app# docker-compose down
Removing app_fred2_1 ... done
Removing app_fred_1  ... done
Removing app_fred3_1 ... done
Removing app_fred1_1 ... done
Removing network app_test1
Removing network app_test
Removing network app_test3
Removing network app_test2
root@TEST:/opt/app# docker-compose up
Creating network "app_test1" with the default driver
Creating network "app_test" with the default driver
Creating network "app_test3" with the default driver
Creating network "app_test2" with the default driver
Creating app_fred1_1 ... 
Creating app_fred_1 ... 
Creating app_fred3_1 ... 
Creating app_fred2_1 ... 
Creating app_fred1_1
Creating app_fred_1
Creating app_fred3_1
Creating app_fred_1 ... done
Attaching to app_fred2_1, app_fred1_1, app_fred3_1, app_fred_1
fred2_1  | WARNING: no logs are available with the 'syslog' log driver
fred1_1  | WARNING: no logs are available with the 'syslog' log driver
fred3_1  | WARNING: no logs are available with the 'syslog' log driver
fred_1   | WARNING: no logs are available with the 'syslog' log driver
app_fred2_1 exited with code 0
app_fred1_1 exited with code 0
app_fred3_1 exited with code 0
app_fred_1 exited with code 0
fredbcode commented 6 years ago

Here some tests I made, I'm trying to reproduce what docker-compose does:

A bash script with:

docker pull test.registry/images/ubuntu:16.04
docker pull test.registry/images/ubuntu:18.04

docker network create test1
docker network create test2
docker network create test3
docker network create test4
docker network create test5
docker network create test6

docker run -itd --net test1 --name testu1 test.registry/images/ubuntu:16.04
docker run -itd --net test2 --name testu2 test.registry/images/ubuntu:16.04
docker run -itd --net test3 --name testu3 test.registry/images/ubuntu:18.04
docker run -itd --net test4 --name testu4 test.registry/images/ubuntu:18.04
docker run -itd --net test5 --name testu5 test.registry/images/ubuntu:18.04
docker run -itd --net test6 --name testu6 test.registry/images/ubuntu:18.04

docker network connect test2 testu1
docker network connect test3 testu1
docker network connect test4 testu1
docker network connect test5 testu1
docker network connect test6 testu1

works well !

./test.sh 
16.04: Pulling from images/ubuntu
50513f4bae65: Pull complete 
854cd4146cdf: Pull complete 
3aa6df3c928d: Pull complete 
3bcf156a92e8: Pull complete 
10de78edcdec: Pull complete 
1a7902332604: Pull complete 
91da1c491187: Pull complete 
f6fe36c00817: Pull complete 
37141a578bef: Pull complete 
Digest: sha256:44f8fe6d233cb01205aa2d924a43fc6ebf9fc13d6d95748e092abb504148976d
Status: Downloaded newer image for test.registry/images/ubuntu:16.04
18.04: Pulling from images/ubuntu
09888d3b717d: Pull complete 
197bf3249b88: Pull complete 
e67b16720f7d: Pull complete 
a8357788d59a: Pull complete 
4c7f1cbe991a: Pull complete 
bcafe3c5b913: Pull complete 
6aa96a742a7e: Pull complete 
cd83ffe921b9: Pull complete 
Digest: sha256:7a2f553a2068d0c8ae95876bf22ff2594fda5a94386f54d686717112fed5c92c
Status: Downloaded newer image for test.registry/images/ubuntu:18.04
d50183924fe5b9421363e8bd77d55fb62fe603adf26a0347b8e2523917ec508f
7c7ab49e2d7d04ebd1fb3f82b6a7222cbea715e2ad3cee355727aef233631a3e
7f754ab3abac4edd015597b3d652d38a287c98105975f69de5bcffd9631cf22d
adf716e65a2ee53dac41dbee1c6eed6eec48477ad3567d6b3ae8b842a9dac107
ec1b4b1fa2bf1cd57b85a06ad2b9050cd21aa68a13505368adea14cd1e80e8ec
4191d6ee8c58fc6750affc6258d8bf30215c6de5e2d08ad4ae85a27bc9d6af1d
a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad
ba5cbee4c0e15f8eff79c97ad40fef8b77599e46a26d230d02e89aa18a28a410
f8d30f156a0a617451c2be18c7778a890617189e41b7dd7eb55172fd85223126
03ac095529891c067dd779ba31aed7e26104d118670857a7c95fc87c1318f96a
d420602a07b2a3f833616127ef99590417fb08d873f666bdedc022e6a0e36af2
02f778dd57feb0f6f55548ab33652eb76cb0e0f1bd548660f88ad942a8c966d3

docker inspect testu1

[
    {
        "Id": "a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad",
        "Created": "2018-10-04T08:01:07.019399113Z",
        "Path": "/bin/bash",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 5918,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2018-10-04T08:01:09.127358791Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:c32058bebc576223f293a7f02e55b9752bb1045edcd3e01612689ad08aa5acd9",
        "ResolvConfPath": "/var/lib/docker/containers/a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad/hostname",
        "HostsPath": "/var/lib/docker/containers/a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad/hosts",
        "LogPath": "",
        "Name": "/testu1",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "syslog",
                "Config": {
                    "syslog-address": "udp://rns-sv-exploit-001.admin:514",
                    "tag": "{{.Name}}"
                }
            },
            "NetworkMode": "test1",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "kata-runtime",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/14f570f6f8d3ace4307f1d9713c2ed474120ce69d259a5d3edd705c0164c4f18-init/diff:/var/lib/docker/overlay2/f0e1ddd045b79425da24fc000f12684e4d90397f1f57b5edf18d4c0fe4f86d6f/diff:/var/lib/docker/overlay2/6acf8d1674a94f2a9dafb9c417d33e9358922092198b84e0bc533984f14ddfbe/diff:/var/lib/docker/overlay2/c13b51b30f06f737308371141252b2223b15682e99c6bfb2970f91dab536d32c/diff:/var/lib/docker/overlay2/c52e355aa61eb36d271d4b4bca0cb3f8ad628e5eb3502b65ccd43892b2b5ca2e/diff:/var/lib/docker/overlay2/a5118628f9f643a10db743d454f360784f0c8fdbaed00553532daa0d11108674/diff:/var/lib/docker/overlay2/a8a0ef248466ed04d9ff93914df4502057d4e96df4092aee1e496b3e4a68e7e7/diff:/var/lib/docker/overlay2/74de80d7090ca44f286d7b900a7cbb75876479b76ce4d56dbaeb6cea6d2985e9/diff:/var/lib/docker/overlay2/4a22b5026632d77f15b1f446730d18e405eadb3d131a0752d81a7deaa0f9316d/diff:/var/lib/docker/overlay2/82d18854e6a535ecf63cade51ff697c7ef06b4c71f48680c12c11413ea91d3b5/diff",
                "MergedDir": "/var/lib/docker/overlay2/14f570f6f8d3ace4307f1d9713c2ed474120ce69d259a5d3edd705c0164c4f18/merged",
                "UpperDir": "/var/lib/docker/overlay2/14f570f6f8d3ace4307f1d9713c2ed474120ce69d259a5d3edd705c0164c4f18/diff",
                "WorkDir": "/var/lib/docker/overlay2/14f570f6f8d3ace4307f1d9713c2ed474120ce69d259a5d3edd705c0164c4f18/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "a9ab1e0a9486",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DEBIAN_FRONTEND=noninteractive"
            ],
            "Cmd": [
                "/bin/bash"
            ],
            "ArgsEscaped": true,
            "Image": "test.registry/images/ubuntu:16.04",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "build_date.ubuntu": "2018-07-26T08:51:08+00:00",
                "commit.ubuntu": "5857e7b5a75204a411678d8434a1f7e59bd553f9",
                "version.ubuntu": "20170915"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "608b8228387d4be30026671ed833c56e215ba2da95303ce7be17a32c563d458d",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/608b8228387d",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "test1": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "a9ab1e0a9486"
                    ],
                    "NetworkID": "d50183924fe5b9421363e8bd77d55fb62fe603adf26a0347b8e2523917ec508f",
                    "EndpointID": "cfbca2d7d522508d1c6e4d335b065323c1d464d2c9e047352fb9e9e5cd6fb140",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:12:00:02",
                    "DriverOpts": null
                },
                "test2": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": [
                        "a9ab1e0a9486"
                    ],
                    "NetworkID": "7c7ab49e2d7d04ebd1fb3f82b6a7222cbea715e2ad3cee355727aef233631a3e",
                    "EndpointID": "0f2a4c40447135e2cd701c9759f741182d41c8222a0460f057552e35ae02efe8",
                    "Gateway": "172.19.0.1",
                    "IPAddress": "172.19.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:13:00:03",
                    "DriverOpts": null
                },
                "test3": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": [
                        "a9ab1e0a9486"
                    ],
                    "NetworkID": "7f754ab3abac4edd015597b3d652d38a287c98105975f69de5bcffd9631cf22d",
                    "EndpointID": "b21bfb0e252502c193584fa59ab0425ba673d7a5d77f3e1de6d870c537fa44f9",
                    "Gateway": "172.20.0.1",
                    "IPAddress": "172.20.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:14:00:03",
                    "DriverOpts": null
                },
                "test4": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": [
                        "a9ab1e0a9486"
                    ],
                    "NetworkID": "adf716e65a2ee53dac41dbee1c6eed6eec48477ad3567d6b3ae8b842a9dac107",
                    "EndpointID": "0f24452f8d6038391110553bcaaff74698071d3a94c50420d6e2ca89b297d36e",
                    "Gateway": "172.21.0.1",
                    "IPAddress": "172.21.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:15:00:03",
                    "DriverOpts": null
                },
                "test5": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": [
                        "a9ab1e0a9486"
                    ],
                    "NetworkID": "ec1b4b1fa2bf1cd57b85a06ad2b9050cd21aa68a13505368adea14cd1e80e8ec",
                    "EndpointID": "48d5870d26bfdec9628bc6eb90a7e5ee242103c1b7cc0de5673ca39b249d3c26",
                    "Gateway": "172.22.0.1",
                    "IPAddress": "172.22.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:16:00:03",
                    "DriverOpts": null
                },
                "test6": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": [
                        "a9ab1e0a9486"
                    ],
                    "NetworkID": "4191d6ee8c58fc6750affc6258d8bf30215c6de5e2d08ad4ae85a27bc9d6af1d",
                    "EndpointID": "078c6f9a2a40d66d45ca160e456635640d25fd4ac0f90bf2326657e05d171acf",
                    "Gateway": "172.23.0.1",
                    "IPAddress": "172.23.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:17:00:03",
                    "DriverOpts": null
                }
            }
        }
    }
]

And FI same issue with: https://github.com/docker/compose/releases/tag/1.23.0-rc1

Let me know if I can help you in any way

fredbcode commented 6 years ago

Latest dockerd build : docker version 18.06.1-ce, build e68fc7a

I have this message in syslog only when the bug occurs:

Oct 4 08:59:38 TEST-001 dockerd[28865]: time="2018-10-04T08:59:38.993840460Z" level=error msg="1337c9b768f7bfaff38e10ae54156ddedd1e064e8a71e69eeda9b91bac9beb2d cleanup: failed to delete container from containerd: no such container" Oct 4 08:59:38 TEST-001 dockerd[28865]: time="2018-10-04T08:59:38.993900278Z" level=error msg="Handler for POST /v1.24/containers/1337c9b768f7bfaff38e10ae54156ddedd1e064e8a71e69eeda9b91bac9beb2d/start returned error: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:1b:00:02 broadcast|multicast 4098 0 0 0xc4201dc5d4 0 0xc4200172c0 ether down 0 0 0 []}}: file exists: unknown"

Docker in debug mode

DEBU[2018-10-04T09:15:16.646247764Z] Released address PoolID:LocalDefault/192.168.80.0/20, Address:192.168.80.3 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.80.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xd0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:4 
DEBU[2018-10-04T09:15:16.646674155Z] Revoking external connectivity on endpoint app_fred_1_b9a69d260e1a (4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc) 
DEBU[2018-10-04T09:15:16.648281440Z] DeleteConntrackEntries purged ipv4:0, ipv6:0 
DEBU[2018-10-04T09:15:16.706595927Z] Programming external connectivity on endpoint app_fred_1_b9a69d260e1a (b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8) 
DEBU[2018-10-04T09:15:16.707361953Z] 4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc (fdf2892).deleteSvcRecords(app_fred_1_b9a69d260e1a, 192.168.96.3, <nil>, true) updateSvcRecord sid:4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc  
DEBU[2018-10-04T09:15:16.707450537Z] 4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc (fdf2892).deleteSvcRecords(082f8e75b4df, 192.168.96.3, <nil>, false) updateSvcRecord sid:4400c5f585c7**3722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc  
DEBU[2018-10-04T09:15:16.707475549Z] 4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc (fdf2892).deleteSvcRecords(fred, 192.168.96.3, <nil>, false) updateSvcRecord sid:4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc  
DEBU[2018-10-04T09:15:16.790172433Z] Releasing addresses for endpoint app_fred_1_b9a69d260e1a's interface on network app_test2 
DEBU[2018-10-04T09:15:16.790205061Z] ReleaseAddress(LocalDefault/192.168.96.0/20, 192.168.96.3) 
DEBU[2018-10-04T09:15:16.790237425Z] Released address PoolID:LocalDefault/192.168.96.0/20, Address:192.168.96.3 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.96.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xd0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:4 
DEBU[2018-10-04T09:15:16.790621371Z] Revoking external connectivity on endpoint app_fred_1_b9a69d260e1a (b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8) 
DEBU[2018-10-04T09:15:16.792236669Z] DeleteConntrackEntries purged ipv4:0, ipv6:0 
DEBU[2018-10-04T09:15:16.830938459Z] b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8 (f70f6b5).deleteSvcRecords(app_fred_1_b9a69d260e1a, 192.168.112.3, <nil>, true) updateSvcRecord sid:b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8  
DEBU[2018-10-04T09:15:16.830990597Z] b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8 (f70f6b5).deleteSvcRecords(082f8e75b4df, 192.168.112.3, <nil>, false) updateSvcRecord sid:b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8  
DEBU[2018-10-04T09:15:16.831021853Z] b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8 (f70f6b5).deleteSvcRecords(fred, 192.168.112.3, <nil>, false) updateSvcRecord sid:b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8  
DEBU[2018-10-04T09:15:16.902267594Z] Releasing addresses for endpoint app_fred_1_b9a69d260e1a's interface on network app_test3 
DEBU[2018-10-04T09:15:16.902290954Z] ReleaseAddress(LocalDefault/192.168.112.0/20, 192.168.112.3) 
DEBU[2018-10-04T09:15:16.902317283Z] Released address PoolID:LocalDefault/192.168.112.0/20, Address:192.168.112.3 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.112.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xd0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:4 
ERRO[2018-10-04T09:15:16.981674628Z] 082f8e75b4df064d8d4d798b5c518e7c81314bdeb11441ce19b83295dcc09498 cleanup: failed to delete container from containerd: no such container 
ERRO[2018-10-04T09:15:16.981733245Z] Handler for POST /v1.24/containers/082f8e75b4df064d8d4d798b5c518e7c81314bdeb11441ce19b83295dcc09498/start returned error: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:c0:a8:70:03 broadcast|multicast 4098 0 0 <nil>  0xc4201dc5d4 0 0xc4200172c0 ether <nil> down 0 0 0 []}}: file exists: unknown 

Same without error (with only two networks)

DEBU[2018-10-04T09:36:31.709013698Z] 5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34 (228ea3b).deleteSvcRecords(app_fred3_1, 192.168.224.2, <nil>, true) updateSvcRecord sid:5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34  
DEBU[2018-10-04T09:36:31.709067150Z] 5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34 (228ea3b).deleteSvcRecords(9c23b9033922, 192.168.224.2, <nil>, false) updateSvcRecord sid:5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34  
DEBU[2018-10-04T09:36:31.709088788Z] 5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34 (228ea3b).deleteSvcRecords(fred3, 192.168.224.2, <nil>, false) updateSvcRecord sid:5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34  
DEBU[2018-10-04T09:36:31.788133198Z] Releasing addresses for endpoint app_fred3_1's interface on network app_test3 
DEBU[2018-10-04T09:36:31.788181561Z] ReleaseAddress(LocalDefault/192.168.224.0/20, 192.168.224.2) 
DEBU[2018-10-04T09:36:31.788223099Z] Released address PoolID:LocalDefault/192.168.224.0/20, Address:192.168.224.2 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.224.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xe0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:3 
DEBU[0227] event published                               ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete
DEBU[0228] event published                               ns=moby topic="/tasks/create" type=containerd.events.TaskCreate
DEBU[2018-10-04T09:36:32.144238019Z] event                                         module=libcontainerd namespace=moby topic=/tasks/create
DEBU[0228] event published                               ns=moby topic="/tasks/start" type=containerd.events.TaskStart
DEBU[2018-10-04T09:36:32.201663950Z] event                                         module=libcontainerd namespace=moby topic=/tasks/start
DEBU[2018-10-04T09:36:32.247504205Z] Calling POST /v1.24/containers/8a30eeae9bb25bf39e3f4619470c79476d83da04afd302361d16ef8f659f8352/wait 
DEBU[2018-10-04T09:36:32.248237596Z] Calling POST /v1.24/containers/385d977c0a6baf95999e6f4da95678a05b83bb227f1596f0de226d378854bd5f/wait 
DEBU[2018-10-04T09:36:32.251847915Z] Calling POST /v1.24/containers/9c23b90339228c3b6315405fa16fb4f12aae83dbcbb8ce2a0c6eaee058d92b17/wait 
DEBU[2018-10-04T09:36:32.254077612Z] Calling POST /v1.24/containers/4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437/wait 
DEBU[2018-10-04T09:36:32.254794565Z] Calling GET /v1.24/events?filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dapp%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D 
ERRO[0005] failed to kill init's children                error="OCI runtime killall failed: Container not ready, running or paused, impossible to signal the container" id=4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437 namespace=moby path="/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437" pid=11055
DEBU[0228] event published                               ns=moby topic="/tasks/exit" type=containerd.events.TaskExit
DEBU[2018-10-04T09:36:32.417929268Z] event                                         module=libcontainerd namespace=moby topic=/tasks/exit
INFO[0228] shim reaped                                   id=4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437
DEBU[0228] event published                               ns=moby topic="/tasks/delete" type=containerd.events.TaskDelete
DEBU[2018-10-04T09:36:32.565223072Z] event                                         module=libcontainerd namespace=moby topic=/tasks/delete
INFO[2018-10-04T09:36:32.565267988Z] ignoring event                                module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
DEBU[2018-10-04T09:36:32.565750086Z] Revoking external connectivity on endpoint app_fred2_1 (2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede) 
DEBU[2018-10-04T09:36:32.567607558Z] Calling GET /v1.24/containers/4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437/json 
DEBU[2018-10-04T09:36:32.568601235Z] DeleteConntrackEntries purged ipv4:0, ipv6:0 
DEBU[2018-10-04T09:36:32.633011532Z] 2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede (310367f).deleteSvcRecords(app_fred2_1, 192.168.240.2, <nil>, true) updateSvcRecord sid:2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede  
DEBU[2018-10-04T09:36:32.633080246Z] 2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede (310367f).deleteSvcRecords(4ce4a9495b1f, 192.168.240.2, <nil>, false) updateSvcRecord sid:2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede  
DEBU[2018-10-04T09:36:32.633102798Z] 2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede (310367f).deleteSvcRecords(fred2, 192.168.240.2, <nil>, false) updateSvcRecord sid:2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede  
DEBU[2018-10-04T09:36:32.716011196Z] Releasing addresses for endpoint app_fred2_1's interface on network app_test2 
DEBU[2018-10-04T09:36:32.716044199Z] ReleaseAddress(LocalDefault/192.168.240.0/20, 192.168.240.2) 
DEBU[2018-10-04T09:36:32.716082403Z] Released address PoolID:LocalDefault/192.168.240.0/20, Address:192.168.240.2 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.240.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xe0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:4 
DEBU[0228] event published                               ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete
DEBU[2018-10-04T09:36:33.000437658Z] Client context cancelled, stop sending events 

It works, although I have this in debug message (but nothing in prompt after docker-compose up)

ERRO[0005] failed to kill init's children error="OCI runtime killall failed: Container not ready, running or paused, impossible to signal the container" id=4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437 namespace=moby path="/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437"

Gabasjob commented 6 years ago

Hello there. Any update on this bug ?

Is there any workaround to use kata-runtime with docker-compose ?

alebourdoulous commented 6 years ago

Hello, We look forward to hearing from you about this problem and hope to have a solution soon. thank you

grahamwhaley commented 6 years ago

Ping @amshinde @mcastelino - any thoughts, input or requests for more tests/info here? I think @sboeuf had a pertinent question back up at https://github.com/kata-containers/runtime/issues/785#issuecomment-425330335 :

@amshinde @mcastelino correct me if I'm wrong but I think docker-compose is relying on dockerd to setup things inside the network namespace of the container usually. And IIRC, this does not work well with our VM setup as we don't/can't propagate the same kind of setup inside the VM.

fredbcode commented 6 years ago

Le 10/10/2018 à 10:42, Frank Cao a écrit :

We have integrated with k8s. For docker-compose, I'm afraid most of use haven't used it.

You can easily reproduce with a basic docker-compose here:

https://github.com/kata-containers/runtime/issues/785#issuecomment-425382208

Gabasjob commented 6 years ago

Hello there, here is some steps to reproduce the bug without docker-compose :

docker network create net1
docker network create net2
docker network create net3
docker create --name bb --network net1 busybox
docker network connect net2 bb
docker network connect net3 bb
docker start bb

output result :

Error response from daemon: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:14:00:02 broadcast|multicast 4098 0 0 <nil>  0xc4201dc5d4 0 0xc4200172c0 ether <nil> down 0 0 0 []}}: file exists: unknown
Error: failed to start containers: bb
amshinde commented 6 years ago

@sboeuf @grahamwhaley We do have a current limitation with docker-compose that causes issues in dns name resolution due to the use of dockerd. But the above issue looks unrelated. I'll take a look to see whats going on.

amshinde commented 6 years ago

@fredbcode @Gabasjob This patch to the runtime should solve your issue for now: https://github.com/amshinde/kata-runtime/commit/d28b5b42b0d7f958182265c2a983da073d8e50dd

Can you give that a shot. I am planning to open a PR with that fix.

fredbcode commented 6 years ago

Great, can you share with us a binary please ?

jodh-intel commented 6 years ago

@fredbcode - we don't really have the facility to do that. But all you'd need to do is:

fredbcode commented 6 years ago

I will soon, thanks @Gabasjob little "fix" for compilation

go get -u github.com/kata-containers/runtime/virtcontainers
cd yourpath/runtime
curl -sL https://github.com/amshinde/kata-runtime/commit/d28b5b42b0d7f958182265c2a983da073d8e50dd.patch | git apply
make && sudo -E PATH=$PATH make install
Gabasjob commented 6 years ago

@jodh-intel Works well !

grahamwhaley commented 6 years ago

Excellent! - thanks @Gabasjob , and kudos @amshinde Note @Gabasjob - you will still have the dns resolution issue that @amshinde referenced above - which is listed (under swam) at: https://github.com/kata-containers/documentation/blob/master/Limitations.md#docker-swarm-support which points to some details at: https://github.com/kata-containers/runtime/issues/175

fredbcode commented 6 years ago

I also confirm, great !

jodh-intel commented 6 years ago

Great - sounds like #814 will dtrt then ;)

Gabasjob commented 6 years ago

@grahamwhaley dns issue still present I'm affraid : docker-compose.yml :

version: '2'
services:
  bb1:
    image: busybox
    networks:
      -  net1
    command: "ping bb2"
  bb2:
    image: busybox
    networks:
      -  net1
    command: "sleep 30"

networks:
  net1:

output without kata-runtime :

$ docker-compose up
Creating network "bugkata_net1" with the default driver
Recreating bugkata_bb2_1 ...
Recreating bugkata_bb1_1 ...
Recreating bugkata_bb1_1
Recreating bugkata_bb2_1 ... done
Attaching to bugkata_bb1_1, bugkata_bb2_1
bb1_1  | PING bb2 (172.21.0.3): 56 data bytes  
bb1_1  | 64 bytes from 172.21.0.3: seq=0 ttl=64 time=1004.177 ms
bb1_1  | 64 bytes from 172.21.0.3: seq=1 ttl=64 time=4.111 ms
bb1_1  | 64 bytes from 172.21.0.3: seq=2 ttl=64 time=0.079 ms
bb1_1  | 64 bytes from 172.21.0.3: seq=3 ttl=64 time=0.065 ms

output with kata-runtime :

$ docker-compose up
Creating network "bugkata_net1" with the default driver
Creating bugkata_bb2_1 ...
Creating bugkata_bb1_1 ...
Creating bugkata_bb1_1
Creating bugkata_bb2_1 ... done
Attaching to bugkata_bb1_1, bugkata_bb2_1
bb1_1  | ping: bad address 'bb2'
bugkata_bb1_1 exited with code 1
grahamwhaley commented 6 years ago

@Gabasjob yep - that is a known limitation. /cc @amshinde @mcastelino - do we have any known workarounds, or long term plans on that?

amshinde commented 6 years ago

@grahamwhaley Unfortunately there are no workarounds for this due to the way name resolution is handled in docker for custom networks. I'll take a look at this with @mcastelino to come up with a plan.