fredbcode
commented
6 years ago No not related with #702
Closed fredbcode closed 3 years ago
fredbcode
commented
6 years ago No not related with #702
jodh-intel
commented
6 years ago Hi @fredbcode - thanks for reporting. However, it's difficult to see what is happening because the proxy log errors captured when you ran kata-collect-data.sh show that agent debug was not enabled and the attached logs also do not have agent debug enabled.
If you are running on a dev / test box and are happy to lose your journal log, it would be useful if you could clear the journal and re-run:
sudo kata-collect-data.sh and paste here.
caoruidong
commented
6 years ago What are interfaces in all containers? Seems there is already a "eth2" interface?
fredbcode
commented
6 years ago There is no ethX at all but enspX, yes network is ready and the same docker-compose works well without kataruntime (on same machine of course)
I don't known where kata find this eth2
fredbcode
commented
6 years ago @jodh-intel I will provide logs tomorrow
jodh-intel
commented
6 years ago @fredbcode - thanks ;)
fredbcode
commented
6 years ago Haveged installed (see #702)
cat /proc/sys/kernel/random/entropy_avail 4066
Full log enabled and purged
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether d0:94:66:16:91:ff brd ff:ff:ff:ff:ff:ff
inet x.x.x.x/26 brd x.x.x.x scope global eno1
valid_lft forever preferred_lft forever
inet6 fe80::d294:66ff:fe16:91ff/64 scope link
valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether d0:94:66:16:92:00 brd ff:ff:ff:ff:ff:ff
inet x.x.x.x/30 brd x.x.x.x scope global eno2
valid_lft forever preferred_lft forever
inet6 fe80::d294:66ff:fe16:9200/64 scope link
valid_lft forever preferred_lft forever
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether d0:94:66:16:92:01 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether d0:94:66:16:92:02 brd ff:ff:ff:ff:ff:ff
6: idrac: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether d0:94:66:16:92:04 brd ff:ff:ff:ff:ff:ff Sorry I can't use kata-log-parser on this sever.
kata.txt proxy.log runtime.log shim.log
ERROR: for app_viewbox_1 Cannot start service viewbox: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:15Creating app_apache_1 ... error
ERROR: for app_apache_1 Cannot start service apache: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil> 0xc4201dc5d4 0 0xc420017440 ether <nil> down 0 0 0 []}}: file exists: unknown
ERROR: for viewbox Cannot start service viewbox: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:15:00:03 broadcast|multicast 4098 0 0 <nil> 0xc4201dc5d4 0 0xc420017440 ether <nil> down 0 0 0 []}}: file exists: unknown
ERROR: for apache Cannot start service apache: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil> 0xc4201dc5d4 0 0xc420017440 ether <nil> down 0 0 0 []}}: file exists: unknown
version: '2.1'
services:
apache:
build:
context: "apache-rp"
networks:
- office
- talkmost
- viewbox
- dokuwiki
ports:
- "x.x.x.x:443:443"
office:
image: registry.test/images/squid:current
networks:
- office
restart: unless-stopped
talkmost:
image: registry.test/images/ubuntu
networks:
- talkmost
restart: unless-stopped
viewbox:
image: registry.test/images/nginx
networks:
- viewbox
restart: unless-stopped
networks:
office:
talkmost:
viewbox:
dokuwiki:
docker-compose up
Starting app_apache_1 ...
app_viewbox_1 is up-to-date
Starting app_apache_1
app_office_1 is up-to-date
Starting app_apache_1 ... error
ERROR: for app_apache_1 Cannot start service apache: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:13:00:03 broadcast|multicast 4098 0 0 <nil> 0xc4201e65d4 0 0xc420017460 ether <nil> down 0 0 0 []}}: file exists: unknown
ERROR: for apache Cannot start service apache: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:13:00:03 broadcast|multicast 4098 0 0 <nil> 0xc4201e65d4 0 0xc420017460 ether <nil> down 0 0 0 []}}: file exists: unknown
ERROR: Encountered errors while bringing up the project.
docker-compose up without kata
docker-compose up
Creating network "app_dokuwiki" with the default driver
Creating network "app_viewbox" with the default driver
Creating network "app_office" with the default driver
Creating network "app_talkmost" with the default driver
Creating app_apache_1 ...
Creating app_viewbox_1 ...
Creating app_office_1 ...
Creating app_talkmost_1 ...
Creating app_viewbox_1
Creating app_office_1
Creating app_talkmost_1
Creating app_apache_1 ... done
Attaching to app_viewbox_1, app_talkmost_1, app_office_1, app_apache_1
office_1 | WARNING: Cannot write log file: stdio:/dev/stdout
office_1 | stdio:/dev/stdout: No such file or directory
office_1 | messages will be sent to 'stderr'.
office_1 | 2018/09/27 08:28:30| Set Current Directory to /var/spool/squid
office_1 | WARNING: Cannot write log file: stdio:/dev/stdout
office_1 | stdio:/dev/stdout: No such file or directory
office_1 | messages will be sent to 'stderr'.
office_1 | 2018/09/27 08:28:30| WARNING: Closing open FD 2
office_1 | 2018/09/27 08:28:30| Starting Squid Cache version 3.5.12 for x86_64-pc-linux-gnu...
office_1 | 2018/09/27 08:28:30| Service Name: squid
office_1 | 2018/09/27 08:28:30| Process ID 9
office_1 | 2018/09/27 08:28:30| Process Roles: master worker
office_1 | 2018/09/27 08:28:30| With 1048576 file descriptors available
office_1 | 2018/09/27 08:28:30| Initializing IP Cache...
office_1 | 2018/09/27 08:28:30| DNS Socket created at [::], FD 7
office_1 | 2018/09/27 08:28:30| DNS Socket created at 0.0.0.0, FD 8
office_1 | 2018/09/27 08:28:30| Adding domain rie from /etc/resolv.conf
office_1 | 2018/09/27 08:28:30| Adding nameserver 127.0.0.11 from /etc/resolv.conf
office_1 | 2018/09/27 08:28:30| Adding ndots 1 from /etc/resolv.conf
office_1 | 2018/09/27 08:28:30| Logfile: opening log stdio:/dev/stdout
office_1 | 2018/09/27 08:28:30| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
office_1 | 2018/09/27 08:28:30| Store logging disabled
office_1 | 2018/09/27 08:28:30| Swap maxSize 0 + 262144 KB, estimated 20164 objects
office_1 | 2018/09/27 08:28:30| Target number of buckets: 1008
office_1 | 2018/09/27 08:28:30| Using 8192 Store buckets
office_1 | 2018/09/27 08:28:30| Max Mem size: 262144 KB
office_1 | 2018/09/27 08:28:30| Max Swap size: 0 KB
office_1 | 2018/09/27 08:28:30| Using Least Load store dir selection
office_1 | 2018/09/27 08:28:30| Set Current Directory to /var/spool/squid
office_1 | 2018/09/27 08:28:30| Finished loading MIME types and icons.
office_1 | 2018/09/27 08:28:30| HTCP Disabled.
office_1 | 2018/09/27 08:28:30| commBind: Cannot bind socket FD 11 to [::1]: (99) Cannot assign requested address
office_1 | 2018/09/27 08:28:30| commBind: Cannot bind socket FD 12 to [::1]: (99) Cannot assign requested address
office_1 | 2018/09/27 08:28:30| ERROR: Failed to create helper child read FD: UDP[::1]
office_1 | 2018/09/27 08:28:30| Squid plugin modules loaded: 0
office_1 | 2018/09/27 08:28:30| Adaptation support is off.
office_1 | 2018/09/27 08:28:30| Accepting HTTP Socket connections at local=[::]:3128 remote=[::] FD 10 flags=9
office_1 | 2018/09/27 08:28:31| storeLateRelease: released 0 objects
apache_1 | [Thu Sep 27 08:28:32.487388 2018] [core:error] [pid 10:tid 139667229403008] (EAI 2)Name or service not known: AH00547: Could not resolve host name block.scn.rie.gouv.fr -- ignoring!
app_talkmost_1 exited with code 0
app_talkmost_1 exited with code 0
app_talkmost_1 exited with code 0
app_talkmost_1 exited with code 0
app_talkmost_1 exited with code 0This message is useless: office_1 | 2018/09/27 08:28:30| commBind: Cannot bind socket FD 12 to [::1]: (99) Cannot assign requested address It's about squid and ipv6, same problem with another image without network needed ...
fredbcode
commented
6 years ago Without ports:
grahamwhaley
commented
6 years ago @amshinde any thoughts from the network side on the eth1/eth2 info here?
caoruidong
commented
6 years ago Maybe related to different nic naming "ethXX" vs "enoXX".
fredbcode
commented
6 years ago I guess I found something very interesting !
version: '2.1'
services:
fred:
build:
context: "ubuntu"
networks:
- test
- test1
- test2
office:
image: registry.test/images/ubuntu
networks:
- test
restart: unless-stopped
talkmost:
image: registry.test/images/ubuntu
networks:
- test1
restart: unless-stopped
talkmost1:
image: registry.test/images/ubuntu
networks:
- test2
restart: unless-stopped
networks:
test:
test1:
test2:docker-compose up
Creating network "app_test1" with the default driver
Creating network "app_test" with the default driver
Creating network "app_test2" with the default driver
Creating app_talkmost1_1 ...
Creating app_fred_1 ...
Creating app_talkmost_1 ...
Creating app_office_1 ...
Creating app_talkmost1_1
Creating app_talkmost_1
Creating app_office_1
Creating app_fred_1 ... error
ERROR: for app_fred_1 Cannot start service fred: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth2 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:Creating app_talkmost1_1 ... done
ERROR: for fred Cannot start service fred: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth2 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil> 0xc4201e05d4 0 0xc420017440 ether <nil> down 0 0 0 []}}: file exists: unknown
ERROR: Encountered errors while bringing up the project.But with only two internal networks:
docker-compose up
Creating network "app_test1" with the default driver
Creating network "app_test" with the default driver
Creating network "app_test2" with the default driver
Creating app_talkmost1_1 ...
Creating app_talkmost_1 ...
Creating app_fred_1 ...
Creating app_office_1 ...
Creating app_talkmost1_1
Creating app_fred_1
Creating app_talkmost_1
Creating app_office_1 ... done
Attaching to app_talkmost1_1, app_fred_1, app_talkmost_1, app_office_1
talkmost1_1 | WARNING: no logs are available with the 'syslog' log driver
fred_1 | WARNING: no logs are available with the 'syslog' log driver
talkmost_1 | WARNING: no logs are available with the 'syslog' log driver
office_1 | WARNING: no logs are available with the 'syslog' log driver
app_fred_1 exited with code 0
app_office_1 exited with code 0
app_talkmost1_1 exited with code 0
app_talkmost_1 exited with code 0version: '2.1'
services:
fred:
build:
context: "ubuntu"
networks:
- test
- test1
# - test2 **-> HERE**
office:
image: registry.test/images/ubuntu
networks:
- test
restart: unless-stopped
talkmost:
image: registry.test/images/ubuntu
networks:
- test1
restart: unless-stopped
talkmost1:
image: registry.test/images/ubuntu
networks:
- test2
restart: unless-stopped
networks:
test:
test1:
test2:I'm exactly in the same situation that before with issue #702 with Kata only one or two containers works well (but here my entropy is 4066)
My Dockerfile contains only a FROM to registry - ubuntu image -
fredbcode
commented
6 years ago Of course I mean, max two containers works well -> containers with network in this case
fredbcode
commented
6 years ago Syslog:
Sep 27 09:58:03 TEST kernel: [ 2720.132900] veth1d5f6d8: renamed from eth0
Sep 27 09:58:03 TEST systemd-udevd[8922]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Sep 27 09:58:03 TEST kernel: [ 2720.202228] br-bc4629d62f80: port 1(veth9fd8479) entered disabled state
Sep 27 09:58:03 TEST networkd-dispatcher[935]: WARNING:Unknown index 380 seen, reloading interface list
Sep 27 09:58:03 TEST kernel: [ 2720.220081] device veth9fd8479 left promiscuous mode
Sep 27 09:58:03 TEST kernel: [ 2720.220086] br-bc4629d62f80: port 1(veth9fd8479) entered disabled state
Sep 27 09:58:03 TEST systemd-udevd[8922]: link_config: could not get ethtool features for veth1d5f6d8
Sep 27 09:58:03 TEST systemd-udevd[8922]: Could not set offload features of veth1d5f6d8: No such device
Sep 27 09:58:03 TEST networkd-dispatcher[935]: ERROR:Unknown interface index 380 seen even after reload
Sep 27 09:58:04 TEST systemd-networkd[971]: br-bc4629d62f80: Lost carrier/cc @sboeuf, @amshinde, @mcastelino.
sboeuf
commented
6 years ago @amshinde @mcastelino correct me if I'm wrong but I think docker-compose is relying on dockerd to setup things inside the network namespace of the container usually. And IIRC, this does not work well with our VM setup as we don't/can't propagate the same kind of setup inside the VM.
grahamwhaley
commented
6 years ago Good question @sboeuf - and that is not going to surprise me. I think we have not tested Kata with docker compose much/enough. If this is the answer, can we check we have this in our Limitations do, and if not, add it.
jodh-intel
commented
6 years ago +1. But also I wonder if there is a way for us to detect this scenario and fail elegantly?
fredbcode
commented
6 years ago Tried without ipv6 (I have a lot of warnings about) without success, someone can try to reproduce with this simple docker-compose ?
1) docker system prune -a and reboot 2) docker-compose up
Comment and uncomment a network and sometimes image works, sometimes not When I have the error message a process kata-proxy still running (even after docker-compose down)
root 11013 1 0 09:25 ? 00:00:00 /usr/libexec/kata-containers/kata-proxy -listen-socket unix:///run/vc/sbs/c231fb2cab62baedfda31a7876ef7413c3270615a09ee243e1640c49cc66882a/proxy.sock -mux-socket /run/vc/vm/c231fb2cab62baedfda31a7876ef7413c3270615a09ee243e1640c49cc66882a/kata.sock -sandbox c231fb2cab62baedfda31a7876ef7413c3270615a09ee243e1640c49cc66882a -log debug -agent-logs-socket /run/vc/vm/c231fb2cab62baedfda31a7876ef7413c3270615a09ee243e1640c49cc66882a/console.sock
A kill command is needed
I'm pretty sure that there is also a timeout/exhausted resources issue here, because right after the same docker-compose can works (if kata-proxy is well dead)
version: '2.1'
services:
fred:
build:
context: "ubuntu"
networks:
- test
- test1
- test2
- test3
fred1:
image: registry.test/images/ubuntu
networks:
- test1
fred2:
image: registry.test/images/ubuntu
networks:
- test2
fred3:
image: registry.test/images/ubuntu
networks:
- test3
networks:
test:
test1:
test2:
test3:A complete sequence:
docker-compose up
Creating network "app_test1" with the default driver
Creating network "app_test" with the default driver
Creating network "app_test3" with the default driver
Creating network "app_test2" with the default driver
Creating app_fred1_1 ...
Creating app_fred3_1 ...
Creating app_fred2_1 ...
Creating app_fred_1 ...
Creating app_fred1_1
Creating app_fred_1
Creating app_fred3_1
Creating app_fred_1 ... error
ERROR: for app_fred_1 Cannot start service fred: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil> 0xc4201e65d4 0 0xc420017400 ether <nil> down 0 0 0 []}}: file exists: unknown
ERROR: for fred Cannot start service fred: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:12:00:03 broadcast|multicast 4098 0 0 <nil> 0xc4201e65d4 0 0xc420017400 ether <nil> down 0 0 0 []}}: file exists: unknown
ERROR: Encountered errors while bringing up the project.
root@TEST:/opt/app# killall kata-proxy
root@TEST:/opt/app# docker-compose down
Removing app_fred2_1 ... done
Removing app_fred_1 ... done
Removing app_fred3_1 ... done
Removing app_fred1_1 ... done
Removing network app_test1
Removing network app_test
Removing network app_test3
Removing network app_test2
root@TEST:/opt/app# docker-compose up
Creating network "app_test1" with the default driver
Creating network "app_test" with the default driver
Creating network "app_test3" with the default driver
Creating network "app_test2" with the default driver
Creating app_fred1_1 ...
Creating app_fred_1 ...
Creating app_fred3_1 ...
Creating app_fred2_1 ...
Creating app_fred1_1
Creating app_fred_1
Creating app_fred3_1
Creating app_fred_1 ... done
Attaching to app_fred2_1, app_fred1_1, app_fred3_1, app_fred_1
fred2_1 | WARNING: no logs are available with the 'syslog' log driver
fred1_1 | WARNING: no logs are available with the 'syslog' log driver
fred3_1 | WARNING: no logs are available with the 'syslog' log driver
fred_1 | WARNING: no logs are available with the 'syslog' log driver
app_fred2_1 exited with code 0
app_fred1_1 exited with code 0
app_fred3_1 exited with code 0
app_fred_1 exited with code 0Here some tests I made, I'm trying to reproduce what docker-compose does:
A bash script with:
docker pull test.registry/images/ubuntu:16.04
docker pull test.registry/images/ubuntu:18.04
docker network create test1
docker network create test2
docker network create test3
docker network create test4
docker network create test5
docker network create test6
docker run -itd --net test1 --name testu1 test.registry/images/ubuntu:16.04
docker run -itd --net test2 --name testu2 test.registry/images/ubuntu:16.04
docker run -itd --net test3 --name testu3 test.registry/images/ubuntu:18.04
docker run -itd --net test4 --name testu4 test.registry/images/ubuntu:18.04
docker run -itd --net test5 --name testu5 test.registry/images/ubuntu:18.04
docker run -itd --net test6 --name testu6 test.registry/images/ubuntu:18.04
docker network connect test2 testu1
docker network connect test3 testu1
docker network connect test4 testu1
docker network connect test5 testu1
docker network connect test6 testu1works well !
./test.sh
16.04: Pulling from images/ubuntu
50513f4bae65: Pull complete
854cd4146cdf: Pull complete
3aa6df3c928d: Pull complete
3bcf156a92e8: Pull complete
10de78edcdec: Pull complete
1a7902332604: Pull complete
91da1c491187: Pull complete
f6fe36c00817: Pull complete
37141a578bef: Pull complete
Digest: sha256:44f8fe6d233cb01205aa2d924a43fc6ebf9fc13d6d95748e092abb504148976d
Status: Downloaded newer image for test.registry/images/ubuntu:16.04
18.04: Pulling from images/ubuntu
09888d3b717d: Pull complete
197bf3249b88: Pull complete
e67b16720f7d: Pull complete
a8357788d59a: Pull complete
4c7f1cbe991a: Pull complete
bcafe3c5b913: Pull complete
6aa96a742a7e: Pull complete
cd83ffe921b9: Pull complete
Digest: sha256:7a2f553a2068d0c8ae95876bf22ff2594fda5a94386f54d686717112fed5c92c
Status: Downloaded newer image for test.registry/images/ubuntu:18.04
d50183924fe5b9421363e8bd77d55fb62fe603adf26a0347b8e2523917ec508f
7c7ab49e2d7d04ebd1fb3f82b6a7222cbea715e2ad3cee355727aef233631a3e
7f754ab3abac4edd015597b3d652d38a287c98105975f69de5bcffd9631cf22d
adf716e65a2ee53dac41dbee1c6eed6eec48477ad3567d6b3ae8b842a9dac107
ec1b4b1fa2bf1cd57b85a06ad2b9050cd21aa68a13505368adea14cd1e80e8ec
4191d6ee8c58fc6750affc6258d8bf30215c6de5e2d08ad4ae85a27bc9d6af1d
a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad
ba5cbee4c0e15f8eff79c97ad40fef8b77599e46a26d230d02e89aa18a28a410
f8d30f156a0a617451c2be18c7778a890617189e41b7dd7eb55172fd85223126
03ac095529891c067dd779ba31aed7e26104d118670857a7c95fc87c1318f96a
d420602a07b2a3f833616127ef99590417fb08d873f666bdedc022e6a0e36af2
02f778dd57feb0f6f55548ab33652eb76cb0e0f1bd548660f88ad942a8c966d3docker inspect testu1
[
{
"Id": "a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad",
"Created": "2018-10-04T08:01:07.019399113Z",
"Path": "/bin/bash",
"Args": [],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 5918,
"ExitCode": 0,
"Error": "",
"StartedAt": "2018-10-04T08:01:09.127358791Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:c32058bebc576223f293a7f02e55b9752bb1045edcd3e01612689ad08aa5acd9",
"ResolvConfPath": "/var/lib/docker/containers/a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad/hostname",
"HostsPath": "/var/lib/docker/containers/a9ab1e0a948658fcc210b5db572645eb8a38d9f69d10be745183c28a729f4bad/hosts",
"LogPath": "",
"Name": "/testu1",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "syslog",
"Config": {
"syslog-address": "udp://rns-sv-exploit-001.admin:514",
"tag": "{{.Name}}"
}
},
"NetworkMode": "test1",
"PortBindings": {},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "shareable",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "kata-runtime",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/14f570f6f8d3ace4307f1d9713c2ed474120ce69d259a5d3edd705c0164c4f18-init/diff:/var/lib/docker/overlay2/f0e1ddd045b79425da24fc000f12684e4d90397f1f57b5edf18d4c0fe4f86d6f/diff:/var/lib/docker/overlay2/6acf8d1674a94f2a9dafb9c417d33e9358922092198b84e0bc533984f14ddfbe/diff:/var/lib/docker/overlay2/c13b51b30f06f737308371141252b2223b15682e99c6bfb2970f91dab536d32c/diff:/var/lib/docker/overlay2/c52e355aa61eb36d271d4b4bca0cb3f8ad628e5eb3502b65ccd43892b2b5ca2e/diff:/var/lib/docker/overlay2/a5118628f9f643a10db743d454f360784f0c8fdbaed00553532daa0d11108674/diff:/var/lib/docker/overlay2/a8a0ef248466ed04d9ff93914df4502057d4e96df4092aee1e496b3e4a68e7e7/diff:/var/lib/docker/overlay2/74de80d7090ca44f286d7b900a7cbb75876479b76ce4d56dbaeb6cea6d2985e9/diff:/var/lib/docker/overlay2/4a22b5026632d77f15b1f446730d18e405eadb3d131a0752d81a7deaa0f9316d/diff:/var/lib/docker/overlay2/82d18854e6a535ecf63cade51ff697c7ef06b4c71f48680c12c11413ea91d3b5/diff",
"MergedDir": "/var/lib/docker/overlay2/14f570f6f8d3ace4307f1d9713c2ed474120ce69d259a5d3edd705c0164c4f18/merged",
"UpperDir": "/var/lib/docker/overlay2/14f570f6f8d3ace4307f1d9713c2ed474120ce69d259a5d3edd705c0164c4f18/diff",
"WorkDir": "/var/lib/docker/overlay2/14f570f6f8d3ace4307f1d9713c2ed474120ce69d259a5d3edd705c0164c4f18/work"
},
"Name": "overlay2"
},
"Mounts": [],
"Config": {
"Hostname": "a9ab1e0a9486",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": true,
"OpenStdin": true,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"DEBIAN_FRONTEND=noninteractive"
],
"Cmd": [
"/bin/bash"
],
"ArgsEscaped": true,
"Image": "test.registry/images/ubuntu:16.04",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {
"build_date.ubuntu": "2018-07-26T08:51:08+00:00",
"commit.ubuntu": "5857e7b5a75204a411678d8434a1f7e59bd553f9",
"version.ubuntu": "20170915"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "608b8228387d4be30026671ed833c56e215ba2da95303ce7be17a32c563d458d",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/608b8228387d",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"test1": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"a9ab1e0a9486"
],
"NetworkID": "d50183924fe5b9421363e8bd77d55fb62fe603adf26a0347b8e2523917ec508f",
"EndpointID": "cfbca2d7d522508d1c6e4d335b065323c1d464d2c9e047352fb9e9e5cd6fb140",
"Gateway": "172.18.0.1",
"IPAddress": "172.18.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:12:00:02",
"DriverOpts": null
},
"test2": {
"IPAMConfig": {},
"Links": null,
"Aliases": [
"a9ab1e0a9486"
],
"NetworkID": "7c7ab49e2d7d04ebd1fb3f82b6a7222cbea715e2ad3cee355727aef233631a3e",
"EndpointID": "0f2a4c40447135e2cd701c9759f741182d41c8222a0460f057552e35ae02efe8",
"Gateway": "172.19.0.1",
"IPAddress": "172.19.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:13:00:03",
"DriverOpts": null
},
"test3": {
"IPAMConfig": {},
"Links": null,
"Aliases": [
"a9ab1e0a9486"
],
"NetworkID": "7f754ab3abac4edd015597b3d652d38a287c98105975f69de5bcffd9631cf22d",
"EndpointID": "b21bfb0e252502c193584fa59ab0425ba673d7a5d77f3e1de6d870c537fa44f9",
"Gateway": "172.20.0.1",
"IPAddress": "172.20.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:14:00:03",
"DriverOpts": null
},
"test4": {
"IPAMConfig": {},
"Links": null,
"Aliases": [
"a9ab1e0a9486"
],
"NetworkID": "adf716e65a2ee53dac41dbee1c6eed6eec48477ad3567d6b3ae8b842a9dac107",
"EndpointID": "0f24452f8d6038391110553bcaaff74698071d3a94c50420d6e2ca89b297d36e",
"Gateway": "172.21.0.1",
"IPAddress": "172.21.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:15:00:03",
"DriverOpts": null
},
"test5": {
"IPAMConfig": {},
"Links": null,
"Aliases": [
"a9ab1e0a9486"
],
"NetworkID": "ec1b4b1fa2bf1cd57b85a06ad2b9050cd21aa68a13505368adea14cd1e80e8ec",
"EndpointID": "48d5870d26bfdec9628bc6eb90a7e5ee242103c1b7cc0de5673ca39b249d3c26",
"Gateway": "172.22.0.1",
"IPAddress": "172.22.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:16:00:03",
"DriverOpts": null
},
"test6": {
"IPAMConfig": {},
"Links": null,
"Aliases": [
"a9ab1e0a9486"
],
"NetworkID": "4191d6ee8c58fc6750affc6258d8bf30215c6de5e2d08ad4ae85a27bc9d6af1d",
"EndpointID": "078c6f9a2a40d66d45ca160e456635640d25fd4ac0f90bf2326657e05d171acf",
"Gateway": "172.23.0.1",
"IPAddress": "172.23.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:17:00:03",
"DriverOpts": null
}
}
}
}
]And FI same issue with: https://github.com/docker/compose/releases/tag/1.23.0-rc1
Let me know if I can help you in any way
fredbcode
commented
6 years ago Latest dockerd build : docker version 18.06.1-ce, build e68fc7a
I have this message in syslog only when the bug occurs:
Oct 4 08:59:38 TEST-001 dockerd[28865]: time="2018-10-04T08:59:38.993840460Z" level=error msg="1337c9b768f7bfaff38e10ae54156ddedd1e064e8a71e69eeda9b91bac9beb2d cleanup: failed to delete container from containerd: no such container"
Oct 4 08:59:38 TEST-001 dockerd[28865]: time="2018-10-04T08:59:38.993900278Z" level=error msg="Handler for POST /v1.24/containers/1337c9b768f7bfaff38e10ae54156ddedd1e064e8a71e69eeda9b91bac9beb2d/start returned error: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:ac:1b:00:02 broadcast|multicast 4098 0 0
Docker in debug mode
DEBU[2018-10-04T09:15:16.646247764Z] Released address PoolID:LocalDefault/192.168.80.0/20, Address:192.168.80.3 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.80.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xd0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:4
DEBU[2018-10-04T09:15:16.646674155Z] Revoking external connectivity on endpoint app_fred_1_b9a69d260e1a (4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc)
DEBU[2018-10-04T09:15:16.648281440Z] DeleteConntrackEntries purged ipv4:0, ipv6:0
DEBU[2018-10-04T09:15:16.706595927Z] Programming external connectivity on endpoint app_fred_1_b9a69d260e1a (b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8)
DEBU[2018-10-04T09:15:16.707361953Z] 4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc (fdf2892).deleteSvcRecords(app_fred_1_b9a69d260e1a, 192.168.96.3, <nil>, true) updateSvcRecord sid:4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc
DEBU[2018-10-04T09:15:16.707450537Z] 4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc (fdf2892).deleteSvcRecords(082f8e75b4df, 192.168.96.3, <nil>, false) updateSvcRecord sid:4400c5f585c7**3722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc
DEBU[2018-10-04T09:15:16.707475549Z] 4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc (fdf2892).deleteSvcRecords(fred, 192.168.96.3, <nil>, false) updateSvcRecord sid:4400c5f585c73722f9fb90294d2be55bc61100561e80aaf30929d8a6d76344fc
DEBU[2018-10-04T09:15:16.790172433Z] Releasing addresses for endpoint app_fred_1_b9a69d260e1a's interface on network app_test2
DEBU[2018-10-04T09:15:16.790205061Z] ReleaseAddress(LocalDefault/192.168.96.0/20, 192.168.96.3)
DEBU[2018-10-04T09:15:16.790237425Z] Released address PoolID:LocalDefault/192.168.96.0/20, Address:192.168.96.3 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.96.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xd0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:4
DEBU[2018-10-04T09:15:16.790621371Z] Revoking external connectivity on endpoint app_fred_1_b9a69d260e1a (b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8)
DEBU[2018-10-04T09:15:16.792236669Z] DeleteConntrackEntries purged ipv4:0, ipv6:0
DEBU[2018-10-04T09:15:16.830938459Z] b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8 (f70f6b5).deleteSvcRecords(app_fred_1_b9a69d260e1a, 192.168.112.3, <nil>, true) updateSvcRecord sid:b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8
DEBU[2018-10-04T09:15:16.830990597Z] b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8 (f70f6b5).deleteSvcRecords(082f8e75b4df, 192.168.112.3, <nil>, false) updateSvcRecord sid:b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8
DEBU[2018-10-04T09:15:16.831021853Z] b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8 (f70f6b5).deleteSvcRecords(fred, 192.168.112.3, <nil>, false) updateSvcRecord sid:b2cf2947515e241812830d0b3247ec5c529be3ccbc6284fac6145bdcd829fff8
DEBU[2018-10-04T09:15:16.902267594Z] Releasing addresses for endpoint app_fred_1_b9a69d260e1a's interface on network app_test3
DEBU[2018-10-04T09:15:16.902290954Z] ReleaseAddress(LocalDefault/192.168.112.0/20, 192.168.112.3)
DEBU[2018-10-04T09:15:16.902317283Z] Released address PoolID:LocalDefault/192.168.112.0/20, Address:192.168.112.3 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.112.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xd0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:4
ERRO[2018-10-04T09:15:16.981674628Z] 082f8e75b4df064d8d4d798b5c518e7c81314bdeb11441ce19b83295dcc09498 cleanup: failed to delete container from containerd: no such container
ERRO[2018-10-04T09:15:16.981733245Z] Handler for POST /v1.24/containers/082f8e75b4df064d8d4d798b5c518e7c81314bdeb11441ce19b83295dcc09498/start returned error: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth3 for interface &{{2 1500 1000 eth0 02:42:c0:a8:70:03 broadcast|multicast 4098 0 0 <nil> 0xc4201dc5d4 0 0xc4200172c0 ether <nil> down 0 0 0 []}}: file exists: unknown Same without error (with only two networks)
DEBU[2018-10-04T09:36:31.709013698Z] 5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34 (228ea3b).deleteSvcRecords(app_fred3_1, 192.168.224.2, <nil>, true) updateSvcRecord sid:5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34
DEBU[2018-10-04T09:36:31.709067150Z] 5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34 (228ea3b).deleteSvcRecords(9c23b9033922, 192.168.224.2, <nil>, false) updateSvcRecord sid:5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34
DEBU[2018-10-04T09:36:31.709088788Z] 5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34 (228ea3b).deleteSvcRecords(fred3, 192.168.224.2, <nil>, false) updateSvcRecord sid:5b51a36b88dcdf4dad7f910751428be1f872b0b8b69493b5bdf91f585f769e34
DEBU[2018-10-04T09:36:31.788133198Z] Releasing addresses for endpoint app_fred3_1's interface on network app_test3
DEBU[2018-10-04T09:36:31.788181561Z] ReleaseAddress(LocalDefault/192.168.224.0/20, 192.168.224.2)
DEBU[2018-10-04T09:36:31.788223099Z] Released address PoolID:LocalDefault/192.168.224.0/20, Address:192.168.224.2 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.224.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xe0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:3
DEBU[0227] event published ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete
DEBU[0228] event published ns=moby topic="/tasks/create" type=containerd.events.TaskCreate
DEBU[2018-10-04T09:36:32.144238019Z] event module=libcontainerd namespace=moby topic=/tasks/create
DEBU[0228] event published ns=moby topic="/tasks/start" type=containerd.events.TaskStart
DEBU[2018-10-04T09:36:32.201663950Z] event module=libcontainerd namespace=moby topic=/tasks/start
DEBU[2018-10-04T09:36:32.247504205Z] Calling POST /v1.24/containers/8a30eeae9bb25bf39e3f4619470c79476d83da04afd302361d16ef8f659f8352/wait
DEBU[2018-10-04T09:36:32.248237596Z] Calling POST /v1.24/containers/385d977c0a6baf95999e6f4da95678a05b83bb227f1596f0de226d378854bd5f/wait
DEBU[2018-10-04T09:36:32.251847915Z] Calling POST /v1.24/containers/9c23b90339228c3b6315405fa16fb4f12aae83dbcbb8ce2a0c6eaee058d92b17/wait
DEBU[2018-10-04T09:36:32.254077612Z] Calling POST /v1.24/containers/4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437/wait
DEBU[2018-10-04T09:36:32.254794565Z] Calling GET /v1.24/events?filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dapp%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D
ERRO[0005] failed to kill init's children error="OCI runtime killall failed: Container not ready, running or paused, impossible to signal the container" id=4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437 namespace=moby path="/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437" pid=11055
DEBU[0228] event published ns=moby topic="/tasks/exit" type=containerd.events.TaskExit
DEBU[2018-10-04T09:36:32.417929268Z] event module=libcontainerd namespace=moby topic=/tasks/exit
INFO[0228] shim reaped id=4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437
DEBU[0228] event published ns=moby topic="/tasks/delete" type=containerd.events.TaskDelete
DEBU[2018-10-04T09:36:32.565223072Z] event module=libcontainerd namespace=moby topic=/tasks/delete
INFO[2018-10-04T09:36:32.565267988Z] ignoring event module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
DEBU[2018-10-04T09:36:32.565750086Z] Revoking external connectivity on endpoint app_fred2_1 (2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede)
DEBU[2018-10-04T09:36:32.567607558Z] Calling GET /v1.24/containers/4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437/json
DEBU[2018-10-04T09:36:32.568601235Z] DeleteConntrackEntries purged ipv4:0, ipv6:0
DEBU[2018-10-04T09:36:32.633011532Z] 2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede (310367f).deleteSvcRecords(app_fred2_1, 192.168.240.2, <nil>, true) updateSvcRecord sid:2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede
DEBU[2018-10-04T09:36:32.633080246Z] 2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede (310367f).deleteSvcRecords(4ce4a9495b1f, 192.168.240.2, <nil>, false) updateSvcRecord sid:2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede
DEBU[2018-10-04T09:36:32.633102798Z] 2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede (310367f).deleteSvcRecords(fred2, 192.168.240.2, <nil>, false) updateSvcRecord sid:2648aad3e87cee31b2f37915dc76ee17741a073c43a4e891545defe038475ede
DEBU[2018-10-04T09:36:32.716011196Z] Releasing addresses for endpoint app_fred2_1's interface on network app_test2
DEBU[2018-10-04T09:36:32.716044199Z] ReleaseAddress(LocalDefault/192.168.240.0/20, 192.168.240.2)
DEBU[2018-10-04T09:36:32.716082403Z] Released address PoolID:LocalDefault/192.168.240.0/20, Address:192.168.240.2 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.240.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xe0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:4
DEBU[0228] event published ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete
DEBU[2018-10-04T09:36:33.000437658Z] Client context cancelled, stop sending events
It works, although I have this in debug message (but nothing in prompt after docker-compose up)
ERRO[0005] failed to kill init's children error="OCI runtime killall failed: Container not ready, running or paused, impossible to signal the container" id=4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437 namespace=moby path="/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/4ce4a9495b1fb0d7aedd694107457bae2a10a5b017770f7db46020893611b437"
Gabasjob
commented
6 years ago Hello there. Any update on this bug ?
Is there any workaround to use kata-runtime with docker-compose ?
alebourdoulous
commented
6 years ago Hello, We look forward to hearing from you about this problem and hope to have a solution soon. thank you
grahamwhaley
commented
6 years ago Ping @amshinde @mcastelino - any thoughts, input or requests for more tests/info here? I think @sboeuf had a pertinent question back up at https://github.com/kata-containers/runtime/issues/785#issuecomment-425330335 :
@amshinde @mcastelino correct me if I'm wrong but I think docker-compose is relying on dockerd to setup things inside the network namespace of the container usually. And IIRC, this does not work well with our VM setup as we don't/can't propagate the same kind of setup inside the VM.
fredbcode
commented
6 years ago Le 10/10/2018 à 10:42, Frank Cao a écrit :
We have integrated with k8s. For docker-compose, I'm afraid most of use haven't used it.
You can easily reproduce with a basic docker-compose here:
https://github.com/kata-containers/runtime/issues/785#issuecomment-425382208
Gabasjob
commented
6 years ago Hello there, here is some steps to reproduce the bug without docker-compose :
docker network create net1
docker network create net2
docker network create net3
docker create --name bb --network net1 busybox
docker network connect net2 bb
docker network connect net3 bb
docker start bboutput result :
Error response from daemon: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth1 for interface &{{2 1500 1000 eth0 02:42:ac:14:00:02 broadcast|multicast 4098 0 0 <nil> 0xc4201dc5d4 0 0xc4200172c0 ether <nil> down 0 0 0 []}}: file exists: unknown
Error: failed to start containers: bb
amshinde
commented
6 years ago @sboeuf @grahamwhaley We do have a current limitation with docker-compose that causes issues in dns name resolution due to the use of dockerd. But the above issue looks unrelated.
I'll take a look to see whats going on.
amshinde
commented
6 years ago @fredbcode @Gabasjob This patch to the runtime should solve your issue for now: https://github.com/amshinde/kata-runtime/commit/d28b5b42b0d7f958182265c2a983da073d8e50dd
Can you give that a shot. I am planning to open a PR with that fix.
fredbcode
commented
6 years ago Great, can you share with us a binary please ?
jodh-intel
commented
6 years ago @fredbcode - we don't really have the facility to do that. But all you'd need to do is:
$ go get -u github.com/kata-containers/runtime
$ curl -sL https://github.com/amshinde/kata- runtime/commit/d28b5b42b0d7f958182265c2a983da073d8e50dd.patch | git apply
$ make && sudo -E PATH=$PATH make install/usr/local/bin/kata-runtime that you just created rather than /usr/bin/kata-runtime.fredbcode
commented
6 years ago I will soon, thanks @Gabasjob little "fix" for compilation
go get -u github.com/kata-containers/runtime/virtcontainers
cd yourpath/runtime
curl -sL https://github.com/amshinde/kata-runtime/commit/d28b5b42b0d7f958182265c2a983da073d8e50dd.patch | git apply
make && sudo -E PATH=$PATH make install@jodh-intel Works well !
grahamwhaley
commented
6 years ago Excellent! - thanks @Gabasjob , and kudos @amshinde Note @Gabasjob - you will still have the dns resolution issue that @amshinde referenced above - which is listed (under swam) at: https://github.com/kata-containers/documentation/blob/master/Limitations.md#docker-swarm-support which points to some details at: https://github.com/kata-containers/runtime/issues/175
fredbcode
commented
6 years ago I also confirm, great !
jodh-intel
commented
6 years ago Great - sounds like #814 will dtrt then ;)
Gabasjob
commented
6 years ago @grahamwhaley dns issue still present I'm affraid : docker-compose.yml :
version: '2'
services:
bb1:
image: busybox
networks:
- net1
command: "ping bb2"
bb2:
image: busybox
networks:
- net1
command: "sleep 30"
networks:
net1:output without kata-runtime :
$ docker-compose up
Creating network "bugkata_net1" with the default driver
Recreating bugkata_bb2_1 ...
Recreating bugkata_bb1_1 ...
Recreating bugkata_bb1_1
Recreating bugkata_bb2_1 ... done
Attaching to bugkata_bb1_1, bugkata_bb2_1
bb1_1 | PING bb2 (172.21.0.3): 56 data bytes
bb1_1 | 64 bytes from 172.21.0.3: seq=0 ttl=64 time=1004.177 ms
bb1_1 | 64 bytes from 172.21.0.3: seq=1 ttl=64 time=4.111 ms
bb1_1 | 64 bytes from 172.21.0.3: seq=2 ttl=64 time=0.079 ms
bb1_1 | 64 bytes from 172.21.0.3: seq=3 ttl=64 time=0.065 msoutput with kata-runtime :
$ docker-compose up
Creating network "bugkata_net1" with the default driver
Creating bugkata_bb2_1 ...
Creating bugkata_bb1_1 ...
Creating bugkata_bb1_1
Creating bugkata_bb2_1 ... done
Attaching to bugkata_bb1_1, bugkata_bb2_1
bb1_1 | ping: bad address 'bb2'
bugkata_bb1_1 exited with code 1@Gabasjob yep - that is a known limitation. /cc @amshinde @mcastelino - do we have any known workarounds, or long term plans on that?
amshinde
commented
6 years ago @grahamwhaley Unfortunately there are no workarounds for this due to the way name resolution is handled in docker for custom networks. I'll take a look at this with @mcastelino to come up with a plan.
Description of problem
docker-compose up stop with an error message (works without kataruntime)
Expected result
All dockers image runs
Actual result
ERROR: for app_apache-rp_1 Cannot start service apache-rp: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth2 for interface &{{3 1500 1000 eth1 02:42:ac:14:00:03 broadcast|multicast 4098 0 0 0xc4202940e0 0 0xc420017760 ether down 0 0 0 []}}: file exists: unknown
ERROR: for apache-rp Cannot start service apache-rp: OCI runtime create failed: rpc error: code = Internal desc = Could not set name eth2 for interface &{{3 1500 1000 eth1 02:42:ac:14:00:03 broadcast|multicast 4098 0 0 0xc4202940e0 0 0xc420017760 ether down 0 0 0 []}}: file exists: unknown
ERROR: Encountered errors while bringing up the project.
Meta details
Running
kata-collect-data.shversion1.3.0-rc1 (commit 22aedc4)at2018-09-25.12:56:13.961551045+0000.Runtime is
/usr/bin/kata-runtime.kata-envOutput of "
/usr/bin/kata-runtime kata-env":Runtime config files
Runtime default config files
Runtime config file contents
Config file
/etc/kata-containers/configuration.tomlnot found Output of "cat "/usr/share/defaults/kata-containers/configuration.toml"":Image details
Initrd details
No initrd
Logfiles
Runtime logs
Recent runtime problems found in system journal:
Proxy logs
Recent proxy problems found in system journal:
Shim logs
Recent shim problems found in system journal:
Container manager details
Have
dockerDocker
Output of "
docker version":Output of "
docker info":Output of "
systemctl show docker":No
kubectlPackages
Have
dpkgOutput of "dpkg -l|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-containers-image|linux-container|qemu-)"":No
rpmPerhaps related with #702
proxy.log runtime.log shim.log