search

srl-labs / containerlab

container-based networking labs
https://containerlab.dev
BSD 3-Clause "New" or "Revised" License
1.54k stars 263 forks source link

Issues creating containers & links with basic topology #370

Closed qlyoung closed 3 years ago

qlyoung commented 3 years ago

This project is extremely cool. Thank you for building it.

I took it for a spin and hit a few issues I'd appreciate some help with.

Environment: Debian 10 (buster) Kernel: Linux buster 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux Arch: x64 Docker version:

root@buster /h/v/lab# docker version
Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.11.6
 Git commit:        4c52b90
 Built:             Sun, 21 Feb 2021 18:18:35 +0100
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.6
  Git commit:       4c52b90
  Built:            Sun Feb 21 17:18:35 2021
  OS/Arch:          linux/amd64
  Experimental:     false

containerlab version: 

version: 0.12.0
 commit: abadb94
   date: 2021-03-29T07:42:16Z
 source: https://github.com/srl-labs/containerlab

This is a stock Debian 10 VM running under Virtualbox.

Description

I wrote the following topology file:

name: pim-basic
topology:
        nodes:
                rp:
                        kind: linux
                        image: frrouting/frr:v7.5.0
                fhr:
                        kind: linux
                        image: frrouting/frr:v7.5.0
                lhr:
                        kind: linux
                        image: frrouting/frr:v7.5.0
                source:
                        kind: linux
                        image: alpine:latest
                consumer:
                        kind: linux
                        image: alpine:latest
        links:
                - endpoints: ["fhr:eth0", "rp:eth0"]
                - endpoints: ["lhr:eth0", "rp:eth1"]

I then run containerlab deploy --topo lab.yaml.

Here's the output:

INFO[0000] Parsing & checking topology file: lab.yaml
INFO[0000] Creating lab directory: /home/vagrant/lab/clab-pim-basic
INFO[0000] Creating docker network: Name='clab', IPv4Subnet='172.20.20.0/24', IPv6Subnet='2001:172:20:20::/80', MTU=
'1500'
INFO[0000] Creating container: source
INFO[0000] Creating container: consumer
INFO[0000] Creating container: rp
INFO[0000] Creating container: fhr
INFO[0000] Creating container: lhr
ERRO[0001] failed to create node source: Error response from daemon: OCI runtime create failed: container_linux.go:3
44: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:407: running p
restart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\"2021-04-08T06:18:13Z\\\\
\\\" level=fatal msg=\\\\\\\"failed to add interface veth134fd21 to sandbox: error setting interface \\\\\\\\\\\\\\\
"veth134fd21\\\\\\\\\\\\\\\" IPv6 to 2001:172:20:20::3/80: failed to enable ipv6: reexec to set IPv6 failed: exit st
atus 2\\\\\\\"\\\\n\\\"\"": unknown
INFO[0001] Creating virtual wire: fhr:eth0 <--> rp:eth0
INFO[0001] Creating virtual wire: lhr:eth0 <--> rp:eth1
ERRO[0001] failed to rename link: file exists
ERRO[0001] failed to rename link: file exists
ERRO[0002] failed to run postdeploy task for node source: failed to Statfs "": no such file or directory
INFO[0002] Writing /etc/hosts file
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+
| # |          Name           | Container ID |        Image         | Kind  | Group |  State  |  IPv4 Address  |     IPv6 Address     |
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+
| 1 | clab-pim-basic-consumer | 10cfdd78e2cd | alpine:latest        | linux |       | running | 172.20.20.4/24 | 2001:172:20:20::4/80 |
| 2 | clab-pim-basic-fhr      | 88120926affe | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.5/24 | 2001:172:20:20::5/80 |
| 3 | clab-pim-basic-lhr      | 01c8b8cdd1b3 | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.6/24 | 2001:172:20:20::6/80 |
| 4 | clab-pim-basic-rp       | 9069fb4f1dc1 | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.2/24 | 2001:172:20:20::2/80 |
| 5 | clab-pim-basic-source   | f935e7829062 | alpine:latest        | linux |       | created | 172.20.20.3/24 | 2001:172:20:20::3/80 |
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+

The issues I noticed:

Am I doing something wrong? Let me know if you need any additional info.

hellt commented 3 years ago

Hi @qlyoung thanks for checking clab out

Before going into the errors themselves, I wanted to point out some issues I see with the endpoints configuration:

links:
    - endpoints: ["fhr:eth0", "rp:eth0"]
    - endpoints: ["lhr:eth0", "rp:eth1"]

You shouldn't have eth0 interfaces in the endpoints sections. eth0 interface is automatically created by docker to connect a container to a management network.

image

If your intention was to interconnect fhr with rp over their data interfaces, then you would need to do the following

links:
  - endpoints: ["fhr:eth1", "rp:eth1"]

if, on the other hand, you wanted all the nodes to share the management network, then no links section is even needed.

Let me know what was the topology you wanted to build and we can sort it out :)

qlyoung commented 3 years ago

Ok, I'll switch to eth1 and see if the issues go away.

qlyoung commented 3 years ago

New topology:

root@buster /h/v/lab# cat lab.yaml
name: pim-basic
topology:
        nodes:
                rp:
                        kind: linux
                        image: frrouting/frr:v7.5.0
                fhr:
                        kind: linux
                        image: frrouting/frr:v7.5.0
                lhr:
                        kind: linux
                        image: frrouting/frr:v7.5.0
                source:
                        kind: linux
                        image: alpine:latest
                consumer:
                        kind: linux
                        image: alpine:latest
        links:
                - endpoints: ["fhr:eth1", "rp:eth1"]
                - endpoints: ["lhr:eth1", "rp:eth2"]

This summarizes the topology I want for now, just lhr -- rp -- fhr. However I still expect the other containers to spin up.

New output:

root@buster /h/v/lab# containerlab deploy --topo ./lab.yaml
INFO[0000] Parsing & checking topology file: lab.yaml
INFO[0000] Creating lab directory: /home/vagrant/lab/clab-pim-basic
INFO[0000] Creating docker network: Name='clab', IPv4Subnet='172.20.20.0/24', IPv6Subnet='2001:172:20:20::/80', MTU='1500'
INFO[0000] Creating container: source
INFO[0000] Creating container: lhr
INFO[0000] Creating container: consumer
INFO[0000] Creating container: rp
INFO[0000] Creating container: fhr
ERRO[0001] failed to create node source: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:407: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\"2021-04-08T15:49:56Z\\\\\\\" level=fatal msg=\\\\\\\"failed to add interface vethf4b055f to sandbox: error setting interface \\\\\\\\\\\\\\\"vethf4b055f\\\\\\\\\\\\\\\" IPv6 to 2001:172:20:20::2/80: failed to enable ipv6: reexec to set IPv6 failed: exit status 2\\\\\\\"\\\\n\\\"\"": unknown
INFO[0001] Creating virtual wire: fhr:eth1 <--> rp:eth1
INFO[0001] Creating virtual wire: lhr:eth1 <--> rp:eth2
ERRO[0001] failed to run postdeploy task for node source: failed to Statfs "": no such file or directory
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+
| # |          Name           | Container ID |        Image         | Kind  | Group |  State  |  IPv4 Address  |     IPv6 Address     |
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+
| 1 | clab-pim-basic-consumer | 8dd29f657b6e | alpine:latest        | linux |       | running | 172.20.20.3/24 | 2001:172:20:20::3/80 |
| 2 | clab-pim-basic-fhr      | 671f5ca9a60c | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.4/24 | 2001:172:20:20::4/80 |
| 3 | clab-pim-basic-lhr      | 2f066810e60d | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.5/24 | 2001:172:20:20::5/80 |
| 4 | clab-pim-basic-rp       | e10a90e91d27 | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.6/24 | 2001:172:20:20::6/80 |
| 5 | clab-pim-basic-source   | 7131aab38f3c | alpine:latest        | linux |       | created | 172.20.20.2/24 | 2001:172:20:20::2/80 |
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+

root@buster /h/v/lab# docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
e10a90e91d27        frrouting/frr:v7.5.0   "/sbin/tini -- /usr/…"   8 seconds ago       Up 6 seconds                            clab-pim-basic-rp
2f066810e60d        frrouting/frr:v7.5.0   "/sbin/tini -- /usr/…"   8 seconds ago       Up 6 seconds                            clab-pim-basic-lhr
8dd29f657b6e        alpine:latest          "/bin/sh"                8 seconds ago       Up 6 seconds                            clab-pim-basic-consumer
671f5ca9a60c        frrouting/frr:v7.5.0   "/sbin/tini -- /usr/…"   8 seconds ago       Up 6 seconds                            clab-pim-basic-fhr

Looks like rp has its interfaces up now:

root@buster /h/v/lab# docker exec -it clab-pim-basic-rp sh
/ # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
69: eth0@if70: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 02:42:ac:14:14:06 brd ff:ff:ff:ff:ff:ff link-netnsid 0
71: eth1@if72: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue state UP mode DEFAULT group default
    link/ether 7a:19:ff:12:06:23 brd ff:ff:ff:ff:ff:ff link-netnsid 1
73: eth2@if74: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue state UP mode DEFAULT group default
    link/ether a2:01:9d:48:79:88 brd ff:ff:ff:ff:ff:ff link-netnsid 2

So now the only issues are

  1. The error output in the deploy
  2. Failure to start one or more of the containres

For 2, I destroyed the lab, restarted Docker, and spun up the lab again. This time two containers are missing, fhr and source:

+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+
| # |          Name           | Container ID |        Image         | Kind  | Group |  State  |  IPv4 Address  |     IPv6 Address     |
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+
| 1 | clab-pim-basic-consumer | 727c2eadb896 | alpine:latest        | linux |       | running | 172.20.20.5/24 | 2001:172:20:20::5/80 |
| 2 | clab-pim-basic-fhr      | aa22f7622aa9 | frrouting/frr:v7.5.0 | linux |       | created | 172.20.20.4/24 | 2001:172:20:20::4/80 |
| 3 | clab-pim-basic-lhr      | b2d92d753cd2 | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.3/24 | 2001:172:20:20::3/80 |
| 4 | clab-pim-basic-rp       | c4c3b0eb5770 | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.2/24 | 2001:172:20:20::2/80 |
| 5 | clab-pim-basic-source   | 17a54d4c97b4 | alpine:latest        | linux |       | created | 172.20.20.6/24 | 2001:172:20:20::6/80 |
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+
root@buster /h/v/lab# docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
b2d92d753cd2        frrouting/frr:v7.5.0   "/sbin/tini -- /usr/…"   6 seconds ago       Up 3 seconds                            clab-pim-basic-lhr
727c2eadb896        alpine:latest          "/bin/sh"                6 seconds ago       Up 3 seconds                            clab-pim-basic-consumer
c4c3b0eb5770        frrouting/frr:v7.5.0   "/sbin/tini -- /usr/…"   6 seconds ago       Up 3 seconds                            clab-pim-basic-rp
qlyoung commented 3 years ago

If it helps you can spin up the same VM with

vagrant init debian/buster64
vagrant up

https://app.vagrantup.com/debian/boxes/buster64

I installed containerlab with

bash -c "$(curl -sL https://get-clab.srlinux.dev)"

and docker with

apt install docker.io
hellt commented 3 years ago

I'll check this lab first on my system and see if the issue appears there

But I can suggest you to try and install docker as it explained in docker docs, it is now docker-ce package that needs to be installed

On Thu, 8 Apr 2021 at 18:43, Quentin Young @.***> wrote:

If it helps you can spin up the same VM with

vagrant init debian/buster64 vagrant up

https://app.vagrantup.com/debian/boxes/buster64

I installed containerlab with

bash -c "$(curl -sL https://get-clab.srlinux.dev)"

and docker with

apt install docker.io

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/srl-labs/containerlab/issues/370#issuecomment-815975634, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLKV5P3UI2VFBV3ELF34B3THXMLRANCNFSM42SG2ACQ .

qlyoung commented 3 years ago

The docker-ce package is the one provided by docker themselves; docker.io is the one provided by distributions. I would expect clab to work with distro docker packages. I'll try using docker-ce and report back.

hellt commented 3 years ago

As you see from the docker version output, docker.io dates back to 2018. We haven't tested such an old version of docker, as such our docs state that we require docker ce 19 at a minimum

While the issue that you see might not be related to outdated docker version, it is still recommended to run a maintained/fresh version of docker which is a docker-ce nowadays

On Thu, 8 Apr 2021 at 19:13, Quentin Young @.***> wrote:

The docker-ce package is the one provided by docker themselves; docker.io is the one provided by distributions. I would expect clab to work with distro docker packages. I'll try using docker-ce and report back.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/srl-labs/containerlab/issues/370#issuecomment-815995507, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLKV5KA6WH6UJ5V37Y34YLTHXP4NANCNFSM42SG2ACQ .

qlyoung commented 3 years ago

I tried with docker-ce on a new VM, and it failed in a different way. One that showed me what the real problem is:

root@buster:/home/vagrant/lab# free -m
              total        used        free      shared  buff/cache   available
Mem:            483         145         185           2         152         322
Swap:             0           0           0

By default this VM image has 512 mb of memory.

I went back to the initial environment, increased memory to 4gb, and the errors disappeared.

However, I feel the error output should have included something about OOM. If that's not feasible it might be prudent to make a note of what OOM failures look like, since it's not really clear (not seeing the usual Killed). I did read the footnote on the installation page indicating that memory usage is dependent on lab size.

Shall I close this or leave it open for tracking that?

hellt commented 3 years ago

Ah, yes, resources... Let's close this and I will open the new one to warn about memory requirements

On Thu, 8 Apr 2021 at 19:27, Quentin Young @.***> wrote:

I tried with docker-ce on a new VM, and it failed in a different way. One that showed me what the real problem is:

@.***:/home/vagrant/lab# free -m total used free shared buff/cache available Mem: 483 145 185 2 152 322 Swap: 0 0 0

By default this VM image has 512 mb of memory.

I went back to the initial environment, increased memory to 4gb, and the errors disappeared.

However, I feel the error output should have included something about OOM. Shall I close this or leave it open for tracking that?

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/srl-labs/containerlab/issues/370#issuecomment-816005050, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLKV5MXFCRAOPFQOVJOBMTTHXRRXANCNFSM42SG2ACQ .