search

metal-stack / mini-lab

a small, virtual setup to locally run the metal-stack
MIT License
57 stars 13 forks source link

Booting mini-lab might hang while setting up the network #118

Closed vknabel closed 9 months ago

vknabel commented 1 year ago

Sometimes make up stops proceeding after setting up the first groups of wires. Though sometimes it proceeds without any issues. I cannot detect any pattern. When I run make down && make only 1 in 5 calls succeed. Waiting several minutes doesn't solve this either.

image (2)

Aborting the process prints ^Cmake: *** [Makefile:80: partition-bake] Error 130.

I already had a quick chat with @Gerrit91 on this. Here is some requested information:

❯ docker ps
CONTAINER ID   IMAGE                                     COMMAND                  CREATED         STATUS         PORTS                                                                                                                                              NAMES
9a951775b316   ghcr.io/metal-stack/mini-lab-vms:latest   "/mini-lab/vms_entry…"   3 minutes ago   Up 3 minutes                                                                                                                                                      vms
3b254f2982cf   grigoriymikh/sandbox:latest               "/usr/local/bin/igni…"   3 minutes ago   Up 3 minutes                                                                                                                                                      ignite-eb8de119eecdaa65
dc03782d709c   kindest/node:v1.24.0                      "/usr/local/bin/entr…"   3 minutes ago   Up 3 minutes   0.0.0.0:4150->4150/tcp, 0.0.0.0:4161->4161/tcp, 0.0.0.0:4443->4443/tcp, 0.0.0.0:6443->6443/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:50051->50051/tcp   metal-control-plane-control-plane
❯ make ssh-leaf01
ssh -o StrictHostKeyChecking=no -o "PubkeyAcceptedKeyTypes +ssh-rsa" -i files/ssh/id_rsa root@leaf01
ssh: Could not resolve hostname leaf01: Temporary failure in name resolution
make: *** [Makefile:142: ssh-leaf01] Error 255
❯ cat /etc/hosts
127.0.0.1   localhost
127.0.1.1   yubihill

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

docker logs ignite-eb8de119eecdaa65

GrigoriyMikhalkin commented 1 year ago

Looks like a containerlab bug. I don't see any entries for started containers in /etc/hosts.

@vknabel Which version of containerlab do you use?

Also, would be helpful if you could update the partition-bake directive in Makefile. Specifically add flags -d --log-level=trace after containerlab deploy. And then post logs here.

vknabel commented 1 year ago

@GrigoriyMikhalkin thanks for the quick response. I use containerlab 0.31.1

❯ containerlab version  

                           _                   _       _     
                 _        (_)                 | |     | |    
 ____ ___  ____ | |_  ____ _ ____   ____  ____| | ____| | _  
/ ___) _ \|  _ \|  _)/ _  | |  _ \ / _  )/ ___) |/ _  | || \ 
( (__| |_|| | | | |_( ( | | | | | ( (/ /| |   | ( ( | | |_) )
\____)___/|_| |_|\___)_||_|_|_| |_|\____)_|   |_|\_||_|____/ 

    version: 0.31.1
     commit: 4fbe732d
       date: 2022-08-15T13:18:12Z
     source: https://github.com/srl-labs/containerlab
 rel. notes: https://containerlab.dev/rn/0.31/#0311

And here are the trace-level logs:

``` ❯ make Obtaining release vector variables... No kind clusters found. Creating cluster "metal-control-plane" ... ✓ Ensuring node image (kindest/node:v1.24.0) 🖼 ✓ Preparing nodes 📦 spoiler ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 Set kubectl context to "kind-metal-control-plane" You can now use your cluster with: kubectl cluster-info --context kind-metal-control-plane --kubeconfig /home/vknabel/dev/mini-lab/.kubeconfig Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂 # docker pull ghcr.io/metal-stack/mini-lab-vms:latest INFO[0000] Parsing & checking topology file: mini-lab.cumulus.yaml INFO[0000] no containers found INFO[0000] Containerlab v0.31.1 started DEBU[0000] envN runtime var value is DEBU[0000] Running runtime.Init with params &{Timeout:2m0s GracefulShutdown:false Debug:false KeepMgmtNet:false} and &{Network: Bridge: IPv4Subnet: IPv4Gw: IPv6Subnet: IPv6Gw: MTU: ExternalAccess:} DEBU[0000] Runtime: Docker DEBU[0000] detected docker network mtu value - 1500 DEBU[0000] initialized a runtime with params &{config:{Timeout:120000000000 GracefulShutdown:false Debug:false KeepMgmtNet:false} Client:0xc000cd8b80 mgmt:0xc000cd8a00} DEBU[0000] template variables: DEBU[0000] topology: name: mini-lab prefix: "" mgmt: network: bridge topology: kinds: cvx: image: networkop/cx:3.7.0 kernel: docker.io/grigoriymikh/kernel:4.1.0 sandbox: grigoriymikh/sandbox:latest binds: - files/ssh/id_rsa.pub:/root/.ssh/authorized_keys linux: image: ${MINI_LAB_VM_IMAGE} nodes: leaf01: kind: cvx leaf02: kind: cvx vms: kind: linux binds: - /dev:/dev - scripts:/mini-lab links: - endpoints: ["leaf01:swp1", "vms:lan0"] - endpoints: ["leaf02:swp1", "vms:lan1"] - endpoints: ["leaf01:swp2", "vms:lan2"] - endpoints: ["leaf02:swp2", "vms:lan3"] - endpoints: ["leaf01:swp3", "vms:lan4"] - endpoints: ["leaf02:swp3", "vms:lan5"] DEBU[0000] method initMgmtNetwork was called mgmt params &{Network:bridge Bridge: IPv4Subnet: IPv4Gw: IPv6Subnet: IPv6Gw: MTU:1500 ExternalAccess:} DEBU[0000] New mgmt params are &{Network:bridge Bridge: IPv4Subnet:172.20.20.0/24 IPv4Gw: IPv6Subnet:2001:172:20:20::/64 IPv6Gw: MTU:1500 ExternalAccess:0xc0004f4f2a} INFO[0000] Parsing & checking topology file: mini-lab.cumulus.yaml DEBU[0000] Runtime: Docker DEBU[0000] node config: &{ShortName:leaf01 LongName:leaf01 Fqdn:leaf01.mini-lab.io LabDir:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf01 Index:0 Group: Kind:cvx StartupConfig: StartupDelay:0 EnforceStartupConfig:false ResStartupConfig: Config: ResConfig: NodeType: Position: License: Image:networkop/cx:3.7.0 Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[] Binds:[] PortBindings:map[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: NSPath: Publish:[] ExtraHosts:[] Labels:map[] Endpoints:[] SANs:[] Sandbox:grigoriymikh/sandbox:latest Kernel:docker.io/grigoriymikh/kernel:4.1.0 Runtime: CPU:0 CPUSet: Memory: HostRequirements:{SSE3:false VirtRequired:false} DeploymentStatus: Extras:} DEBU[0000] node config: &{ShortName:leaf02 LongName:leaf02 Fqdn:leaf02.mini-lab.io LabDir:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf02 Index:1 Group: Kind:cvx StartupConfig: StartupDelay:0 EnforceStartupConfig:false ResStartupConfig: Config: ResConfig: NodeType: Position: License: Image:networkop/cx:3.7.0 Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[] Binds:[] PortBindings:map[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: NSPath: Publish:[] ExtraHosts:[] Labels:map[] Endpoints:[] SANs:[] Sandbox:grigoriymikh/sandbox:latest Kernel:docker.io/grigoriymikh/kernel:4.1.0 Runtime: CPU:0 CPUSet: Memory: HostRequirements:{SSE3:false VirtRequired:false} DeploymentStatus: Extras:} DEBU[0000] node config: &{ShortName:vms LongName:vms Fqdn:vms.mini-lab.io LabDir:/home/vknabel/dev/mini-lab/clab-mini-lab/vms Index:2 Group: Kind:linux StartupConfig: StartupDelay:0 EnforceStartupConfig:false ResStartupConfig: Config: ResConfig: NodeType: Position: License: Image:ghcr.io/metal-stack/mini-lab-vms:latest Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[] Binds:[] PortBindings:map[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: NSPath: Publish:[] ExtraHosts:[] Labels:map[] Endpoints:[] SANs:[] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: HostRequirements:{SSE3:false VirtRequired:false} DeploymentStatus: Extras:} DEBU[0000] lab Conf: &{Name:mini-lab Prefix:0xc000207b60 Mgmt:0xc000cd8a00 Topology:0xc0006b3590} DEBU[0000] Env: CLAB_VERSION_CHECK= DEBU[0000] Filter key: label, filter value: containerlab=mini-lab INFO[0000] Removing /home/vknabel/dev/mini-lab/clab-mini-lab directory... DEBU[0000] Ensuring image networkop/cx:3.7.0 exists, or importing it... DEBU[0000] Found image with UID dc0682667fc8d604 DEBU[0000] Ensuring image grigoriymikh/kernel:4.1.0 exists, or importing it... DEBU[0000] Found image with UID 5c2665d13df41a44 DEBU[0000] Ensuring image grigoriymikh/sandbox:latest exists, or importing it... DEBU[0000] Found image with UID d9d2d278021dbd28 DEBU[0000] Looking up ghcr.io/metal-stack/mini-lab-vms:latest Docker image DEBU[0000] Image ghcr.io/metal-stack/mini-lab-vms:latest present, skip pulling DEBU[0000] Number of vcpu: 12 INFO[0000] Creating lab directory: /home/vknabel/dev/mini-lab/clab-mini-lab DEBU[0000] error while trying to access file /home/vknabel/.ssh/authorized_keys: stat /home/vknabel/.ssh/authorized_keys: no such file or directory DEBU[0000] found public key files ["/home/vknabel/.ssh/id_ed25519.pub" "/home/vknabel/.ssh/id_rsa.pub"] DEBU[0000] Checking if docker network "bridge" exists DEBU[0000] network "bridge" was found. Reusing it... DEBU[0000] Docker network "bridge", bridge name "docker0" DEBU[0000] Disable RPF check on the docker host DEBU[0000] Enable LLDP on the linux bridge docker0 DEBU[0000] Disabling TX checksum offloading for the docker0 bridge interface... DEBU[0000] found iptables forwarding rule targeting the bridge "docker0". Skipping creation of the forwarding rule. DEBU[0000] scheduling nodes with dynamic IPs... DEBU[0000] Worker 3 received node: &{ShortName:leaf01 LongName:leaf01 Fqdn:leaf01.mini-lab.io LabDir:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf01 Index:0 Group: Kind:cvx StartupConfig: StartupDelay:0 EnforceStartupConfig:false ResStartupConfig: Config:0xc0007bc998 ResConfig: NodeType: Position: License: Image:networkop/cx:3.7.0 Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[CLAB_INTFS:3 CLAB_LABEL_CLAB_NODE_GROUP: CLAB_LABEL_CLAB_NODE_KIND:cvx CLAB_LABEL_CLAB_NODE_LAB_DIR:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf01 CLAB_LABEL_CLAB_NODE_NAME:leaf01 CLAB_LABEL_CLAB_NODE_TYPE: CLAB_LABEL_CLAB_TOPO_FILE:/home/vknabel/dev/mini-lab/mini-lab.cumulus.yaml CLAB_LABEL_CONTAINERLAB:mini-lab] Binds:[/home/vknabel/dev/mini-lab/files/ssh/id_rsa.pub:/root/.ssh/authorized_keys] PortBindings:map[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: NSPath: Publish:[] ExtraHosts:[] Labels:map[clab-mgmt-net-bridge:docker0 clab-node-group: clab-node-kind:cvx clab-node-lab-dir:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf01 clab-node-name:leaf01 clab-node-type: clab-topo-file:/home/vknabel/dev/mini-lab/mini-lab.cumulus.yaml containerlab:mini-lab] Endpoints:[{Node:0xc00017a000 EndpointName:swp1 MAC:aa:c1:ab:3b:fd:db} {Node:0xc00017a000 EndpointName:swp2 MAC:aa:c1:ab:66:0a:af} {Node:0xc00017a000 EndpointName:swp3 MAC:aa:c1:ab:18:29:8c}] SANs:[] Sandbox:grigoriymikh/sandbox:latest Kernel:docker.io/grigoriymikh/kernel:4.1.0 Runtime: CPU:0 CPUSet: Memory:768MB HostRequirements:{SSE3:false VirtRequired:false} DeploymentStatus: Extras:} DEBU[0000] Ensuring kernel grigoriymikh/kernel:4.1.0 exists, or importing it... DEBU[0000] Found kernel with UID ab0caedaad64f109 DEBU[0000] Ensuring image networkop/cx:3.7.0 exists, or importing it... DEBU[0000] Found image with UID dc0682667fc8d604 INFO[0001] Networking is handled by "docker-bridge" INFO[0001] Started Firecracker VM "b88e22aa374fd1b5" in a container with ID "7201c81012c1f1aba367797bc326e2bdef644fbb6e769c843ae8108655bac8fa" DEBU[0001] creating links... DEBU[0001] Worker 3 received node: &{ShortName:leaf02 LongName:leaf02 Fqdn:leaf02.mini-lab.io LabDir:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf02 Index:1 Group: Kind:cvx StartupConfig: StartupDelay:0 EnforceStartupConfig:false ResStartupConfig: Config:0xc0007bca10 ResConfig: NodeType: Position: License: Image:networkop/cx:3.7.0 Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[CLAB_INTFS:3 CLAB_LABEL_CLAB_NODE_GROUP: CLAB_LABEL_CLAB_NODE_KIND:cvx CLAB_LABEL_CLAB_NODE_LAB_DIR:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf02 CLAB_LABEL_CLAB_NODE_NAME:leaf02 CLAB_LABEL_CLAB_NODE_TYPE: CLAB_LABEL_CLAB_TOPO_FILE:/home/vknabel/dev/mini-lab/mini-lab.cumulus.yaml CLAB_LABEL_CONTAINERLAB:mini-lab] Binds:[/home/vknabel/dev/mini-lab/files/ssh/id_rsa.pub:/root/.ssh/authorized_keys] PortBindings:map[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: NSPath: Publish:[] ExtraHosts:[] Labels:map[clab-mgmt-net-bridge:docker0 clab-node-group: clab-node-kind:cvx clab-node-lab-dir:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf02 clab-node-name:leaf02 clab-node-type: clab-topo-file:/home/vknabel/dev/mini-lab/mini-lab.cumulus.yaml containerlab:mini-lab] Endpoints:[{Node:0xc00017a700 EndpointName:swp1 MAC:aa:c1:ab:61:62:a8} {Node:0xc00017a700 EndpointName:swp2 MAC:aa:c1:ab:27:61:00} {Node:0xc00017a700 EndpointName:swp3 MAC:aa:c1:ab:20:d7:82}] SANs:[] Sandbox:grigoriymikh/sandbox:latest Kernel:docker.io/grigoriymikh/kernel:4.1.0 Runtime: CPU:0 CPUSet: Memory:768MB HostRequirements:{SSE3:false VirtRequired:false} DeploymentStatus: Extras:} DEBU[0001] Worker 2 terminating... DEBU[0001] Ensuring kernel grigoriymikh/kernel:4.1.0 exists, or importing it... DEBU[0001] Worker 1 terminating... DEBU[0001] Worker 0 received node: &{ShortName:vms LongName:vms Fqdn:vms.mini-lab.io LabDir:/home/vknabel/dev/mini-lab/clab-mini-lab/vms Index:2 Group: Kind:linux StartupConfig: StartupDelay:0 EnforceStartupConfig:false ResStartupConfig: Config:0xc0007bca88 ResConfig: NodeType: Position: License: Image:ghcr.io/metal-stack/mini-lab-vms:latest Sysctls:map[net.ipv6.conf.all.disable_ipv6:0] User: Entrypoint: Cmd: Exec:[] Env:map[CLAB_INTFS:6 CLAB_LABEL_CLAB_NODE_GROUP: CLAB_LABEL_CLAB_NODE_KIND:linux CLAB_LABEL_CLAB_NODE_LAB_DIR:/home/vknabel/dev/mini-lab/clab-mini-lab/vms CLAB_LABEL_CLAB_NODE_NAME:vms CLAB_LABEL_CLAB_NODE_TYPE: CLAB_LABEL_CLAB_TOPO_FILE:/home/vknabel/dev/mini-lab/mini-lab.cumulus.yaml CLAB_LABEL_CONTAINERLAB:mini-lab] Binds:[/dev:/dev /home/vknabel/dev/mini-lab/scripts:/mini-lab] PortBindings:map[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: NSPath: Publish:[] ExtraHosts:[] Labels:map[clab-mgmt-net-bridge:docker0 clab-node-group: clab-node-kind:linux clab-node-lab-dir:/home/vknabel/dev/mini-lab/clab-mini-lab/vms clab-node-name:vms clab-node-type: clab-topo-file:/home/vknabel/dev/mini-lab/mini-lab.cumulus.yaml containerlab:mini-lab] Endpoints:[{Node:0xc00017ae00 EndpointName:lan0 MAC:aa:c1:ab:f6:ca:12} {Node:0xc00017ae00 EndpointName:lan1 MAC:aa:c1:ab:23:e0:b5} {Node:0xc00017ae00 EndpointName:lan2 MAC:aa:c1:ab:11:83:26} {Node:0xc00017ae00 EndpointName:lan3 MAC:aa:c1:ab:38:cd:eb} {Node:0xc00017ae00 EndpointName:lan4 MAC:aa:c1:ab:f8:ac:b3} {Node:0xc00017ae00 EndpointName:lan5 MAC:aa:c1:ab:b5:83:3f}] SANs:[] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: HostRequirements:{SSE3:false VirtRequired:false} DeploymentStatus: Extras:} INFO[0001] Creating container: "vms" DEBU[0001] Found kernel with UID ab0caedaad64f109 DEBU[0001] Ensuring image networkop/cx:3.7.0 exists, or importing it... DEBU[0001] Found image with UID dc0682667fc8d604 DEBU[0001] Container "vms" create response: {ID:c2e5b4df8192478d37ff5bdc75f0c4553f64659e883e7e623cd0ddabae377fd7 Warnings:[]} DEBU[0001] Start container: "vms" DEBU[0002] Container started: "vms" DEBU[0002] Worker 0 terminating... DEBU[0002] Link worker 1 received link: link [leaf01:swp1, vms:lan0] DEBU[0002] Link worker 3 received link: link [leaf01:swp2, vms:lan2] INFO[0002] Creating virtual wire: leaf01:swp1 <--> vms:lan0 INFO[0002] Creating virtual wire: leaf01:swp2 <--> vms:lan2 DEBU[0002] Link worker 5 received link: link [leaf01:swp3, vms:lan4] INFO[0002] Creating virtual wire: leaf01:swp3 <--> vms:lan4 ```
GrigoriyMikhalkin commented 1 year ago

@vknabel Is your /etc/hosts owned by root user? Is user allowed to write to the file? You can check this info by running ll /etc/hosts.

vknabel commented 1 year ago

@GrigoriyMikhalkin my /etc/hosts is owned by root and and my user has no write access:

❯ ll /etc/hosts
-rw-r--r-- 1 root 340 Sep 20 14:11 /etc/hosts

As mentioned above, about 1 of 5 make up succeed. Here is a truncated log of a succeeding call. for comparision

``` ❯ make Obtaining release vector variables... No kind clusters found. Creating cluster "metal-control-plane" ... ✓ Ensuring node image (kindest/node:v1.24.0) 🖼 ✓ Preparing nodes 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 Set kubectl context to "kind-metal-control-plane" You can now use your cluster with: kubectl cluster-info --context kind-metal-control-plane --kubeconfig /home/vknabel/dev/mini-lab/.kubeconfig Have a nice day! 👋 # docker pull ghcr.io/metal-stack/mini-lab-vms:latest INFO[0000] Parsing & checking topology file: mini-lab.cumulus.yaml INFO[0000] no containers found INFO[0000] Containerlab v0.31.1 started DEBU[0000] envN runtime var value is DEBU[0000] Running runtime.Init with params &{Timeout:2m0s GracefulShutdown:false Debug:false KeepMgmtNet:false} and &{Network: Bridge: IPv4Subnet: IPv4Gw: IPv6Subnet: IPv6Gw: MTU: ExternalAccess:} DEBU[0000] Runtime: Docker DEBU[0000] detected docker network mtu value - 1500 DEBU[0000] initialized a runtime with params &{config:{Timeout:120000000000 GracefulShutdown:false Debug:false KeepMgmtNet:false} Client:0xc0008d8c00 mgmt:0xc0008d8a80} DEBU[0000] template variables: DEBU[0000] topology: name: mini-lab prefix: "" mgmt: network: bridge topology: kinds: cvx: image: networkop/cx:3.7.0 kernel: docker.io/grigoriymikh/kernel:4.1.0 sandbox: grigoriymikh/sandbox:latest binds: - files/ssh/id_rsa.pub:/root/.ssh/authorized_keys linux: image: ${MINI_LAB_VM_IMAGE} nodes: leaf01: kind: cvx leaf02: kind: cvx vms: kind: linux binds: - /dev:/dev - scripts:/mini-lab links: - endpoints: ["leaf01:swp1", "vms:lan0"] - endpoints: ["leaf02:swp1", "vms:lan1"] - endpoints: ["leaf01:swp2", "vms:lan2"] - endpoints: ["leaf02:swp2", "vms:lan3"] - endpoints: ["leaf01:swp3", "vms:lan4"] - endpoints: ["leaf02:swp3", "vms:lan5"] DEBU[0000] method initMgmtNetwork was called mgmt params &{Network:bridge Bridge: IPv4Subnet: IPv4Gw: IPv6Subnet: IPv6Gw: MTU:1500 ExternalAccess:} DEBU[0000] New mgmt params are &{Network:bridge Bridge: IPv4Subnet:172.20.20.0/24 IPv4Gw: IPv6Subnet:2001:172:20:20::/64 IPv6Gw: MTU:1500 ExternalAccess:0xc000741b2a} INFO[0000] Parsing & checking topology file: mini-lab.cumulus.yaml DEBU[0000] Runtime: Docker DEBU[0000] node config: &{ShortName:leaf01 LongName:leaf01 Fqdn:leaf01.mini-lab.io LabDir:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf01 Index:0 Group: Kind:cvx StartupConfig: StartupDelay:0 EnforceStartupConfig:false ResStartupConfig: Config: ResConfig: NodeType: Position: License: Image:networkop/cx:3.7.0 Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[] Binds:[] PortBindings:map[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: NSPath: Publish:[] ExtraHosts:[] Labels:map[] Endpoints:[] SANs:[] Sandbox:grigoriymikh/sandbox:latest Kernel:docker.io/grigoriymikh/kernel:4.1.0 Runtime: CPU:0 CPUSet: Memory: HostRequirements:{SSE3:false VirtRequired:false} DeploymentStatus: Extras:} DEBU[0000] node config: &{ShortName:leaf02 LongName:leaf02 Fqdn:leaf02.mini-lab.io LabDir:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf02 Index:1 Group: Kind:cvx StartupConfig: StartupDelay:0 EnforceStartupConfig:false ResStartupConfig: Config: ResConfig: NodeType: Position: License: Image:networkop/cx:3.7.0 Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[] Binds:[] PortBindings:map[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: NSPath: Publish:[] ExtraHosts:[] Labels:map[] Endpoints:[] SANs:[] Sandbox:grigoriymikh/sandbox:latest Kernel:docker.io/grigoriymikh/kernel:4.1.0 Runtime: CPU:0 CPUSet: Memory: HostRequirements:{SSE3:false VirtRequired:false} DeploymentStatus: Extras:} DEBU[0000] node config: &{ShortName:vms LongName:vms Fqdn:vms.mini-lab.io LabDir:/home/vknabel/dev/mini-lab/clab-mini-lab/vms Index:2 Group: Kind:linux StartupConfig: StartupDelay:0 EnforceStartupConfig:false ResStartupConfig: Config: ResConfig: NodeType: Position: License: Image:ghcr.io/metal-stack/mini-lab-vms:latest Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[] Binds:[] PortBindings:map[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: NSPath: Publish:[] ExtraHosts:[] Labels:map[] Endpoints:[] SANs:[] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: HostRequirements:{SSE3:false VirtRequired:false} DeploymentStatus: Extras:} DEBU[0000] lab Conf: &{Name:mini-lab Prefix:0xc000207220 Mgmt:0xc0008d8a80 Topology:0xc000aca780} DEBU[0000] Env: CLAB_VERSION_CHECK= DEBU[0000] Filter key: label, filter value: containerlab=mini-lab INFO[0000] Removing /home/vknabel/dev/mini-lab/clab-mini-lab directory... DEBU[0000] Ensuring image grigoriymikh/kernel:4.1.0 exists, or importing it... DEBU[0000] Found image with UID 5c2665d13df41a44 DEBU[0000] Ensuring image grigoriymikh/sandbox:latest exists, or importing it... DEBU[0000] Found image with UID d9d2d278021dbd28 DEBU[0000] Looking up ghcr.io/metal-stack/mini-lab-vms:latest Docker image DEBU[0000] Image ghcr.io/metal-stack/mini-lab-vms:latest present, skip pulling DEBU[0000] Ensuring image networkop/cx:3.7.0 exists, or importing it... DEBU[0000] Found image with UID dc0682667fc8d604 DEBU[0000] Number of vcpu: 12 INFO[0000] Creating lab directory: /home/vknabel/dev/mini-lab/clab-mini-lab DEBU[0000] error while trying to access file /home/vknabel/.ssh/authorized_keys: stat /home/vknabel/.ssh/authorized_keys: no such file or directory DEBU[0000] found public key files ["/home/vknabel/.ssh/id_ed25519.pub" "/home/vknabel/.ssh/id_rsa.pub"] DEBU[0000] Checking if docker network "bridge" exists DEBU[0000] network "bridge" was found. Reusing it... DEBU[0000] Docker network "bridge", bridge name "docker0" DEBU[0000] Disable RPF check on the docker host DEBU[0000] Enable LLDP on the linux bridge docker0 DEBU[0000] Disabling TX checksum offloading for the docker0 bridge interface... DEBU[0000] found iptables forwarding rule targeting the bridge "docker0". Skipping creation of the forwarding rule. DEBU[0000] scheduling nodes with dynamic IPs... DEBU[0000] Worker 3 received node: &{ShortName:leaf01 LongName:leaf01 Fqdn:leaf01.mini-lab.io LabDir:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf01 Index:0 Group: Kind:cvx StartupConfig: StartupDelay:0 EnforceStartupConfig:false ResStartupConfig: Config:0xc000672b78 ResConfig: NodeType: Position: License: Image:networkop/cx:3.7.0 Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[CLAB_INTFS:3 CLAB_LABEL_CLAB_NODE_GROUP: CLAB_LABEL_CLAB_NODE_KIND:cvx CLAB_LABEL_CLAB_NODE_LAB_DIR:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf01 CLAB_LABEL_CLAB_NODE_NAME:leaf01 CLAB_LABEL_CLAB_NODE_TYPE: CLAB_LABEL_CLAB_TOPO_FILE:/home/vknabel/dev/mini-lab/mini-lab.cumulus.yaml CLAB_LABEL_CONTAINERLAB:mini-lab] Binds:[/home/vknabel/dev/mini-lab/files/ssh/id_rsa.pub:/root/.ssh/authorized_keys] PortBindings:map[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: NSPath: Publish:[] ExtraHosts:[] Labels:map[clab-mgmt-net-bridge:docker0 clab-node-group: clab-node-kind:cvx clab-node-lab-dir:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf01 clab-node-name:leaf01 clab-node-type: clab-topo-file:/home/vknabel/dev/mini-lab/mini-lab.cumulus.yaml containerlab:mini-lab] Endpoints:[{Node:0xc000ab4000 EndpointName:swp1 MAC:aa:c1:ab:47:9f:62} {Node:0xc000ab4000 EndpointName:swp2 MAC:aa:c1:ab:5e:0e:cf} {Node:0xc000ab4000 EndpointName:swp3 MAC:aa:c1:ab:da:45:af}] SANs:[] Sandbox:grigoriymikh/sandbox:latest Kernel:docker.io/grigoriymikh/kernel:4.1.0 Runtime: CPU:0 CPUSet: Memory:768MB HostRequirements:{SSE3:false VirtRequired:false} DeploymentStatus: Extras:} DEBU[0000] Ensuring kernel grigoriymikh/kernel:4.1.0 exists, or importing it... DEBU[0000] Found kernel with UID ab0caedaad64f109 DEBU[0000] Ensuring image networkop/cx:3.7.0 exists, or importing it... DEBU[0000] Found image with UID dc0682667fc8d604 INFO[0001] Networking is handled by "docker-bridge" INFO[0001] Started Firecracker VM "32f752a4a22a86a3" in a container with ID "4b91f91287ed38547c09a8cc5008ca198d69a6378ea6b30bfd550af4f00958a0" DEBU[0001] Worker 3 received node: &{ShortName:leaf02 LongName:leaf02 Fqdn:leaf02.mini-lab.io LabDir:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf02 Index:1 Group: Kind:cvx StartupConfig: StartupDelay:0 EnforceStartupConfig:false ResStartupConfig: Config:0xc000672bf0 ResConfig: NodeType: Position: License: Image:networkop/cx:3.7.0 Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[CLAB_INTFS:3 CLAB_LABEL_CLAB_NODE_GROUP: CLAB_LABEL_CLAB_NODE_KIND:cvx CLAB_LABEL_CLAB_NODE_LAB_DIR:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf02 CLAB_LABEL_CLAB_NODE_NAME:leaf02 CLAB_LABEL_CLAB_NODE_TYPE: CLAB_LABEL_CLAB_TOPO_FILE:/home/vknabel/dev/mini-lab/mini-lab.cumulus.yaml CLAB_LABEL_CONTAINERLAB:mini-lab] Binds:[/home/vknabel/dev/mini-lab/files/ssh/id_rsa.pub:/root/.ssh/authorized_keys] PortBindings:map[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: NSPath: Publish:[] ExtraHosts:[] Labels:map[clab-mgmt-net-bridge:docker0 clab-node-group: clab-node-kind:cvx clab-node-lab-dir:/home/vknabel/dev/mini-lab/clab-mini-lab/leaf02 clab-node-name:leaf02 clab-node-type: clab-topo-file:/home/vknabel/dev/mini-lab/mini-lab.cumulus.yaml containerlab:mini-lab] Endpoints:[{Node:0xc000ab4700 EndpointName:swp1 MAC:aa:c1:ab:ff:6c:3d} {Node:0xc000ab4700 EndpointName:swp2 MAC:aa:c1:ab:60:53:87} {Node:0xc000ab4700 EndpointName:swp3 MAC:aa:c1:ab:4a:f2:b0}] SANs:[] Sandbox:grigoriymikh/sandbox:latest Kernel:docker.io/grigoriymikh/kernel:4.1.0 Runtime: CPU:0 CPUSet: Memory:768MB HostRequirements:{SSE3:false VirtRequired:false} DeploymentStatus: Extras:} DEBU[0001] Worker 0 terminating... DEBU[0001] Worker 2 terminating... DEBU[0001] Ensuring kernel grigoriymikh/kernel:4.1.0 exists, or importing it... DEBU[0001] Worker 1 received node: &{ShortName:vms LongName:vms Fqdn:vms.mini-lab.io LabDir:/home/vknabel/dev/mini-lab/clab-mini-lab/vms Index:2 Group: Kind:linux StartupConfig: StartupDelay:0 EnforceStartupConfig:false ResStartupConfig: Config:0xc000672c68 ResConfig: NodeType: Position: License: Image:ghcr.io/metal-stack/mini-lab-vms:latest Sysctls:map[net.ipv6.conf.all.disable_ipv6:0] User: Entrypoint: Cmd: Exec:[] Env:map[CLAB_INTFS:6 CLAB_LABEL_CLAB_NODE_GROUP: CLAB_LABEL_CLAB_NODE_KIND:linux CLAB_LABEL_CLAB_NODE_LAB_DIR:/home/vknabel/dev/mini-lab/clab-mini-lab/vms CLAB_LABEL_CLAB_NODE_NAME:vms CLAB_LABEL_CLAB_NODE_TYPE: CLAB_LABEL_CLAB_TOPO_FILE:/home/vknabel/dev/mini-lab/mini-lab.cumulus.yaml CLAB_LABEL_CONTAINERLAB:mini-lab] Binds:[/dev:/dev /home/vknabel/dev/mini-lab/scripts:/mini-lab] PortBindings:map[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: NSPath: Publish:[] ExtraHosts:[] Labels:map[clab-mgmt-net-bridge:docker0 clab-node-group: clab-node-kind:linux clab-node-lab-dir:/home/vknabel/dev/mini-lab/clab-mini-lab/vms clab-node-name:vms clab-node-type: clab-topo-file:/home/vknabel/dev/mini-lab/mini-lab.cumulus.yaml containerlab:mini-lab] Endpoints:[{Node:0xc000ab4e00 EndpointName:lan0 MAC:aa:c1:ab:c7:75:82} {Node:0xc000ab4e00 EndpointName:lan1 MAC:aa:c1:ab:bb:7f:90} {Node:0xc000ab4e00 EndpointName:lan2 MAC:aa:c1:ab:c1:f1:a2} {Node:0xc000ab4e00 EndpointName:lan3 MAC:aa:c1:ab:90:4d:29} {Node:0xc000ab4e00 EndpointName:lan4 MAC:aa:c1:ab:c6:6a:d7} {Node:0xc000ab4e00 EndpointName:lan5 MAC:aa:c1:ab:34:8d:15}] SANs:[] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: HostRequirements:{SSE3:false VirtRequired:false} DeploymentStatus: Extras:} INFO[0001] Creating container: "vms" DEBU[0001] creating links... DEBU[0001] Found kernel with UID ab0caedaad64f109 DEBU[0001] Ensuring image networkop/cx:3.7.0 exists, or importing it... DEBU[0001] Found image with UID dc0682667fc8d604 DEBU[0001] Container "vms" create response: {ID:57a6db88f9fbbd4f168e614292c5cd0d2e593b0557bac3458eb98a8197fee921 Warnings:[]} DEBU[0001] Start container: "vms" DEBU[0002] Container started: "vms" DEBU[0002] Worker 1 terminating... DEBU[0002] Link worker 2 received link: link [leaf01:swp3, vms:lan4] INFO[0002] Creating virtual wire: leaf01:swp3 <--> vms:lan4 DEBU[0002] Link worker 0 received link: link [leaf01:swp1, vms:lan0] INFO[0002] Creating virtual wire: leaf01:swp1 <--> vms:lan0 DEBU[0002] Link worker 1 received link: link [leaf01:swp2, vms:lan2] INFO[0002] Creating virtual wire: leaf01:swp2 <--> vms:lan2 INFO[0005] Networking is handled by "docker-bridge" INFO[0005] Started Firecracker VM "e2488e79b7cc999d" in a container with ID "2335119e7f2e8e376a15ac5929ae280dd0eb776bd1d352bbd7ab75b0c6039f5a" DEBU[0005] Worker 3 terminating... DEBU[0005] Link worker 5 received link: link [leaf02:swp2, vms:lan3] DEBU[0005] Link worker 0 terminating... INFO[0005] Creating virtual wire: leaf02:swp2 <--> vms:lan3 DEBU[0005] Link worker 4 received link: link [leaf02:swp1, vms:lan1] INFO[0005] Creating virtual wire: leaf02:swp1 <--> vms:lan1 DEBU[0005] Link worker 3 received link: link [leaf02:swp3, vms:lan5] INFO[0005] Creating virtual wire: leaf02:swp3 <--> vms:lan5 DEBU[0005] Link worker 1 terminating... DEBU[0005] Link worker 2 terminating... DEBU[0005] Link worker 5 terminating... DEBU[0005] Link worker 3 terminating... DEBU[0005] Link worker 4 terminating... DEBU[0005] containers created, retrieving state and IP addresses... DEBU[0005] Filter key: label, filter value: containerlab=mini-lab DEBU[0005] enriching nodes with IP information... DEBU[0005] Exported topology data using /etc/containerlab/templates/export/auto.tmpl template DEBU[0005] Running postdeploy actions for Linux 'vms' node DEBU[0005] Running postdeploy actions for cvx 'leaf01' node DEBU[0005] Running postdeploy actions for cvx 'leaf02' node DEBU[0005] Filter key: label, filter value: containerlab=mini-lab INFO[0005] Adding containerlab host entries to /etc/hosts file +---+--------+--------------+-----------------------------------------+-------+---------+---------------+--------------+ | # | Name | Container ID | Image | Kind | State | IPv4 Address | IPv6 Address | +---+--------+--------------+-----------------------------------------+-------+---------+---------------+--------------+ | 1 | leaf01 | 4b91f91287ed | docker.io/networkop/cx:3.7.0 | cvx | running | 172.17.0.2/24 | N/A | | 2 | leaf02 | 2335119e7f2e | docker.io/networkop/cx:3.7.0 | cvx | running | 172.17.0.4/24 | N/A | | 3 | vms | 57a6db88f9fb | ghcr.io/metal-stack/mini-lab-vms:latest | linux | running | 172.17.0.3/16 | N/A | +---+--------+--------------+-----------------------------------------+-------+---------+---------------+--------------+ ...... ```
vknabel commented 1 year ago

If you wish we can have a look together with screen sharing or vscode liveshare.

Gerrit91 commented 1 year ago

Which version of containerlab do we actually support? In the README we still recommend v0.25.1?

vknabel commented 1 year ago

Good point! When using v0.25.1 it works!

GrigoriyMikhalkin commented 1 year ago

Which version of containerlab do we actually support? In the README we still recommend v0.25.1?

I tried the latest version of containetlab locally, it works for me.

From the private talk with @vknabel we narrowed down the problem to ignite failing to start switch VMs. Probably a problem in compatibility with Ubuntu 22.04. Still working on it.

Gerrit91 commented 1 year ago

If you want me to update the runners to a newer Containerlab version, let me know.

robertvolkmann commented 9 months ago

@vknabel Do you still have issues on your side with the latest version of containerlab?

majst01 commented 9 months ago

I think we can close this @vknabel should open if still persist

vknabel commented 9 months ago

Thanks! It now works with the latest containerlab on ubuntu 22 :muscle: