search

srl-labs / containerlab

container-based networking labs
https://containerlab.dev
BSD 3-Clause "New" or "Revised" License
1.52k stars 261 forks source link

ceos 4.27.3F intermittently not creating interfaces #853

Closed notworker123 closed 2 years ago

notworker123 commented 2 years ago

Hi,

ceos 4.27.3f containerlab 0.25.1

from CLI :

show int descr Interface Status Protocol Description Et1 up up Et2 up up Et4 up up Et5 up up Et6 up up Et7 up up

Interface 3 is missing

from yaml file I can see it's there:

links:

leaf to leaf for mlag

- endpoints: ["r1-lf01:eth3","r1-lf02:eth3"]
- endpoints: ["r2-lf01:eth3","r2-lf02:eth3"]

doesn't happen all the time, sometimes it is created or other interfaces don't come installed

arista logs show nothing of importance. docker logs i think nothing of importance

ip netns exec clab-vmx-ceos-ubuntu-r2-lf01 ip link

427: eth0@if428: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default link/ether 00:1c:73:4d:50:dc brd ff:ff:ff:ff:ff:ff link-netnsid 0 439: eth7@if440: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default link/ether aa:c1:ab:e5:34:bb brd ff:ff:ff:ff:ff:ff link-netns clab-vmx-ceos-ubuntu-spine04 453: eth5@if454: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default link/ether aa:c1:ab:bb:8f:9d brd ff:ff:ff:ff:ff:ff link-netns clab-vmx-ceos-ubuntu-spine02 459: eth6@if460: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default link/ether aa:c1:ab:c3:3a:70 brd ff:ff:ff:ff:ff:ff link-netns clab-vmx-ceos-ubuntu-spine03 463: eth1@if464: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9214 qdisc noqueue state UP mode DEFAULT group default link/ether aa:c1:ab:64:ed:72 brd ff:ff:ff:ff:ff:ff link-netns clab-vmx-ceos-ubuntu-r2-h01 485: eth4@if486: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default link/ether aa:c1:ab:c4:9c:10 brd ff:ff:ff:ff:ff:ff link-netns clab-vmx-ceos-ubuntu-spine01 491: eth2@if492: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9214 qdisc noqueue state UP mode DEFAULT group default link/ether aa:c1:ab:ce:da:05 brd ff:ff:ff:ff:ff:ff link-netns clab-vmx-ceos-ubuntu-r2-h02 498: eth3@if497: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP mode DEFAULT group default link/ether aa:c1:ab:91:cf:75 brd ff:ff:ff:ff:ff:ff link-netns clab-vmx-ceos-ubuntu-r2-lf02

notice eth3 is there but MTU is 9500 which is different from all the others.

also happens when i have cEOS connected to VR's (such as vMX).

hellt commented 2 years ago

I can imagine that this might be possible due to the fact that interfaces are added after the container is started. My assumption is that the scripts that ceos runs to connect container interfaces finishes before all the interfaces are actually created by containerlab.

There are a few options we can have a look to make sure that arista ceos first waits for interfaces to appear before starting its boot process

notworker123 commented 2 years ago

thanks.

my clab file if u need it :

clab-config.txt

hw is a dual-proc 8260 w/ 500gb of ram

hellt commented 2 years ago

@notworker123 can you check this beta build which includes a potential fix for your problem? This docker pull command will get a containerlab binary at your PWD so you can use it instead with ./containerlab ...

docker run --rm -v $(pwd):/workspace ghcr.io/oras-project/oras:v0.12.0 pull ttl.sh/clab-2d610f44:1d
notworker123 commented 2 years ago

from just one attempt all the interfaces are there. so it seems fixed.

i'll likely be more destroying/deploying during the work week and will report back if any issues.

thanks for a quick fix and this software. i'm really liking it thus far.

hellt commented 2 years ago

🥳 thats great yes, I will push this code so that it will be part of the next public release ~ next week