search

sonic-net / sonic-utilities

Command line utilities for the SONiC project
Other
157 stars 655 forks source link

The "show interfaces status" command returns traceback in case of adding a nonexistent port to a PortChannel. Removing a nonexistent port from portchannel makes teamd container gets rebooted #787

Open MaxYaremchuk opened 4 years ago

MaxYaremchuk commented 4 years ago

Description The show interfaces status command returns traceback in case of adding a nonexistent port to a PortChannel

Moreover, In case of removing nonexistent port from portchannel teamd container gets rebooted.

Steps to reproduce the issue

  1. create a portchannel : # config portchannel add PortChannel0001
  2. Add a nonexistent interface to PortChannel0001: # config portchannel member add PortChannel0001 Ethernet128

Pay attention, interface Ethernet128 is nonexistent in the system.

  1. Issue the command: show interfaces status

Describe the results you received The system returns traceback:

Traceback (most recent call last):
  File "/usr/bin/intfutil", line 529, in <module>
    main(sys.argv[1:])
  File "/usr/bin/intfutil", line 522, in main
    interface_stat = IntfStatus(intf_name)
  File "/usr/bin/intfutil", line 443, in __init__
    self.portchannel_speed_dict = po_speed_dict(self.po_int_dict, self.appl_db)
  File "/usr/bin/intfutil", line 274, in po_speed_dict
    interface_speed = '{}G'.format(interface_speed[:-3])
TypeError: 'NoneType' object has no attribute '__getitem__'

No one docker has crashed:

# docker ps
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS              PORTS               NAMES
ac7ec90e715d        docker-sflow:latest               "/usr/bin/supervisord"   2 weeks ago         Up 14 minutes                           sflow
1efa6fef0bed        docker-snmp-sv2:latest            "/usr/bin/supervisord"   2 weeks ago         Up 14 minutes                           snmp
7cfc21362564        docker-router-advertiser:latest   "/usr/bin/supervisord"   2 weeks ago         Up 14 minutes                           radv
2a2d7be06fe8        docker-dhcp-relay:latest          "/usr/bin/docker_ini…"   2 weeks ago         Up 14 minutes                           dhcp_relay
5fb3b0430422        docker-platform-monitor:latest    "/usr/bin/docker_ini…"   2 weeks ago         Up 14 minutes                           pmon
82fbfde9fda5        docker-lldp-sv2:latest            "/usr/bin/supervisord"   2 weeks ago         Up 14 minutes                           lldp
46c789c9a866        docker-syncd-mlnx:latest          "/usr/bin/supervisord"   2 weeks ago         Up 14 minutes                           syncd
5b179f30d842        docker-teamd:latest               "/usr/bin/supervisord"   2 weeks ago         Up 14 minutes                           teamd
bbbe4d6e3f8b        docker-orchagent:latest           "/usr/bin/supervisord"   2 weeks ago         Up 14 minutes                           swss
c22e0216c4b4        docker-fpm-frr:latest             "/usr/bin/supervisord"   2 weeks ago         Up 14 minutes                           bgp
1704a3a26835        docker-database:latest            "/usr/local/bin/dock…"   7 weeks ago         Up 15 minutes                           database
  1. Having removed a nonexistent port from portchannel # config portchannel member del PortChannel0001 Ethernet128 teamd container got rebooted with errors in syslog: 
    Jan 29 19:36:49.241982 sonic INFO teamd#supervisord: teammgrd libteamdctl: cli_usock_process_msg: usock: Error message received: "NoSuchDev"
Jan 29 19:36:49.241982 sonic INFO teamd#supervisord: teammgrd libteamdctl: cli_usock_process_msg: usock: Error message content: "No such device."
Jan 29 19:36:49.241982 sonic INFO teamd#supervisord: teammgrd command call failed (Invalid argument)
Jan 29 19:36:49.246672 sonic INFO teamd#supervisord: teammgrd Cannot find device "Ethernet128"
Jan 29 19:36:49.249896 sonic ERR teamd#teammgrd: :- exec: /usr/bin/teamdctl PortChannel0001 port remove Ethernet128; /sbin/ip link set dev "Ethernet128" "down"; /sbin/ip link set dev "Ethernet128" mtu "9100": No such file or directory
Jan 29 19:36:49.250109 sonic ERR teamd#teammgrd: :- main: Runtime error: /usr/bin/teamdctl PortChannel0001 port remove Ethernet128; /sbin/ip link set dev "Ethernet128" "down"; /sbin/ip link set dev "Ethernet128" mtu "9100" : 
Jan 29 19:36:50.252397 sonic INFO teamd#supervisor-proc-exit-listener: Process teammgrd exited unxepectedly. Terminating supervisor...
Jan 29 19:36:50.683564 sonic INFO teamd#supervisord 2020-01-29 19:36:49,250 INFO exited: teammgrd (exit status 255; not expected)

Describe the results you expected The config portchannel member add command should prevent from assigning to a portchannel nonexistent interface.

Output of show version

SONiC Software Version: SONiC.HEAD.129-0c9040de
Distribution: Debian 9.11
Kernel: 4.9.0-9-2-amd64
Build commit: 0c9040de
Build date: Thu Nov 21 12:50:41 UTC 2019
Built by: johnar@jenkins-worker-4

Platform: x86_64-mlnx_msn2100-r0
HwSKU: ACS-MSN2100
ASIC: mellanox
Serial Number: MT1646X05314
Uptime: 19:51:11 up 16 min,  2 users,  load average: 3.28, 3.40, 2.52

Docker images:
REPOSITORY                 TAG                 IMAGE ID            SIZE
docker-syncd-mlnx          HEAD.129-0c9040de   4f3b126274c2        373MB
docker-syncd-mlnx          latest              4f3b126274c2        373MB
docker-fpm-frr             HEAD.129-0c9040de   7345e105bf26        321MB
docker-fpm-frr             latest              7345e105bf26        321MB
docker-sflow               HEAD.129-0c9040de   f240b81ea842        305MB
docker-sflow               latest              f240b81ea842        305MB
docker-lldp-sv2            HEAD.129-0c9040de   f1b94915bf13        299MB
docker-lldp-sv2            latest              f1b94915bf13        299MB
docker-dhcp-relay          HEAD.129-0c9040de   7ba3cee353b6        289MB
docker-dhcp-relay          latest              7ba3cee353b6        289MB
docker-database            HEAD.129-0c9040de   c5c92210277c        281MB
docker-database            latest              c5c92210277c        281MB
docker-snmp-sv2            HEAD.129-0c9040de   ad01f8a547b1        335MB
docker-snmp-sv2            latest              ad01f8a547b1        335MB
docker-orchagent           HEAD.129-0c9040de   7fc54899dc92        322MB
docker-orchagent           latest              7fc54899dc92        322MB
docker-teamd               HEAD.129-0c9040de   c0316e26bbbd        304MB
docker-teamd               latest              c0316e26bbbd        304MB
docker-sonic-telemetry     HEAD.129-0c9040de   43df42f49e91        304MB
docker-sonic-telemetry     latest              43df42f49e91        304MB
docker-router-advertiser   HEAD.129-0c9040de   b0aff9280842        281MB
docker-router-advertiser   latest              b0aff9280842        281MB
docker-platform-monitor    HEAD.129-0c9040de   10013f82e4d7        565MB
docker-platform-monitor    latest              10013f82e4d7        565MB

Debug file sonic_dump_r-ethernet-sw230_20200117_194152.tar.gz

Hedgehog-Guru commented 3 years ago

The bug has been fixed by the community under the latest branch 201911, hash dea38d1

robertlperry commented 1 year ago

Hi @Hedgehog-Guru, are we sure this issue was fixed in the above mentioned commit? It can still be reproduced in release 202211.

Sep  1 13:52:00.491344 INFO teamd#supervisord: teammgrd libteamdctl: cli_usock_process_msg: usock: Error message received: "NoSuchDev"
Sep  1 13:52:00.491822 INFO teamd#supervisord: teammgrd libteamdctl: cli_usock_process_msg: usock: Error message content: "No such device."
Sep  1 13:52:00.492043 INFO teamd#supervisord: teammgrd command call failed (Invalid argument)
Sep  1 13:52:00.537436 INFO teamd#supervisord: teammgrd Cannot find device "Ethernet48"
Sep  1 13:52:00.583468 ERR teamd#teammgrd: :- main: Runtime error: /usr/bin/teamdctl PortChannel2 port remove Ethernet48; /sbin/ip link set dev "Ethernet48" "down"; /sbin/ip link set dev "Ethernet48" mtu "9100" :
Sep  1 13:52:00.683021 INFO teamd#supervisord 2023-09-01 13:52:00,682 INFO exited: teammgrd (exit status 1; not expected)
Sep  1 13:52:01.693996 INFO teamd#supervisor-proc-exit-listener: Process 'teammgrd' exited unexpectedly. Terminating supervisor 'teamd'