search

sonic-net / sonic-mgmt

Configuration management examples for SONiC
Other
201 stars 727 forks source link

"show interfaces counters" are always 0 on T2-min topology with 202305 image #12184

Closed parmarkj closed 4 months ago

parmarkj commented 7 months ago

Description

I have a T2-min topology up with vsonic neighbors. I am trying to work on a port channel load balance test, however, counters are always 0. All counters are staying 0 even when sending continuous pings.

Steps to reproduce the issue:

  1. "show ip bgp summary", "show lldp table", "show interfaces counters" on DUT and neighbor
  2. ping the neighbor
  3. "show interface counters" on DUT and neighbor

Describe the results you received: Logs from DUT (vlab-t2-01): output of 'show ip bgp summary':

IPv4 Unicast Summary:
BGP router identifier 10.1.0.1, local AS number 65100 vrf-id 0
BGP table version 190875
RIB entries 101641, using 19515072 bytes of memory
Peers 4, using 2967904 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbhor      V     AS    MsgRcvd    MsgSent    TblVer    InQ    OutQ  Up/Down      State/PfxRcd  NeighborName
-----------  ---  -----  ---------  ---------  --------  -----  ------  ---------  --------------  --------------
10.0.0.1       4  65200     142515     142667         0      0       0  4d06h30m            34050  ARISTA01T3
10.0.0.5       4  65200     142210     142351         0      0       0  4d22h17m            34049  ARISTA03T3
10.0.0.7       4  65200     142201     142343         0      0       0  4d22h17m            33922  ARISTA04T3
10.0.0.11      4  65200     142765     143090         0      0       0  22:20:43            34049  ARISTA06T3

output of 'show lldp table':

Capability codes: (R) Router, (B) Bridge, (O) Other
LocalPort    RemoteDevice    RemotePortID    Capability    RemotePortDescr
-----------  --------------  --------------  ------------  -----------------
Ethernet16   ARISTA01T3      fortyGigE0/0    BR            Ethernet1
Ethernet20   ARISTA01T3      fortyGigE0/4    BR            Ethernet2
Ethernet32   ARISTA03T3      fortyGigE0/0    BR            Ethernet1
Ethernet92   ARISTA06T3      fortyGigE0/0    BR            Ethernet1
Ethernet104  ARISTA04T3      fortyGigE0/0    BR            Ethernet1
Ethernet108  ARISTA04T3      fortyGigE0/4    BR            Ethernet2
--------------------------------------------------
Total entries displayed:  6

output of 'show interfaces counters | grep U' before starting continuous pings to 'ARISTA01T3':

      IFACE    STATE    RX_OK    RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK    TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
 Ethernet16        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
 Ethernet20        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
 Ethernet32        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
 Ethernet92        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet104        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet108        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0

Sending continuous pings to 'ARISTA01T3':

PING 10.250.0.51 (10.250.0.51) 56(84) bytes of data.
64 bytes from 10.250.0.51: icmp_seq=1 ttl=64 time=3.90 ms
64 bytes from 10.250.0.51: icmp_seq=2 ttl=64 time=1.01 ms
64 bytes from 10.250.0.51: icmp_seq=3 ttl=64 time=0.951 ms
64 bytes from 10.250.0.51: icmp_seq=4 ttl=64 time=1.07 ms
64 bytes from 10.250.0.51: icmp_seq=5 ttl=64 time=1.42 ms
...
64 bytes from 10.250.0.51: icmp_seq=128 ttl=64 time=22.3 ms
64 bytes from 10.250.0.51: icmp_seq=129 ttl=64 time=13.1 ms
64 bytes from 10.250.0.51: icmp_seq=130 ttl=64 time=0.912 ms
64 bytes from 10.250.0.51: icmp_seq=131 ttl=64 time=1.43 ms
^C
--- 10.250.0.51 ping statistics ---
131 packets transmitted, 131 received, 0% packet loss, time 130395ms
rtt min/avg/max/mdev = 0.441/1.470/22.308/2.472 ms

While pings were being sent: 

      IFACE    STATE    RX_OK    RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK    TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
 Ethernet16        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
 Ethernet20        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
 Ethernet32        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
 Ethernet92        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet104        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet108        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0

End of pings:

      IFACE    STATE    RX_OK    RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK    TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
 Ethernet16        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
 Ethernet20        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
 Ethernet32        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
 Ethernet92        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet104        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet108        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0

Logs from 'ARISTA01T3': Output of 'show ip bgp summary':

IPv4 Unicast Summary:
BGP router identifier 100.1.0.1, local AS number 65200 vrf-id 0
BGP table version 102153
RIB entries 68099, using 13075008 bytes of memory
Peers 2, using 1483952 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbhor        V     AS    MsgRcvd    MsgSent    TblVer    InQ    OutQ  Up/Down      State/PfxRcd  NeighborName
-------------  ---  -----  ---------  ---------  --------  -----  ------  ---------  --------------  ---------------------------------
10.0.0.0         4  65100     142650     142512         0      0       0  4d06h30m                1  vlab-t2-01,vlab-t2-02,vlab-t2-sup
10.10.246.254    4  65200     209908     141836         0      0       0  4d06h29m            34049  exabgp

Output of 'show lldp table':

Capability codes: (R) Router, (B) Bridge, (O) Other
LocalPort    RemoteDevice    RemotePortID    Capability    RemotePortDescr
-----------  --------------  --------------  ------------  --------------------
Ethernet1    vlab-t2-01      fortyGigE0/16   BR            ARISTA01T3:Ethernet1
Ethernet2    vlab-t2-01      fortyGigE0/20   BR            ARISTA01T3:Ethernet2
--------------------------------------------------
Total entries displayed:  2

Output of 'show interfaces counters' from 'ARISTA01T3':

    IFACE    STATE    RX_OK    RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK    TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
---------  -------  -------  --------  ---------  --------  --------  --------  -------  --------  ---------  --------  --------  --------
Ethernet1        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet2        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet3        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet4        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0
Ethernet5        U        0  0.00 B/s      0.00%         0         0         0        0  0.00 B/s      0.00%         0         0         0

Describe the results you expected: I expected the Tx and Rx rates to be incrementing on the interfaces where bgp sessions are established as well as when sending pings a neighbor.

Additional information you deem important:

**Output of `show version`:**

```

SONiC Software Version: SONiC.202305.498016-09eb07da5 SONiC OS Version: 11 Distribution: Debian 11.8 Kernel: 5.10.0-23-2-amd64 Build commit: 09eb07da5 Build date: Wed Mar 13 11:53:21 UTC 2024 Built by: AzDevOps@vmss-soni0039E8

Platform: x86_64-kvm_x86_64-r0 HwSKU: Force10-S6000 ASIC: vs ASIC Count: 1 Serial Number: N/A Model Number: N/A Hardware Revision: N/A Uptime: 14:00:02 up 7 days, 21:58, 1 user, load average: 0.37, 0.32, 0.28 Date: Tue 26 Mar 2024 14:00:02

Docker images: REPOSITORY TAG IMAGE ID SIZE docker-orchagent 202305.498016-09eb07da5 316e93b64349 330MB docker-orchagent latest 316e93b64349 330MB docker-fpm-frr 202305.498016-09eb07da5 94c80fef4f5a 349MB docker-fpm-frr latest 94c80fef4f5a 349MB docker-nat 202305.498016-09eb07da5 4849b1d0217e 321MB docker-nat latest 4849b1d0217e 321MB docker-sflow 202305.498016-09eb07da5 ea351f6ac1cf 319MB docker-sflow latest ea351f6ac1cf 319MB docker-teamd 202305.498016-09eb07da5 dbc1cda6b709 318MB docker-teamd latest dbc1cda6b709 318MB docker-macsec latest 821224d3779e 320MB docker-syncd-vs 202305.498016-09eb07da5 a7543107c66c 315MB docker-syncd-vs latest a7543107c66c 315MB docker-gbsyncd-vs 202305.498016-09eb07da5 46aa0897f2f0 311MB docker-gbsyncd-vs latest 46aa0897f2f0 311MB docker-dhcp-relay latest 2d53f987941e 308MB docker-eventd 202305.498016-09eb07da5 a64a864234cb 300MB docker-eventd latest a64a864234cb 300MB docker-snmp 202305.498016-09eb07da5 e4a77987891a 339MB docker-snmp latest e4a77987891a 339MB docker-sonic-telemetry 202305.498016-09eb07da5 976b94d6cf30 387MB docker-sonic-telemetry latest 976b94d6cf30 387MB docker-platform-monitor 202305.498016-09eb07da5 e69b5714a774 422MB docker-platform-monitor latest e69b5714a774 422MB docker-lldp 202305.498016-09eb07da5 5cc2f6208ab1 343MB docker-lldp latest 5cc2f6208ab1 343MB docker-database 202305.498016-09eb07da5 6deecfc79608 300MB docker-database latest 6deecfc79608 300MB docker-mux 202305.498016-09eb07da5 6190d4aecc2f 349MB docker-mux latest 6190d4aecc2f 349MB docker-router-advertiser 202305.498016-09eb07da5 a8f2aed2d8a7 300MB docker-router-advertiser latest a8f2aed2d8a7 300MB docker-sonic-mgmt-framework 202305.498016-09eb07da5 b0a94042e7d7 414MB docker-sonic-mgmt-framework latest b0a94042e7d7 414MB


    **Attach debug file `sudo generate_dump`:**

Lock succesfully accquired and installed signal handlers

Traceback (most recent call last): File "/usr/local/bin/decode-syseeprom", line 18, in import sonic_platform ModuleNotFoundError: No module named 'sonic_platform' can't get debug descriptor: Resource temporarily unavailable No sensors found! Make sure you loaded all the kernel drivers you need. Try sensors-detect to find out which these are. sysctl: reading key "net.ipv4.tcp_min_tso_segs" /usr/local/bin/generate_dump: line 536: $2: unbound variable ERR: RC:-1 observed on line 536 conntrack v1.4.6 (conntrack-tools): 0 flow entries have been shown. conntrack v1.4.6 (conntrack-tools): 0 flow entries have been shown. conntrack v1.4.6 (conntrack-tools): 1418 flow entries have been shown. conntrack v1.4.6 (conntrack-tools): 1418 flow entries have been shown. bfdd is not running bfdd is not running bfdd is not running bfdd is not running /dev/sda: No such file or directory The route table's size is 101669(ipv4 50827, ipv6 50842) Dump by using Redis SAVE. Remove secret from etc files. sed: can't read /var/dump/sonic_dump_vlab-t2-01_20240326_142030/etc/pam_radius_auth.d/*.conf: No such file or directory sed: can't read /etc/cron.d/logrotate: No such file or directory ERR: RC:-2 observed on line 1456 sed: can't read /etc/cron.d/logrotate: No such file or directory ERR: RC:-2 observed on line 1496 tar: sonic_dump_vlab-t2-01_20240326_142030/etc/runit/runsvdir/default/ssh/log/supervise: File removed before we read it ERR: RC:-1 observed on line 280 find: ‘/var/log/sai_failure_dump/’: No such file or directory Cleaning up working directory /var/dump/sonic_dump_vlab-t2-01_20240326_142030 Removing lock. Exit: 0 /var/dump/sonic_dump_vlab-t2-01_20240326_142030.tar.gz

rlhui commented 5 months ago

@judyjoseph will check

rlhui commented 5 months ago

@anamehra will take a look

judyjoseph commented 5 months ago

@parmarkj are you still seeing issue

parmarkj commented 5 months ago

@judyjoseph yes I am

judyjoseph commented 4 months ago

Checked in the latest master based sonic vm 

admin@vlab-01:~$ show ver

SONiC Software Version: SONiC.master.0-dirty-20240620.055306
SONiC OS Version: 12
Distribution: Debian 12.5
Kernel: 6.1.0-11-2-amd64
Build commit: 1fba66c5d
Build date: Thu Jun 20 07:01:53 UTC 2024
 Platform: x86_64-kvm_x86_64-r0
HwSKU: Force10-S6000
ASIC: vs
ASIC Count: 1
Serial Number: N/A
Model Number: N/A
Hardware Revision: N/A
Uptime: 06:18:33 up 1 day, 19:24,  1 user,  load average: 0.07, 0.14, 0.13
Date: Wed 26 Jun 2024 06:18:33

image

Did you try with the latest master image ?

parmarkj commented 4 months ago

@judyjoseph I just upgraded to the latest master image and counters are no longer 0.

admin@vlab-t2-01:~$ show ver 

SONiC Software Version: SONiC.master.580987-a42e9aa9f
SONiC OS Version: 12
Distribution: Debian 12.5
Kernel: 6.1.0-11-2-amd64
Build commit: a42e9aa9f
Build date: Wed Jun 26 11:46:36 UTC 2024
Built by: cloudtest@ffc080a3c000001

Platform: x86_64-kvm_x86_64-r0
HwSKU: Force10-S6000
ASIC: vs
ASIC Count: 1
Serial Number: N/A
Model Number: N/A
Hardware Revision: N/A
Uptime: 03:39:41 up 28 min,  1 user,  load average: 0.49, 1.16, 1.20
Date: Thu 27 Jun 2024 03:39:41
admin@vlab-t2-01:~$ show interfaces counters 
      IFACE    STATE    RX_OK      RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK      TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
-----------  -------  -------  ----------  ---------  --------  --------  --------  -------  ----------  ---------  --------  --------  --------
  Ethernet0        X        0    0.00 B/s      0.00%         0         8         0        0    0.00 B/s      0.00%         0         0         0
  Ethernet4        X        0    0.00 B/s      0.00%         0         5         0        0    0.00 B/s      0.00%         0         0         0
  Ethernet8        X        0    0.00 B/s      0.00%         0         5         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet12        X        0    0.00 B/s      0.00%         0         2         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet16        U       82    0.24 B/s      0.00%         0         0         0       94   24.42 B/s      0.00%         0         0         0
 Ethernet20        U    1,411  112.75 B/s      0.00%         0        32         0    1,401  136.94 B/s      0.00%         0         0         0
 Ethernet24        X        0    0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet28        X        0    0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet32        U    1,350  119.69 B/s      0.00%         0         0         0    6,705  205.28 B/s      0.00%         0         0         0
 Ethernet36        X        0    0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet40        X        0    0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet44        X        0    0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet48        X        0    0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet52        X        0    0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet56        X        0    0.00 B/s      0.00%         0         2         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet60        X        0    0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet64        X        0    0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet68        X        0    0.00 B/s      0.00%         0        26         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet72        X        0    0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet76        X        0    0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet80        X        0    0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet84        X        0    0.00 B/s      0.00%         0        54         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet88        X        0    0.00 B/s      0.00%         0         2         0        0    0.00 B/s      0.00%         0         0         0
 Ethernet92        U    1,283  107.67 B/s      0.00%         0        13         0    1,366  115.67 B/s      0.00%         0         0         0
 Ethernet96        X        0    0.00 B/s      0.00%         0         4         0        0    0.00 B/s      0.00%         0         0         0
Ethernet100        X        0    0.00 B/s      0.00%         0         2         0        0    0.00 B/s      0.00%         0         0         0
Ethernet104        U       79   22.51 B/s      0.00%         0         2         0       88   24.45 B/s      0.00%         0         0         0
Ethernet108        U    1,393  131.12 B/s      0.00%         0         0         0    1,427  131.94 B/s      0.00%         0         0         0
Ethernet112        X        0    0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet116        X        0    0.00 B/s      0.00%         0         2         0        0    0.00 B/s      0.00%         0         0         0
Ethernet120        X        0    0.00 B/s      0.00%         0         2         0        0    0.00 B/s      0.00%         0         0         0
Ethernet124        X        0    0.00 B/s      0.00%         0         2         0        0    0.00 B/s      0.00%         0         0         0
admin@vlab-t2-01:~$

Closing the issue.