search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
717 stars 1.38k forks source link

Port stats - negative rate counters after speed change #19594

Open Stas0k opened 1 month ago

Stas0k commented 1 month ago

Description

Steps to reproduce the issue:

  1. Disable auto-negotiation on ports 
    sudo config interface autoneg Ethernet0 disabled
sudo config interface autoneg Ethernet1 disabled
sudo config interface autoneg Ethernet2 disabled
  2. Change ports speed to 10M 
    sudo config interface speed Ethernet0 10
sudo config interface speed Ethernet1 10
sudo config interface speed Ethernet2 10
  3. Verify configuration applied properly and ports are in link-up state toward the link-partner 
    admin@sonic:~$ show interfaces autoneg status Ethernet0
Interface    Auto-Neg Mode    Speed    Adv Speeds    Rmt Adv Speeds    Type    Adv Types    Oper    Admin
-----------  ---------------  -------  ------------  ----------------  ------  -----------  ------  -------
Ethernet0         disabled      10M           N/A               N/A     N/A          N/A      up       up
admin@sonic:~$ show interfaces autoneg status Ethernet1
Interface    Auto-Neg Mode    Speed    Adv Speeds    Rmt Adv Speeds    Type    Adv Types    Oper    Admin
-----------  ---------------  -------  ------------  ----------------  ------  -----------  ------  -------
Ethernet1         disabled      10M           N/A               N/A     N/A          N/A      up       up
admin@sonic:~$ show interfaces autoneg status Ethernet2
Interface    Auto-Neg Mode    Speed    Adv Speeds    Rmt Adv Speeds    Type    Adv Types    Oper    Admin
-----------  ---------------  -------  ------------  ----------------  ------  -----------  ------  -------
Ethernet2         disabled      10M           N/A               N/A     N/A          N/A      up       up
admin@sonic:~$ show interfaces status Ethernet0-2
Interface    Lanes    Speed    MTU    FEC     Alias    Vlan    Oper    Admin    Type    Asym PFC
-----------  -------  -------  -----  -----  --------  ------  ------  -------  ------  ----------
Ethernet0        0      10M   9100   none  oneGigE0   trunk      up       up     N/A         N/A
Ethernet1        1      10M   9100   none  oneGigE1   trunk      up       up     N/A         N/A
Ethernet2        2      10M   9100   none  oneGigE2   trunk      up       up     N/A         N/A
  4. Transmit traffic to ports. Display counters and verify that TX/RX values were updated 
    admin@sonic:~$ portstat 
Last cached time was 2024-01-26T22:58:38.480983
 IFACE    STATE    RX_OK       RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK       TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
----------  -------  -------  -----------  ---------  --------  --------  --------  -------  -----------  ---------  --------  --------  --------
Ethernet0        U  281,256  200.80 KB/s     16.06%         0         0         0  284,390  209.80 KB/s     16.78%         0         0         0
Ethernet1        U  281,256  200.77 KB/s     16.06%         0         0         0  284,397  209.79 KB/s     16.78%         0         0         0
Ethernet2        U  187,246  188.64 KB/s     15.09%         0         0         0  284,396  209.75 KB/s     16.78%         0         0         0
Ethernet3        D        0     0.00 B/s      0.00%         0         0         0        0     0.00 B/s      0.00%         0         0         0
  5. Clear port counters 
    admin@sonic:~$ portstat -c
  6. Change ports speed to 100M 
    sudo config interface speed Ethernet0 10
sudo config interface speed Ethernet1 10
sudo config interface speed Ethernet2 10
admin@sonic:~$ show interfaces status Ethernet0-2
Interface    Lanes    Speed    MTU    FEC     Alias    Vlan    Oper    Admin    Type    Asym PFC
-----------  -------  -------  -----  -----  --------  ------  ------  -------  ------  ----------
Ethernet0        0     100M   9100   none  oneGigE0   trunk      up       up     N/A         N/A
Ethernet1        1     100M   9100   none  oneGigE1   trunk      up       up     N/A         N/A
Ethernet2        2     100M   9100   none  oneGigE2   trunk      up       up     N/A         N/A
  7. Transmit traffic to ports. Display counters and] observe that TX_BPS and RX_BPS values are negative 
    admin@sonic:~$ portstat 
Last cached time was 2024-01-26T23:04:50.989027
 IFACE    STATE    RX_OK          RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK          TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
----------  -------  -------  --------------  ---------  --------  --------  --------  -------  --------------  ---------  --------  --------  --------
Ethernet0        U        0      -18.04 B/s     -0.00%         0         0         0        0        8.76 B/s      0.00%         0         0         0
Ethernet1        U        0     -782.85 B/s     -0.01%         0         0         0        0     -764.03 B/s     -0.01%         0         0         0
Ethernet2        U        0  -136307.52 B/s     -1.09%         0         0         0        0  -137821.28 B/s     -1.10%         0         0         0
Ethernet3        D        0        0.00 B/s      0.00%         0         0         0        0        0.00 B/s      0.00%         0         0         0

Describe the results you received:

Negative port counters, probably due to wrong computation

Describe the results you expected:

Expected to get positive values aligned to the transmitted traffic rate

Output of show version:

Source: Jul 14 '24 master build from https://sonic-build.azurewebsites.net/

admin@sonic:~$ show version

SONiC Software Version: SONiC.master.594957-e2b30cc49
SONiC OS Version: 12
Distribution: Debian 12.5
Kernel: 6.1.0-11-2-arm64
Build commit: e2b30cc49
Build date: Sun Jul 14 08:48:58 UTC 2024
Built by: cloudtest@a7d7055cc000001

Platform: arm64-marvell_rd98DX35xx-r0
HwSKU: rd98DX35xx
ASIC: marvell
ASIC Count: 1
Serial Number: NA
Model Number: NA
Hardware Revision: N/A
Uptime: 23:07:10 up  1:18,  1 user,  load average: 2.98, 3.08, 2.71
Date: Fri 26 Jan 2024 23:07:10

Docker images:
REPOSITORY                    TAG                       IMAGE ID       SIZE
docker-platform-monitor       latest                    45d46157579f   813MB
docker-platform-monitor       master.594957-e2b30cc49   45d46157579f   813MB
docker-macsec                 latest                    0890630e9a2f   489MB
docker-teamd                  latest                    048fe67e471b   486MB
docker-teamd                  master.594957-e2b30cc49   048fe67e471b   486MB
docker-orchagent              latest                    15a6f83213c0   500MB
docker-orchagent              master.594957-e2b30cc49   15a6f83213c0   500MB
docker-sflow                  latest                    33581824827c   488MB
docker-sflow                  master.594957-e2b30cc49   33581824827c   488MB
docker-fpm-frr                latest                    93d5dea224e7   520MB
docker-fpm-frr                master.594957-e2b30cc49   93d5dea224e7   520MB
docker-nat                    latest                    ed3011525535   495MB
docker-nat                    master.594957-e2b30cc49   ed3011525535   495MB
docker-eventd                 latest                    24e9d5eee8af   458MB
docker-eventd                 master.594957-e2b30cc49   24e9d5eee8af   458MB
docker-dhcp-relay             latest                    9422e1696bf3   468MB
docker-snmp                   latest                    971717b4464d   499MB
docker-snmp                   master.594957-e2b30cc49   971717b4464d   499MB
docker-router-advertiser      latest                    ce0222fa7266   459MB
docker-router-advertiser      master.594957-e2b30cc49   ce0222fa7266   459MB
docker-lldp                   latest                    382aae030fb8   466MB
docker-lldp                   master.594957-e2b30cc49   382aae030fb8   466MB
docker-mux                    latest                    0606cc9d8109   473MB
docker-mux                    master.594957-e2b30cc49   0606cc9d8109   473MB
docker-database               latest                    bfbad7e27672   467MB
docker-database               master.594957-e2b30cc49   bfbad7e27672   467MB
docker-sonic-gnmi             latest                    2e2b6a424195   539MB
docker-sonic-gnmi             master.594957-e2b30cc49   2e2b6a424195   539MB
docker-syncd-mrvl             latest                    eff2889a84e9   568MB
docker-syncd-mrvl             master.594957-e2b30cc49   eff2889a84e9   568MB
docker-sonic-mgmt-framework   latest                    3e21921a1bc6   504MB
docker-sonic-mgmt-framework   master.594957-e2b30cc49   3e21921a1bc6   504MB

admin@sonic:~$

Output of show techsupport:

Skipped

Additional information you deem important (e.g. issue happens only occasionally):

Log: portstats_negative_rate_counters.txt

vmittal-msft commented 1 month ago

@Stas0k Please help forward to right person.

Stas0k commented 1 month ago

Forwarded to @matiAlfaro

Stas0k commented 1 month ago

relative issue in swss repo - https://github.com/sonic-net/sonic-swss/issues/1892

Stas0k commented 3 weeks ago

PR swss #3237 fixes the issue (verified)