search

aristanetworks / sonic

Open source drivers and initialization library for Arista platforms running SONiC
GNU General Public License v2.0
22 stars 30 forks source link

[chassis] speed change fails with SAI error #85

Closed arlakshm closed 1 year ago

arlakshm commented 1 year ago

in the latest images from branch 202205, on boot-up with configuration with speed changes, the following sai failure are seen

Mar  6 16:27:43.989373 str2-7804-lc5-1 INFO syncd0#syncd: message repeated 7 times: [ [06:00.0] SAI_API_PORT:brcm_sai_get_port_stats:5470 Multi stats get failed with error Feature unavailable (0xfffffff0).]
Mar  6 16:27:43.989373 str2-7804-lc5-1 NOTICE syncd0#syncd: :- addQueueCounterPlugin: Queue counters plugin 1bb02798ef54372833ae5ec06d49d788c4da1cf5 registered
Mar  6 16:27:43.989373 str2-7804-lc5-1 NOTICE syncd0#syncd: :- addQueueCounterPlugin: Queue counters plugin 81da146fb0bdfad1a834e5b4448fdb9de034fc5b registered
Mar  6 16:27:43.990392 str2-7804-lc5-1 NOTICE syncd0#syncd: :- processNotifySyncd: very first run is TRUE, op = APPLY_VIEW
Mar  6 16:27:43.990392 str2-7804-lc5-1 NOTICE syncd0#syncd: :- processNotifySyncd: setting very first run to FALSE, op = APPLY_VIEW
Mar  6 16:27:43.999371 str2-7804-lc5-1 INFO syncd0#syncd: [06:00.0] SAI_API_PORT:brcm_sai_remove_port_serdes:8231 Serdes object remove port 18 serdes id 12005700000000
Mar  6 16:27:43.999898 str2-7804-lc5-1 NOTICE syncd0#syncd: :- collectPortRelatedObjects: obtained 33 port oid:0x100000012 related RIDs
Mar  6 16:27:43.999929 str2-7804-lc5-1 INFO syncd0#syncd: [06:00.0] SAI_API_PORT:brcm_sai_remove_port:6841 REMOVE_PORT port_id:0x100000012, bcm_port:18, phy:0
Mar  6 16:27:44.110904 str2-7804-lc5-1 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:brcm_sai_remove_port:6995 driver_dnx_SetEgressOfpRate failed with error Internal error (0xffffffff).
Mar  6 16:27:44.110914 str2-7804-lc5-1 ERR syncd0#syncd: :- sendApiResponse: api SAI_COMMON_API_REMOVE failed in syncd mode: SAI_STATUS_FAILURE

Image details:

admin@str2-7804-lc5-1:~$ show vers

SONiC Software Version: SONiC.202205.228502-ecb528fac
Distribution: Debian 11.6
Kernel: 5.10.0-18-2-amd64
Build commit: ecb528fac
Build date: Sat Mar  4 12:22:30 UTC 2023
Built by: AzDevOps@vmss-soni000M4N

Platform: x86_64-arista_7800r3a_36dm2_lc
HwSKU: Arista-7800R3A-36DM2-D36
ASIC: broadcom
ASIC Count: 2
Serial Number: SGD21190878
Model Number: 7800R3A-36DM2-LC
Hardware Revision: 2a.00
Uptime: 16:30:05 up 18 min,  2 users,  load average: 2.08, 2.35, 1.94
Date: Mon 06 Mar 2023 16:30:05

Docker images:
REPOSITORY                    TAG                       IMAGE ID       SIZE
docker-orchagent              202205.228502-ecb528fac   e456f4090fc1   444MB
docker-orchagent              latest                    e456f4090fc1   444MB
docker-fpm-frr                202205.228502-ecb528fac   52dcfdd7c647   455MB
docker-fpm-frr                latest                    52dcfdd7c647   455MB
docker-teamd                  202205.228502-ecb528fac   2107a9f01347   425MB
docker-teamd                  latest                    2107a9f01347   425MB
docker-macsec                 latest                    ec2f6f66e023   428MB
docker-syncd-brcm-dnx         202205.228502-ecb528fac   d23e618949b0   755MB
docker-syncd-brcm-dnx         latest                    d23e618949b0   755MB
docker-gbsyncd-broncos        202205.228502-ecb528fac   f2c5a1135290   456MB
docker-gbsyncd-broncos        latest                    f2c5a1135290   456MB
docker-gbsyncd-credo          202205.228502-ecb528fac   976cfe412d67   427MB
docker-gbsyncd-credo          latest                    976cfe412d67   427MB
docker-dhcp-relay             latest                    384da9646cb9   419MB
docker-snmp                   202205.228502-ecb528fac   60250b163488   454MB
docker-snmp                   latest                    60250b163488   454MB
docker-platform-monitor       202205.228502-ecb528fac   46e81ad2684c   535MB
docker-platform-monitor       latest                    46e81ad2684c   535MB
docker-sonic-telemetry        202205.228502-ecb528fac   f901c85a7d0b   490MB
docker-sonic-telemetry        latest                    f901c85a7d0b   490MB
docker-lldp                   202205.228502-ecb528fac   f532116f8be4   452MB
docker-lldp                   latest                    f532116f8be4   452MB
docker-mux                    202205.228502-ecb528fac   c6ff173b0b52   458MB
docker-mux                    latest                    c6ff173b0b52   458MB
docker-database               202205.228502-ecb528fac   a5900145bda6   410MB
docker-database               latest                    a5900145bda6   410MB
docker-router-advertiser      202205.228502-ecb528fac   33db8edd3e33   410MB
docker-router-advertiser      latest                    33db8edd3e33   410MB
docker-nat                    202205.228502-ecb528fac   3d7fe48b13e3   397MB
docker-nat                    latest                    3d7fe48b13e3   397MB
docker-sflow                  202205.228502-ecb528fac   1baad02eb5a8   395MB
docker-sflow                  latest                    1baad02eb5a8   395MB
docker-sonic-mgmt-framework   202205.228502-ecb528fac   9a8a531e1def   523MB
docker-sonic-mgmt-framework   latest                    9a8a531e1def   523MB
arlakshm commented 1 year ago

This problem is not seen in from 03/1 with the latest sai. Logs below

admin@str2-7804-lc5-1:~$ show version

SONiC Software Version: SONiC.202205.226744-1f6456a60
Distribution: Debian 11.6
Kernel: 5.10.0-18-2-amd64
Build commit: 1f6456a60
Build date: Wed Mar  1 11:27:55 UTC 2023
Built by: AzDevOps@vmss-soni000LD7

Platform: x86_64-arista_7800r3a_36dm2_lc
HwSKU: Arista-7800R3A-36DM2-D36
ASIC: broadcom
ASIC Count: 2
Serial Number: SGD21190878
Model Number: 7800R3A-36DM2-LC
Hardware Revision: 2a.00
Uptime: 16:03:32 up 41 min,  2 users,  load average: 3.81, 3.00, 2.14
Date: Mon 06 Mar 2023 16:03:32

Docker images:
REPOSITORY                    TAG                       IMAGE ID       SIZE
docker-orchagent              202205.226744-1f6456a60   9fa15a15fd3b   444MB
docker-orchagent              latest                    9fa15a15fd3b   444MB
docker-fpm-frr                202205.226744-1f6456a60   7b6ce638d2c5   455MB
docker-fpm-frr                latest                    7b6ce638d2c5   455MB
docker-teamd                  202205.226744-1f6456a60   3e7149fdd754   425MB
docker-teamd                  latest                    3e7149fdd754   425MB
docker-macsec                 latest                    94c30270924c   427MB
docker-syncd-brcm-dnx         202205.226744-1f6456a60   10bde5afa36d   755MB
docker-syncd-brcm-dnx         latest                    10bde5afa36d   755MB
docker-gbsyncd-broncos        202205.226744-1f6456a60   21b5b6f1d692   456MB
docker-gbsyncd-broncos        latest                    21b5b6f1d692   456MB
docker-gbsyncd-credo          202205.226744-1f6456a60   ff09b6e30efc   427MB
docker-gbsyncd-credo          latest                    ff09b6e30efc   427MB
docker-dhcp-relay             latest                    874fa696314c   419MB
docker-snmp                   202205.226744-1f6456a60   24f0ace94781   454MB
docker-snmp                   latest                    24f0ace94781   454MB
docker-platform-monitor       202205.226744-1f6456a60   a0c638011ee3   534MB
docker-platform-monitor       latest                    a0c638011ee3   534MB
docker-sonic-telemetry        202205.226744-1f6456a60   27817e193aab   490MB
docker-sonic-telemetry        latest                    27817e193aab   490MB
docker-lldp                   202205.226744-1f6456a60   fe14fd56360f   452MB
docker-lldp                   latest                    fe14fd56360f   452MB
docker-database               202205.226744-1f6456a60   94986da7bd9e   409MB
docker-database               latest                    94986da7bd9e   409MB
docker-mux                    202205.226744-1f6456a60   2f9c368e7795   458MB
docker-mux                    latest                    2f9c368e7795   458MB
docker-router-advertiser      202205.226744-1f6456a60   ca84cdd76011   409MB
docker-router-advertiser      latest                    ca84cdd76011   409MB
docker-nat                    202205.226744-1f6456a60   7b3384676fb3   397MB
docker-nat                    latest                    7b3384676fb3   397MB
docker-sflow                  202205.226744-1f6456a60   fedc4d35d58f   395MB
docker-sflow                  latest                    fedc4d35d58f   395MB
docker-sonic-mgmt-framework   202205.226744-1f6456a60   03c7d6dbd11f   523MB
docker-sonic-mgmt-framework   latest                    03c7d6dbd11f   523MB

admin@str2-7804-lc5-1:~$ bcmcm -n 0 "bsv"
-bash: bcmcm: command not found
admin@str2-7804-lc5-1:~$ bcmcmd -n 0 "bsv"
bsv
BRCM SAI ver: [7.1.36.4], OCP SAI ver: [1.10.2], SDK ver: [sdk-6.5.24]
drivshell>
admin@str2-7804-lc5-1:~$

admin@str2-7804-lc5-1:~$ show interface status
      Interface                    Lanes    Speed    MTU    FEC         Alias             Vlan    Oper    Admin             Type    Asym PFC
---------------  -----------------------  -------  -----  -----  ------------  ---------------  ------  -------  ---------------  ----------
      Ethernet0              72,73,74,75     100G   9100     rs   Ethernet1/1   PortChannel102      up       up  QSFP28 or later         off
      Ethernet8              80,81,82,83     100G   9100     rs   Ethernet2/1   PortChannel102      up       up  QSFP28 or later         off
     Ethernet16              88,89,90,91     100G   9100     rs   Ethernet3/1   PortChannel104      up       up  QSFP28 or later         off
     Ethernet24              96,97,98,99     100G   9100     rs   Ethernet4/1   PortChannel104      up       up  QSFP28 or later         off
     Ethernet32          104,105,106,107     100G   9100     rs   Ethernet5/1   PortChannel106      up       up  QSFP28 or later         off
     Ethernet40          112,113,114,115     100G   9100     rs   Ethernet6/1   PortChannel106      up       up  QSFP28 or later         off
     Ethernet48          120,121,122,123     100G   9100     rs   Ethernet7/1   PortChannel108      up       up  QSFP28 or later         off
     Ethernet56          128,129,130,131     100G   9100     rs   Ethernet8/1   PortChannel108      up       up  QSFP28 or later         off
     Ethernet64          136,137,138,139     100G   9100     rs   Ethernet9/1  PortChannel1010      up       up  QSFP28 or later         off
     Ethernet72              64,65,66,67     100G   9100     rs  Ethernet10/1  PortChannel1010      up       up  QSFP28 or later         off
     Ethernet80              56,57,58,59     100G   9100     rs  Ethernet11/1  PortChannel1012      up       up  QSFP28 or later         off
     Ethernet88              48,49,50,51     100G   9100     rs  Ethernet12/1  PortChannel1012      up       up  QSFP28 or later         off
     Ethernet96              40,41,42,43     100G   9100     rs  Ethernet13/1           routed      up       up  QSFP28 or later         off
    Ethernet104              32,33,34,35     100G   9100     rs  Ethernet14/1  PortChannel1016      up       up  QSFP28 or later         off
    Ethernet112              24,25,26,27     100G   9100     rs  Ethernet15/1  PortChannel1016      up       up  QSFP28 or later         off
    Ethernet120              16,17,18,19     100G   9100     rs  Ethernet16/1           routed      up       up  QSFP28 or later         off
    Ethernet128                8,9,10,11     100G   9100     rs  Ethernet17/1  PortChannel1020      up       up  QSFP28 or later         off
    Ethernet136                  0,1,2,3     100G   9100     rs  Ethernet18/1  PortChannel1020      up       up  QSFP28 or later         off
    Ethernet144              72,73,74,75     100G   9100     rs  Ethernet19/1           routed      up       up  QSFP28 or later         off
    Ethernet152              80,81,82,83     100G   9100     rs  Ethernet20/1           routed      up       up  QSFP28 or later         off
    Ethernet160              88,89,90,91     100G   9100     rs  Ethernet21/1           routed      up       up  QSFP28 or later         off
    Ethernet168              96,97,98,99     100G   9100     rs  Ethernet22/1           routed      up       up  QSFP28 or later         off
    Ethernet176          104,105,106,107     100G   9100     rs  Ethernet23/1           routed      up       up  QSFP28 or later         off
    Ethernet184          112,113,114,115     100G   9100     rs  Ethernet24/1           routed      up       up  QSFP28 or later         off
    Ethernet192          120,121,122,123     100G   9100     rs  Ethernet25/1           routed      up       up  QSFP28 or later         off
    Ethernet200          128,129,130,131     100G   9100     rs  Ethernet26/1           routed      up       up  QSFP28 or later         off
    Ethernet208          136,137,138,139     100G   9100     rs  Ethernet27/1           routed      up       up  QSFP28 or later         off
    Ethernet216              64,65,66,67     100G   9100     rs  Ethernet28/1           routed      up       up  QSFP28 or later         off
    Ethernet224              56,57,58,59     100G   9100     rs  Ethernet29/1           routed      up       up  QSFP28 or later         off
    Ethernet232              48,49,50,51     100G   9100     rs  Ethernet30/1           routed      up       up  QSFP28 or later         off
    Ethernet240              40,41,42,43     100G   9100     rs  Ethernet31/1           routed      up       up  QSFP28 or later         off
    Ethernet248              32,33,34,35     100G   9100     rs  Ethernet32/1           routed      up       up  QSFP28 or later         off
    Ethernet256  24,25,26,27,28,29,30,31     400G   9100    N/A  Ethernet33/1           routed    down     down  QSFP28 or later         off
    Ethernet264  16,17,18,19,20,21,22,23     400G   9100    N/A  Ethernet34/1           routed    down     down  QSFP28 or later         off
    Ethernet272    8,9,10,11,12,13,14,15     400G   9100    N/A  Ethernet35/1           routed    down     down  QSFP28 or later         off
    Ethernet280          0,1,2,3,4,5,6,7     400G   9100    N/A  Ethernet36/1           routed    down     down  QSFP28 or later         off
 PortChannel102                      N/A     200G   9100    N/A           N/A           routed      up       up              N/A         N/A
 PortChannel104                      N/A     200G   9100    N/A           N/A           routed      up       up              N/A         N/A
 PortChannel106                      N/A     200G   9100    N/A           N/A           routed      up       up              N/A         N/A
 PortChannel108                      N/A     200G   9100    N/A           N/A           routed      up       up              N/A         N/A
PortChannel1010                      N/A     200G   9100    N/A           N/A           routed      up       up              N/A         N/A
PortChannel1012                      N/A     200G   9100    N/A           N/A           routed      up       up              N/A         N/A
PortChannel1016                      N/A     200G   9100    N/A           N/A           routed      up       up              N/A         N/A
PortChannel1020                      N/A     200G   9100    N/A           N/A           routed      up       up              N/A         N/A
kenneth-arista commented 1 year ago

With https://github.com/sonic-net/sonic-buildimage/pull/14167 merged, this issue should be resolved.