search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
730 stars 1.4k forks source link

Portchannels are up, bgp sessions are up, but physical interfaces shown down in "show interface status" on t1 4600 with 20220531.18 image #14087

Closed ZhaohuiS closed 1 year ago

ZhaohuiS commented 1 year ago

Description

On 4600 T1 testbed, I load 20220531.17 or master image, all ports are up. Only on 20220531.18 image, portchannel interfaces are up, but Ethernet64, Ethernet 68, Ethernet244, etc, these interfaces are shown down. It’s just show problem, I checked in APPL_DB, they are oper down. BGP sessions are up, packets can be sent out. But this problem will impact sanity check, 202205 pipeline will fail at sanity check.

admin@str2-msn4600c-acs-01:~$ show interface status
      Interface            Lanes    Speed    MTU    FEC    Alias             Vlan    Oper    Admin             Type    Asym PFC
---------------  ---------------  -------  -----  -----  -------  ---------------  ------  -------  ---------------  ----------
      Ethernet0          0,1,2,3     100G   9100     rs     etp1   PortChannel102    down       up  QSFP28 or later         off
      Ethernet4        8,9,10,11     100G   9100     rs     etp2   PortChannel102    down       up  QSFP28 or later         off
      Ethernet8      16,17,18,19     100G   9100     rs     etp3           routed    down     down  QSFP28 or later         off
     Ethernet12      24,25,26,27     100G   9100     rs     etp4           routed    down     down  QSFP28 or later         off
     Ethernet16      32,33,34,35     100G   9100     rs     etp5   PortChannel105    down       up  QSFP28 or later         off
     Ethernet20      40,41,42,43     100G   9100     rs     etp6   PortChannel105    down       up  QSFP28 or later         off
     Ethernet24      48,49,50,51     100G   9100     rs     etp7           routed    down     down  QSFP28 or later         off
     Ethernet28      56,57,58,59     100G   9100     rs     etp8           routed    down     down  QSFP28 or later         off
     Ethernet32      64,65,66,67     100G   9100     rs     etp9           routed    down     down  QSFP28 or later         off
     Ethernet36      72,73,74,75     100G   9100     rs    etp10           routed    down     down  QSFP28 or later         off
     Ethernet40      80,81,82,83     100G   9100     rs    etp11           routed    down     down  QSFP28 or later         off
     Ethernet44      88,89,90,91     100G   9100     rs    etp12           routed    down     down  QSFP28 or later         off
     Ethernet48      96,97,98,99     100G   9100     rs    etp13           routed    down     down  QSFP28 or later         off
     Ethernet52  104,105,106,107     100G   9100     rs    etp14           routed    down     down  QSFP28 or later         off
     Ethernet56  112,113,114,115     100G   9100     rs    etp15           routed    down     down  QSFP28 or later         off
     Ethernet60  120,121,122,123     100G   9100     rs    etp16           routed    down     down  QSFP28 or later         off
     Ethernet64  128,129,130,131     100G   9100     rs    etp17   PortChannel108    down       up  QSFP28 or later         off
     Ethernet68  136,137,138,139     100G   9100     rs    etp18   PortChannel108    down       up  QSFP28 or later         off
     Ethernet72  144,145,146,147     100G   9100     rs    etp19           routed    down     down  QSFP28 or later         off
     Ethernet76  152,153,154,155     100G   9100     rs    etp20           routed    down     down  QSFP28 or later         off
     Ethernet80  160,161,162,163     100G   9100     rs    etp21  PortChannel1011    down       up  QSFP28 or later         off
     Ethernet84  168,169,170,171     100G   9100     rs    etp22  PortChannel1011    down       up  QSFP28 or later         off
     Ethernet88  176,177,178,179     100G   9100     rs    etp23           routed    down     down  QSFP28 or later         off
     Ethernet92  184,185,186,187     100G   9100     rs    etp24           routed    down     down  QSFP28 or later         off
     Ethernet96  192,193,194,195     100G   9100     rs    etp25           routed    down     down  QSFP28 or later         off
    Ethernet100  200,201,202,203     100G   9100     rs    etp26           routed    down     down  QSFP28 or later         off
    Ethernet104  208,209,210,211     100G   9100     rs    etp27           routed    down     down  QSFP28 or later         off
    Ethernet108  216,217,218,219     100G   9100     rs    etp28           routed    down     down  QSFP28 or later         off
    Ethernet112  224,225,226,227     100G   9100     rs    etp29           routed    down     down  QSFP28 or later         off
    Ethernet116  232,233,234,235     100G   9100     rs    etp30           routed    down     down  QSFP28 or later         off
    Ethernet120  240,241,242,243     100G   9100     rs    etp31           routed    down     down  QSFP28 or later         off
    Ethernet124  248,249,250,251     100G   9100     rs    etp32           routed    down     down  QSFP28 or later         off
    Ethernet128  256,257,258,259     100G   9100     rs    etp33           routed    down     down  QSFP28 or later         off
    Ethernet132  264,265,266,267     100G   9100     rs    etp34           routed    down     down  QSFP28 or later         off
    Ethernet136  272,273,274,275     100G   9100     rs    etp35   PortChannel101    down       up  QSFP28 or later         off
    Ethernet140  280,281,282,283     100G   9100     rs    etp36           routed    down     down  QSFP28 or later         off
    Ethernet144  288,289,290,291     100G   9100     rs    etp37   PortChannel103    down       up  QSFP28 or later         off
    Ethernet148  296,297,298,299     100G   9100     rs    etp38   PortChannel104    down       up  QSFP28 or later         off
    Ethernet152  304,305,306,307     100G   9100     rs    etp39   PortChannel106    down       up  QSFP28 or later         off
    Ethernet156  312,313,314,315     100G   9100     rs    etp40   PortChannel107    down       up  QSFP28 or later         off
    Ethernet160  320,321,322,323     100G   9100     rs    etp41           routed    down     down  QSFP28 or later         off
    Ethernet164  328,329,330,331     100G   9100     rs    etp42           routed    down     down  QSFP28 or later         off
    Ethernet168  336,337,338,339     100G   9100     rs    etp43   PortChannel109    down       up  QSFP28 or later         off
    Ethernet172  344,345,346,347     100G   9100     rs    etp44           routed    down     down  QSFP28 or later         off
    Ethernet176  352,353,354,355     100G   9100     rs    etp45  PortChannel1010    down       up  QSFP28 or later         off
    Ethernet180  360,361,362,363     100G   9100     rs    etp46  PortChannel1012      up       up  QSFP28 or later         off
    Ethernet184  368,369,370,371     100G   9100     rs    etp47  PortChannel1013    down       up  QSFP28 or later         off
    Ethernet188  376,377,378,379     100G   9100     rs    etp48  PortChannel1014    down       up  QSFP28 or later         off
    Ethernet192  384,385,386,387     100G   9100     rs    etp49           routed    down     down  QSFP28 or later         off
    Ethernet196  392,393,394,395     100G   9100     rs    etp50           routed    down     down  QSFP28 or later         off
    Ethernet200  400,401,402,403     100G   9100     rs    etp51  PortChannel1015    down       up  QSFP28 or later         off
    Ethernet204  408,409,410,411     100G   9100     rs    etp52           routed    down     down  QSFP28 or later         off
    Ethernet208  416,417,418,419     100G   9100     rs    etp53  PortChannel1016    down       up  QSFP28 or later         off
    Ethernet212  424,425,426,427     100G   9100     rs    etp54  PortChannel1017      up       up  QSFP28 or later         off
    Ethernet216  432,433,434,435     100G   9100     rs    etp55  PortChannel1018    down       up  QSFP28 or later         off
    Ethernet220  440,441,442,443     100G   9100     rs    etp56  PortChannel1019    down       up  QSFP28 or later         off
    Ethernet224  448,449,450,451     100G   9100     rs    etp57           routed    down     down  QSFP28 or later         off
    Ethernet228  456,457,458,459     100G   9100     rs    etp58           routed    down     down  QSFP28 or later         off
    Ethernet232  464,465,466,467     100G   9100     rs    etp59  PortChannel1020    down       up  QSFP28 or later         off
    Ethernet236  472,473,474,475     100G   9100     rs    etp60           routed    down     down  QSFP28 or later         off
    Ethernet240  480,481,482,483     100G   9100     rs    etp61  PortChannel1021    down       up  QSFP28 or later         off
    Ethernet244  488,489,490,491     100G   9100     rs    etp62  PortChannel1022    down       up  QSFP28 or later         off
    Ethernet248  496,497,498,499     100G   9100     rs    etp63  PortChannel1023    down       up  QSFP28 or later         off
    Ethernet252  504,505,506,507     100G   9100     rs    etp64  PortChannel1024    down       up  QSFP28 or later         off
PortChannel101              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel102              N/A     200G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel103              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel104              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel105              N/A     200G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel106              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel107              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel108              N/A     200G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel109              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1010              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1011              N/A     200G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1012              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1013              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1014              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1015              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1016              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1017              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1018              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1019              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1020              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1021              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1022              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1023              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel1024              N/A     100G   9100    N/A      N/A           routed      up       up              N/A         N/A
admin@str2-msn4600c-acs-01:~$ redis-cli -n 0 hgetall  "PORT_TABLE:Ethernet244" 
 1) "mtu"
2) "9100"
3) "admin_status"
4) "up"
5) "alias"
6) "etp62"
7) "description"
8) "ARISTA18T0:Ethernet1"
9) "fec"
10) "rs"
11) "index"
12) "62"
13) "lanes"
14) "488,489,490,491"
15) "pfc_asym"
16) "off"
17) "speed"
18) "100000"
19) "tpid"
20) "0x8100"
21) "oper_status"
22) "down"
admin@str2-msn4600c-acs-01:~$ redis-cli -n 6 hgetall  "PORT_TABLE|Ethernet244" 
 1) "state"
2) "ok"
3) "netdev_oper_status"
4) "up"
5) "admin_status"
6) "up"
7) "mtu"
8) "9100"
9) "supported_speeds"
10) "100000,50000,40000,25000,10000,1000"
11) "supported_fecs"
12) "fc,none,rs"
13) "host_tx_ready"
14) "true"
admin@str2-msn4600c-acs-01:~$ show interface portchannel 
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
       S - selected, D - deselected, * - not synced
  No.  Team Dev         Protocol     Ports
-----  ---------------  -----------  ---------------------------
  101  PortChannel101   LACP(A)(Up)  Ethernet136(S)
  102  PortChannel102   LACP(A)(Up)  Ethernet4(S) Ethernet0(S)
  103  PortChannel103   LACP(A)(Up)  Ethernet144(S)
  104  PortChannel104   LACP(A)(Up)  Ethernet148(S)
  105  PortChannel105   LACP(A)(Up)  Ethernet20(S) Ethernet16(S)
  106  PortChannel106   LACP(A)(Up)  Ethernet152(S)
  107  PortChannel107   LACP(A)(Up)  Ethernet156(S)
  108  PortChannel108   LACP(A)(Up)  Ethernet68(S) Ethernet64(S)
  109  PortChannel109   LACP(A)(Up)  Ethernet168(S)
1010  PortChannel1010  LACP(A)(Up)  Ethernet176(S)
1011  PortChannel1011  LACP(A)(Up)  Ethernet84(S) Ethernet80(S)
1012  PortChannel1012  LACP(A)(Up)  Ethernet180(S)
1013  PortChannel1013  LACP(A)(Up)  Ethernet184(S)
1014  PortChannel1014  LACP(A)(Up)  Ethernet188(S)
1015  PortChannel1015  LACP(A)(Up)  Ethernet200(S)
1016  PortChannel1016  LACP(A)(Up)  Ethernet208(S)
1017  PortChannel1017  LACP(A)(Up)  Ethernet212(S)
1018  PortChannel1018  LACP(A)(Up)  Ethernet216(S)
1019  PortChannel1019  LACP(A)(Up)  Ethernet220(S)
1020  PortChannel1020  LACP(A)(Up)  Ethernet232(S)
1021  PortChannel1021  LACP(A)(Up)  Ethernet240(S)
1022  PortChannel1022  LACP(A)(Up)  Ethernet244(S)
1023  PortChannel1023  LACP(A)(Up)  Ethernet248(S)
1024  PortChannel1024  LACP(A)(Up)  Ethernet252(S)

admin@str2-msn4600c-acs-01:~$ show ip bgp sum

IPv4 Unicast Summary:
BGP router identifier 10.1.0.32, local AS number 65100 vrf-id 0
BGP table version 13360
RIB entries 12851, using 2364584 bytes of memory
Peers 24, using 17779392 KiB of memory
Peer groups 4, using 256 bytes of memory

Neighbhor      V     AS    MsgRcvd    MsgSent    TblVer    InQ    OutQ  Up/Down      State/PfxRcd  NeighborName
-----------  ---  -----  ---------  ---------  --------  -----  ------  ---------  --------------  --------------
10.0.0.1       4  65200       9576         95         0      0       0  00:02:25             6370  ARISTA01T2
10.0.0.5       4  65200       6387         50         0      0       0  00:00:03             6370  ARISTA03T2
10.0.0.9       4  65200       6386         72         0      0       0  00:00:17             6370  ARISTA05T2
10.0.0.13      4  65200       3197         48         0      0       0  00:07:26             6370  ARISTA07T2
10.0.0.33      4  64001         18       6448         0      0       0  00:00:58                3  ARISTA01T0
10.0.0.35      4  64002         13       3242         0      0       0  00:07:26                3  ARISTA02T0
10.0.0.37      4  64003         14       3242         0      0       0  00:07:26                4  ARISTA03T0
10.0.0.39      4  64004         13       3242         0      0       0  00:07:26                3  ARISTA04T0
10.0.0.41      4  64005         14       3242         0      0       0  00:07:26                4  ARISTA05T0
10.0.0.43      4  64006         13       3242         0      0       0  00:07:26                3  ARISTA06T0
10.0.0.45      4  64007         18       6450         0      0       0  00:00:33                3  ARISTA07T0
10.0.0.47      4  64008         13       3242         0      0       0  00:07:26                3  ARISTA08T0
10.0.0.49      4  64009         13       3242         0      0       0  00:07:26                3  ARISTA09T0
10.0.0.51      4  64010         13       3242         0      0       0  00:07:26                3  ARISTA10T0
10.0.0.53      4  64011         18       6450         0      0       0  00:01:31                3  ARISTA11T0
10.0.0.55      4  64012         19       6449         0      0       0  00:01:12                3  ARISTA12T0
10.0.0.57      4  64013         13       3242         0      0       0  00:07:26                3  ARISTA13T0
10.0.0.59      4  64014         13       3242         0      0       0  00:07:26                3  ARISTA14T0
10.0.0.61      4  64015         13       3242         0      0       0  00:07:26                3  ARISTA15T0
10.0.0.63      4  64016         18       6453         0      0       0  00:00:48                3  ARISTA16T0
10.0.0.65      4  64017         16       6453         0      0       0  00:01:44                1  ARISTA17T0
10.0.0.67      4  64018         16       6450         0      0       0  00:02:00                1  ARISTA18T0
10.0.0.69      4  64019         20       6450         0      0       0  00:02:12                1  ARISTA19T0
10.0.0.71      4  64020         12       3242         0      0       0  00:07:26                1  ARISTA20T0

ar 3 06:26:32.581983 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet244 admin:1 oper:0 addr:1c:34:da:bb:83:00 ifindex:157 master:0 type:sx_netdev Mar 3 06:26:32.582145 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: Publish Ethernet244(ok:down) to state db Mar 3 06:26:32.714149 str2-msn4600c-acs-01 NOTICE swss#orchagent: :- processQueue: Set buffer queue Ethernet244:0-2 to BUFFER_PROFILE_TABLE:q_lossy_profile Mar 3 06:26:32.717340 str2-msn4600c-acs-01 NOTICE swss#orchagent: :- processQueue: Set buffer queue Ethernet244:3-4 to BUFFER_PROFILE_TABLE:egress_lossless_profile Mar 3 06:26:32.719287 str2-msn4600c-acs-01 NOTICE swss#orchagent: :- processQueue: Set buffer queue Ethernet244:5-6 to BUFFER_PROFILE_TABLE:q_lossy_profile Mar 3 06:26:33.049589 str2-msn4600c-acs-01 NOTICE swss#orchagent: :- processPriorityGroup: Set buffer PG Ethernet244:0 to BUFFER_PROFILE_TABLE:ingress_lossy_profile Mar 3 06:26:33.051462 str2-msn4600c-acs-01 NOTICE swss#orchagent: :- processPriorityGroup: Set buffer PG Ethernet244:3-4 to BUFFER_PROFILE_TABLE:pg_lossless_100000_300m_profile Mar 3 06:26:33.136331 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet244 admin:0 oper:0 addr:1c:34:da:bb:83:00 ifindex:157 master:0 type:sx_netdev Mar 3 06:26:33.136482 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: Publish Ethernet244(ok:down) to state db Mar 3 06:26:33.217566 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet244 admin:1 oper:0 addr:1c:34:da:bb:83:00 ifindex:157 master:0 type:sx_netdev Mar 3 06:26:33.217790 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: Publish Ethernet244(ok:down) to state db Mar 3 06:26:33.218645 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet244 admin:1 oper:0 addr:1c:34:da:bb:83:00 ifindex:157 master:18 type:sx_netdev Mar 3 06:26:33.219083 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: Publish Ethernet244(ok:down) to state db Mar 3 06:26:33.219083 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet244 admin:1 oper:0 addr:1c:34:da:bb:83:00 ifindex:157 master:18 type:sx_netdev Mar 3 06:26:33.219249 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: Publish Ethernet244(ok:down) to state db Mar 3 06:26:33.220865 str2-msn4600c-acs-01 INFO kernel: [ 85.694095] 8021q: adding VLAN 0 to HW filter on device Ethernet244 Mar 3 06:26:33.220877 str2-msn4600c-acs-01 INFO kernel: [ 85.694492] PortChannel1022: Port device Ethernet244 added Mar 3 06:26:33.222583 str2-msn4600c-acs-01 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet244 to port channel PortChannel1022 Mar 3 06:26:33.223864 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet244 admin:1 oper:0 addr:1c:34:da:bb:83:00 ifindex:157 master:18 type:sx_netdev Mar 3 06:26:33.224148 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: Publish Ethernet244(ok:down) to state db Mar 3 06:26:33.224181 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet244 admin:1 oper:0 addr:1c:34:da:bb:83:00 ifindex:157 master:18 type:sx_netdev Mar 3 06:26:33.224802 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: Publish Ethernet244(ok:down) to state db Mar 3 06:26:33.414683 str2-msn4600c-acs-01 NOTICE swss#orchagent: :- doPortTask: Set port Ethernet244 speed from 0 to 100000 Mar 3 06:26:33.415592 str2-msn4600c-acs-01 NOTICE swss#orchagent: :- doPortTask: Set port Ethernet244 MTU to 9100 Mar 3 06:26:33.419682 str2-msn4600c-acs-01 NOTICE swss#orchagent: :- setPortFec: Set port Ethernet244 FEC mode rs Mar 3 06:26:33.419682 str2-msn4600c-acs-01 NOTICE swss#orchagent: :- doPortTask: Set port Ethernet244 asymmetric PFC to off Mar 3 06:26:33.421323 str2-msn4600c-acs-01 NOTICE swss#orchagent: :- doPortTask: Set port Ethernet244 admin status to up Mar 3 06:26:39.872924 str2-msn4600c-acs-01 INFO kernel: [ 92.345239] IPv6: ADDRCONF(NETDEV_CHANGE): Ethernet244: link becomes ready Mar 3 06:26:39.873328 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet244 admin:1 oper:1 addr:1c:34:da:bb:83:00 ifindex:157 master:18 type:sx_netdev Mar 3 06:26:39.873705 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: Publish Ethernet244(ok:up) to state db Mar 3 06:26:39.873705 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet244 admin:1 oper:1 addr:1c:34:da:bb:83:00 ifindex:157 master:18 type:sx_netdev Mar 3 06:26:39.873948 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: Publish Ethernet244(ok:up) to state db Mar 3 06:26:42.709566 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet244 admin:1 oper:1 addr:1c:34:da:bb:83:00 ifindex:157 master:18 type:sx_netdev Mar 3 06:26:42.710295 str2-msn4600c-acs-01 NOTICE swss#portsyncd: :- onMsg: Publish Ethernet244(ok:up) to state db Mar 3 06:27:35.539512 str2-msn4600c-acs-01 NOTICE swss#orchagent: :- addLagMember: Add member Ethernet244 to LAG PortChannel1022 lid:2000000000b2b pid:1000000000581 Mar 3 06:27:35.541370 str2-msn4600c-acs-01 NOTICE swss#orchagent: :- setCollectionOnLagMember: Enable collection on LAG member Ethernet244 Mar 3 06:27:35.543601 str2-msn4600c-acs-01 NOTICE swss#orchagent: :- setDistributionOnLagMember: Enable distribution on LAG member Ethernet244

Steps to reproduce the issue:

  1. Load 20220531.18 image
  2. Check the output of show interface status

Describe the results you received:

Describe the results you expected:

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

ppikh commented 1 year ago

Hi @ZhaohuiS Could you please share link to SONiC image or image hash?

ZhaohuiS commented 1 year ago

@ppikh Is show version enough?

admin@str2-msn4600c-acs-01:~$ show version

SONiC Software Version: SONiC.20220531.18 Distribution: Debian 11.6 Kernel: 5.10.0-18-2-amd64 Build commit: 8c25164188 Build date: Wed Mar 1 19:39:19 UTC 2023 Built by: cloudtest@80f288abc000000

Platform: x86_64-mlnx_msn4600c-r0 HwSKU: ACS-MSN4600C ASIC: mellanox ASIC Count: 1 Serial Number: MT2115X26246 Model Number: MSN4600-CS2RO Hardware Revision: A1 Uptime: 09:19:18 up 5:03, 1 user, load average: 1.43, 1.39, 1.34 Date: Tue 07 Mar 2023 09:19:18

Docker images: REPOSITORY TAG IMAGE ID SIZE docker-platform-monitor 20220531.18 319185470dc1 909MB docker-platform-monitor latest 319185470dc1 909MB docker-syncd-mlnx 20220531.18 867c68303ac2 904MB docker-syncd-mlnx latest 867c68303ac2 904MB docker-orchagent 20220531.18 79542038ecd2 519MB docker-orchagent latest 79542038ecd2 519MB docker-fpm-frr 20220531.18 4a642430a68f 530MB docker-fpm-frr latest 4a642430a68f 530MB docker-teamd 20220531.18 db7a3c4e6901 501MB docker-teamd latest db7a3c4e6901 501MB docker-macsec 20220531.18 4bf5bbd75049 503MB docker-dhcp-relay 20220531.18 f51f4977e2fa 497MB docker-snmp 20220531.18 bcbb63a4bde1 530MB docker-snmp latest bcbb63a4bde1 530MB docker-sonic-telemetry 20220531.18 11a7dc21a013 565MB docker-sonic-telemetry latest 11a7dc21a013 565MB docker-router-advertiser 20220531.18 0185d6fe5abe 485MB docker-router-advertiser latest 0185d6fe5abe 485MB docker-mux 20220531.18 8800ee38561e 533MB docker-mux latest 8800ee38561e 533MB docker-lldp 20220531.18 0e761fa316b7 527MB docker-lldp latest 0e761fa316b7 527MB docker-database 20220531.18 7d8a537b66a9 485MB docker-database latest 7d8a537b66a9 485MB docker-acms 20220531.18 1072136ee328 532MB docker-acms latest 1072136ee328 532MB k8s.gcr.io/pause 3.5 ed210e3e4a5b 683kB

ppikh commented 1 year ago

Hi @ZhaohuiS I am trying to reproduce this issue locally. But I'm not able to find image .bin file in https://sonic-build.azurewebsites.net/ui/sonic/pipelines/139/builds?branchName=202205.

Could you please provide link to image or to build job from where I can download this image? Thanks

ZhaohuiS commented 1 year ago

@ppikh Okay, I found the commit id is not in the list. The image should be built internally. Could you please try the 20220531.18 image which built on Mar 1st? image

stepanblyschak commented 1 year ago

Orchagent unsuccessfully tries creating ACL table in a loop:

Mar 7 11:18:15.673886 r-tigon-11 ERR syncd#SDK: [SAI_ACL.ERR] mlnx_sai_acl.c[13937]- mlnx_create_acl_table: Failed to create flex key - Internal Error. Mar 7 11:18:15.673886 r-tigon-11 ERR syncd#SDK: :- sendApiResponse: api SAI_COMMON_API_CREATE failed in syncd mode: SAI_STATUS_FAILURE Mar 7 11:18:15.682426 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_ACL_BIND_POINT_TYPE_LIST: 2:SAI_ACL_BIND_POINT_TYPE_PORT,SAI_ACL_BIND_POINT_TYPE_LAG Mar 7 11:18:15.682426 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_FIELD_SRC_IPV6: true Mar 7 11:18:15.682505 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_FIELD_DST_IPV6: true Mar 7 11:18:15.682505 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_FIELD_IN_PORTS: true Mar 7 11:18:15.682505 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_FIELD_OUTER_VLAN_ID: true Mar 7 11:18:15.682505 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_FIELD_L4_SRC_PORT: true Mar 7 11:18:15.682546 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_FIELD_L4_DST_PORT: true Mar 7 11:18:15.682546 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_FIELD_DSCP: true Mar 7 11:18:15.682574 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_FIELD_TCP_FLAGS: true Mar 7 11:18:15.682641 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_FIELD_ACL_IP_TYPE: true Mar 7 11:18:15.682641 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_FIELD_ICMPV6_TYPE: true Mar 7 11:18:15.682641 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_FIELD_ICMPV6_CODE: true Mar 7 11:18:15.682641 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_FIELD_ACL_RANGE_TYPE: 2:SAI_ACL_RANGE_TYPE_L4_SRC_PORT_RANGE,SAI_ACL_RANGE_TYPE_L4_DST_PORT_RANGE Mar 7 11:18:15.682641 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_FIELD_IPV6_NEXT_HEADER: true Mar 7 11:18:15.682724 r-tigon-11 ERR syncd#SDK: :- processQuadEvent: attr: SAI_ACL_TABLE_ATTR_ACL_STAGE: SAI_ACL_STAGE_INGRESS Mar 7 11:18:15.682775 r-tigon-11 ERR swss#orchagent: :- create: create status: SAI_STATUS_FAILURE Mar 7 11:18:15.682775 r-tigon-11 ERR swss#orchagent: :- addAclTable: Failed to create ACL table EVERFLOWV6

Since it does it in an endless loop, orchagent has no chance of handling oper status update from SAI, therefore oper status is out of sync for some interfaces. The relevant ACL change has already been reverted - https://github.com/sonic-net/sonic-swss/pull/2687

liat-grozovik commented 1 year ago

as of the above, the issue is not relevant. it is now closed please retest once you have an image with the reverted PR.