search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
741 stars 1.44k forks source link

[202405][VOQ] SAI error in speed change #20261

Open ysmanman opened 2 months ago

ysmanman commented 2 months ago

Description

We noticed the following SAI error in addPortBulk with 202405 SONiC image (SAI 11.2). The error was seen only when we configured both 100G and 400G ports on the same ASIC. 

2024 Sep  3 23:27:59.648366 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_SYSTEM_PORT:_brcm_sai_dnx_e2e_scheme_sysport_create:18843 system port create and connect failed with error Invalid port (0xffffffee).
2024 Sep  3 23:27:59.648366 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_SYSTEM_PORT:_brcm_sai_dnx_add_system_port:18930 e2e scheme sysport create failed with error -9.
2024 Sep  3 23:27:59.648495 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:_brcm_sai_dnx_dyn_port_create:19536 dyn system port add failed, lane:0 rv:-9
2024 Sep  3 23:27:59.648553 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:brcm_sai_create_port:9484 dyn port create failed with error -9.
2024 Sep  3 23:27:59.648666 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:_brcm_sai_dnx_dyn_port_add:19351 BCM port add failed with error Invalid port (0xffffffee).
2024 Sep  3 23:27:59.648713 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:_brcm_sai_dnx_dyn_port_create:19512 dyn port add failed, lane:16 rv:-9
2024 Sep  3 23:27:59.649025 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:brcm_sai_create_port:9484 dyn port create failed with error -9.
2024 Sep  3 23:27:59.649072 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:_brcm_sai_dnx_dyn_port_add:19351 BCM port add failed with error Invalid port (0xffffffee).
2024 Sep  3 23:27:59.649112 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:_brcm_sai_dnx_dyn_port_create:19512 dyn port add failed, lane:40 rv:-9
2024 Sep  3 23:27:59.649286 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:brcm_sai_create_port:9484 dyn port create failed with error -9.
2024 Sep  3 23:27:59.649475 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:_brcm_sai_dnx_dyn_port_add:19351 BCM port add failed with error Invalid port (0xffffffee).
2024 Sep  3 23:27:59.649475 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:_brcm_sai_dnx_dyn_port_create:19512 dyn port add failed, lane:64 rv:-9
2024 Sep  3 23:27:59.649736 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:brcm_sai_create_port:9484 dyn port create failed with error -9.
2024 Sep  3 23:27:59.649736 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:_brcm_sai_dnx_dyn_port_add:19351 BCM port add failed with error Invalid port (0xffffffee).
2024 Sep  3 23:27:59.649736 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:_brcm_sai_dnx_dyn_port_create:19512 dyn port add failed, lane:96 rv:-9
2024 Sep  3 23:27:59.649736 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:brcm_sai_create_port:9484 dyn port create failed with error -9.
2024 Sep  3 23:27:59.649736 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:_brcm_sai_dnx_dyn_port_add:19351 BCM port add failed with error Invalid port (0xffffffee).
2024 Sep  3 23:27:59.649753 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:_brcm_sai_dnx_dyn_port_create:19512 dyn port add failed, lane:104 rv:-9
2024 Sep  3 23:27:59.649782 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:brcm_sai_create_port:9484 dyn port create failed with error -9.
2024 Sep  3 23:27:59.650031 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:_brcm_sai_dnx_dyn_port_add:19351 BCM port add failed with error Invalid port (0xffffffee).
2024 Sep  3 23:27:59.650075 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:_brcm_sai_dnx_dyn_port_create:19512 dyn port add failed, lane:136 rv:-9
2024 Sep  3 23:27:59.650340 cmp329-6 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:brcm_sai_create_port:9484 dyn port create failed with error -9.
2024 Sep  3 23:27:59.650388 cmp329-6 ERR swss0#orchagent: :- addPortBulk: Failed to create ports with bulk operation, rv:-1
2024 Sep  3 23:27:59.650603 cmp329-6 ERR swss0#orchagent: :- handleSaiCreateStatus: Encountered failure in create operation, exiting orchagent, SAI API: SAI_API_PORT, status: SAI_STATUS_FAILURE
2024 Sep  3 23:27:59.650775 cmp329-6 NOTICE swss1#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet200 admin:0 oper:0 addr:02:42:f0:7f:01:02 ifindex:137 master:0
2024 Sep  3 23:27:59.651010 cmp329-6 ERR syncd0#syncd: :- sendApiResponse: api SAI_COMMON_API_BULK_CREATE failed in syncd mode: SAI_STATUS_FAILURE

Steps to reproduce the issue:

Describe the results you received:

Describe the results you expected:

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

ysmanman commented 2 months ago

add @kenneth-arista @arlakshm for viz.

ysmanman commented 2 months ago

CSP CS00012367997 was filed to track the issue.

bingwang-ms commented 2 months ago

Issue triaged in CSP