Open chitra-raghavan opened 3 years ago
@yxieca fyi
This is a know issue from Broadcom SDK.
@yxieca If this is a known issue with BRCM SDK should we pursuit the issue with BRCM to get a fix or this is not fixable by BRCM in which case we should go ahead make changes in testcase to not expect the exact address being reported in syslog?
@chitra-raghavan I just checked the test run results on TH3 for test_ser and they are passing for us. Perhaps some recent changes already addressed the issue? Can you please try again and see if the issue still present? Thanks!
@gechiang , the issue is still seen . These entries are mapped into "cannot pass" list . https://github.com/Azure/sonic-mgmt/blob/master/tests/platform_tests/broadcom/files/ser_injector.py#L274
root@sonic-10429:~# show ver
SONiC Software Version: SONiC.202012.27910-ada56abe6
Distribution: Debian 10.10
Kernel: 4.19.0-12-2-amd64
Build commit: ada56abe6
Build date: Sun Aug 8 16:53:21 UTC 2021
Built by: AzDevOps@sonic-build-workers-000KUH
Platform: x86_64-dellemc_z9332f_d1508-r0
HwSKU: DellEMC-Z9332f-M-O16C64
ASIC: broadcom
ASIC Count: 1
Sample entry1
root@sonic-10429:~# bcmcmd "ser inject memory=L2_ENTRY.ipipe0"
ser inject memory=L2_ENTRY.ipipe0
Error injected on L2_ENTRY.ipipe0 at index 0 pipe_x
drivshell>
root@sonic-10429:~# bcmcmd "list L2_ENTRY.ipipe0" | grep list -A5
list L2_ENTRY.ipipe0
Memory: L2X.ipipe0 aka L2_ENTRY alias L2X address 0x0e6c0000
Flags: valid cachable(on) hashed multiview
Blocks: ipipe0/dma/slam (1 copy, 1 dmaable, 1 slamable)
Entries: 8192 with indices 0-8191 (0x0-0x1fff), each 13 bytes 4 words
Entry mask: -1 -1 -1 0x00000007
root@sonic-10429:~#
syslog:
Aug 11 07:51:00.338319 sonic-10429 CRIT syncd#syncd: [none] SAI_API_SWITCH:_brcm_sai_switch_event_cb:498 5902592 Received switch event 2 on unit 0: 6 4 0
Aug 11 07:51:00.338364 sonic-10429 INFO syncd#/supervisord: syncd 0:soc_ser_correction:
Aug 11 07:51:00.338398 sonic-10429 CRIT syncd#syncd: [none] SAI_API_SWITCH:_brcm_sai_switch_event_cb:498 5902592 Received switch event 2 on unit 0: 2 4000001 e680000
Aug 11 07:51:00.338434 sonic-10429 INFO syncd#/supervisord: syncd SER_CORRECTION: reg/mem:9141 btype:17 sblk:1 at:-1 stage:3 addr:0x0e680000 port: 0 index: 0#015
Aug 11 07:51:00.338468 sonic-10429 INFO syncd#/supervisord: syncd 0:_soc_ser_mem_correction: mem: 9141=L2_ENTRY_ECC blkoffset:101#015
Aug 11 07:51:00.338504 sonic-10429 INFO syncd#/supervisord: syncd 0:_soc_ser_sram_correction: CLEAR_RESTORE: L2_ENTRY_ECC[9141] start_index: 0#015
Aug 11 07:51:00.338547 sonic-10429 INFO syncd#/supervisord: syncd 0:soc_tomahawk_process_ser_fifo: #015
Aug 11 07:51:00.338580 sonic-10429 INFO syncd#/supervisord: syncd Unit: 0 #015
Aug 11 07:51:00.338614 sonic-10429 INFO syncd#/supervisord: syncd 0:soc_tomahawk_process_ser_fifo: Mem: 0:soc_tomahawk_process_ser_fifo: Double or Multiple bit ECC error..#015
Aug 11 07:51:00.338676 sonic-10429 INFO syncd#/supervisord: syncd 0:_soc_tomahawk_print_ser_fifo_details: Error in: SBUS transaction.#015
Aug 11 07:51:00.338710 sonic-10429 INFO syncd#/supervisord: syncd 0:_soc_tomahawk_print_ser_fifo_details: Blk: 1, Pipe: 0, Address: 0x0e680000, base: 0x9a, stage: 3, index: 0#015
Aug 11 07:51:00.338743 sonic-10429 CRIT syncd#syncd: [none] SAI_API_SWITCH:_brcm_sai_switch_event_cb:498 5902592 Received switch event 2 on unit 0: 5 23a5 0
Aug 11 07:51:00.338776 sonic-10429 CRIT syncd#syncd: [none] SAI_API_SWITCH:_brcm_sai_switch_event_cb:498 5902592 Received switch event 2 on unit 0: 6 4 0
Aug 11 07:51:00.338808 sonic-10429 INFO syncd#/supervisord: syncd 0:soc_ser_correction: SER_CORRECTION: reg/mem:9141 btype:17 sblk:1 at:-1 stage:3 addr:0x0e680000 port: 0 index: 0#015
Aug 11 07:51:00.338841 sonic-10429 INFO syncd#/supervisord: syncd 0:_soc_ser_mem_correction: mem: 9141=L2_ENTRY_ECC blkoffset:101#015
Aug 11 07:51:00.338875 sonic-10429 INFO syncd#/supervisord: syncd 0:_soc_ser_sram_correction: CLEAR_RESTORE: L2_ENTRY_ECC[9141] start_index: 0#015
Sample entry2:
root@sonic-10429:~# bcmcmd "ser inject memory=L3_DEFIP_LEVEL1.ipipe0"
ser inject memory=L3_DEFIP_LEVEL1.ipipe0
Error injected on L3_DEFIP_LEVEL1.ipipe0 at index 0 pipe_x
drivshell>
root@sonic-10429:~# bcmcmd "list L3_DEFIP_LEVEL1.ipipe0"
list L3_DEFIP_LEVEL1.ipipe0
Memory: L3_DEFIP_LEVEL1.ipipe0 address 0x0e980000
Flags: valid cbp cachable(on) bist-epic
Blocks: ipipe0 (1 copy)
Entries: 1024 with indices 0-1023 (0x0-0x3ff), each 54 bytes 14 words
Entry mask: -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 0x0000ffff
Description: L3_DEFIP_LEVEL1 table.
syslog :
Aug 11 07:54:57.001863 sonic-10429 CRIT syncd#syncd: [none] SAI_API_SWITCH:_brcm_sai_switch_event_cb:498 5902592 Received switch event 2 on unit 0: 1 4000001 e880400
Aug 11 07:54:57.001863 sonic-10429 INFO syncd#/supervisord: syncd 0:soc_tomahawk_process_ser_fifo: #015
Aug 11 07:54:57.001863 sonic-10429 INFO syncd#/supervisord: syncd Unit: 0 #015
Aug 11 07:54:57.003166 sonic-10429 CRIT syncd#syncd: [none] SAI_API_SWITCH:_brcm_sai_switch_event_cb:498 5902592 Received switch event 2 on unit 0: 5 2424 400
Aug 11 07:54:57.003166 sonic-10429 CRIT syncd#syncd: [none] SAI_API_SWITCH:_brcm_sai_switch_event_cb:498 5902592 Received switch event 2 on unit 0: 6 5 0
Aug 11 07:54:57.003166 sonic-10429 INFO syncd#/supervisord: syncd 0:soc_tomahawk_process_ser_fifo: Multiple: 0:soc_tomahawk_process_ser_fifo: Mem: 0:soc_tomahawk_process_ser_fifo: Parity error..#015
Aug 11 07:54:57.003166 sonic-10429 INFO syncd#/supervisord: syncd 0:_soc_tomahawk_print_ser_fifo_details: Error in: transaction - refresh, aging etc.#015
Aug 11 07:54:57.003166 sonic-10429 INFO syncd#/supervisord: syncd 0:_soc_tomahawk_print_ser_fifo_details: Blk: 1, Pipe: 0, Address: 0x0e880400, base: 0xa2, stage: 3, index: 1024#015
Aug 11 07:54:57.003166 sonic-10429 INFO syncd#/supervisord: syncd 0:soc_ser_correction: SER_CORRECTION: reg/mem:9252 btype:17 sblk:1 at:-1 stage:3 addr:0x0e880000 port: 0 index: 1024#015
Aug 11 07:54:57.003166 sonic-10429 INFO syncd#/supervisord: syncd 0:_soc_ser_mem_correction: mem: 9252=L3_DEFIP_TCAM_LEVEL1 blkoffset:101#015
Aug 11 07:54:57.003166 sonic-10429 INFO syncd#/supervisord: syncd 0:_soc_ser_recovery_hw_cache: RESTORE pipe 0 [from pipe 1]: L3_DEFIP_TCAM_LEVEL1[9252] blk: ipipe0 index: 1024#015
@chitra-raghavan Thanks for trying it again. Just spoke with @yxieca about this and he told me that there are some prep work required to clean up this test for TH3 platform. He will be sharing a document with me on how to do that and once this prep work is done we should be able to pass this test. I will update this case when that is complete. For now We will keep this issue at lower priority as this is indeed a known inefficiency issue with the BRCM platform.
Description
Platform : Z9332 ASIC : Th3
Script : https://github.com/Azure/sonic-mgmt/blob/master/tests/platform_tests/broadcom/test_ser.py When Ser is injected to a memory , the correction syslog event shows a different address. Hence script fails as the memory address are not similar.
Command :
Memory address for L3_DEFIP_LEVEL1.ipipe0 : Memory corresponding to L3_DEFIP_LEVEL1.ipipe0 is 0x0e980000
syslog: In syslog, it is shown as different address ( Address: 0x0e880400,)
Steps to reproduce the issue:
Describe the results you received:
Describe the results you expected: When Ser is injected to a memory , the correction syslog event shows a the same address.
Additional information you deem important (e.g. issue happens only occasionally):
Output of
show version
:Attach debug file
sudo generate_dump
: unable to upload show tech support.