search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
724 stars 1.38k forks source link

[z9332]orchagent crash observed when tunnel termination is not supported (TH3) #5530

Open srideepDell opened 3 years ago

srideepDell commented 3 years ago

Description:

Below logs is seen with errors and syncd sends shutdown notification to orchagent which creates crash of orchagent. This issue is seen only when BCM SAI is changed to 3.7.4, issue not seen with 3.7.3

Platform :

DellEMC-Z9332f-O32 with T1 config loaded

Steps to reproduce:

  1. Install latest master build on z9332
  2. System is configured with T1 config
  3. OA crash is seen with sai errors.

drivshell>bsv bsv BRCM SAI ver: [3.7.4.2], OCP SAI ver: [1.6.0], SDK ver: [6.5.16]

syslogs with errors

Oct 2 16:20:43.711422 sonic ERR syncd#syncd: [none] brcm_sai_create_tunnel_term_table_entry:1811 P2MP tunnel terminator not supported on this platform. Oct 2 16:20:43.711496 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_TERM_TABLE_ENTRY_ATTR_VR_ID: oid:0x3000000000024 Oct 2 16:20:43.711528 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_TERM_TABLE_ENTRY_ATTR_TYPE: SAI_TUNNEL_TERM_TABLE_ENTRY_TYPE_P2MP Oct 2 16:20:43.711554 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_TERM_TABLE_ENTRY_ATTR_TUNNEL_TYPE: SAI_TUNNEL_TYPE_IPINIP Oct 2 16:20:43.711579 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_TERM_TABLE_ENTRY_ATTR_ACTION_TUNNEL_ID: oid:0x2a00000000055a Oct 2 16:20:43.711604 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_TERM_TABLE_ENTRY_ATTR_DST_IP: 10.0.0.0 Oct 2 16:20:43.711627 sonic ERR syncd#syncd: :- processQuadEvent: failed to execute api: create, key: SAI_OBJECT_TYPE_TUNNEL_TERM_TABLE_ENTRY:oid:0x2b00000000055b, status: SAI_STATUS_NOT_SUPPORTED Oct 2 16:20:43.711663 sonic ERR syncd#syncd: :- run: Runtime error: :- processQuadEvent: failed to execute api: create, key: SAI_OBJECT_TYPE_TUNNEL_TERM_TABLE_ENTRY:oid:0x2b00000000055b, status: SAI_STATUS_NOT_SUPPORTED Oct 2 16:20:43.711663 sonic NOTICE syncd#syncd: :- sendShutdownRequest: sending switch_shutdown_request notification to OA for switch: oid:0x21000000000000 Oct 2 16:20:43.711787 sonic NOTICE syncd#syncd: :- sendShutdownRequestAfterException: notification send successfull

drivshell>Attach Attach Attach: Unit 0 (BCM56980_B0): attached (current unit)

drivshell>bsv bsv BRCM SAI ver: [3.7.4.2], OCP SAI ver: [1.6.0], SDK ver: [6.5.16]

Syslog

syslog.log

srideepDell commented 3 years ago

Issue seen with 436 Image which has latest SDK and SA I versions

root@sonic:~# root@sonic:~# bcmcmd "bsv" bsv BRCM SAI ver: [4.2.1.3], OCP SAI ver: [1.6.5], SDK ver: [sdk-6.5.19] drivshell> root@sonic:~#

SONiC Software Version: SONiC.master.436-d03de95e Distribution: Debian 10.6 Kernel: 4.19.0-9-2-amd64 Build commit: d03de95e Build date: Wed Oct 7 17:20:56 UTC 2020 Built by: johnar@jenkins-worker-11

Platform: x86_64-dellemc_z9332f_d1508-r0 HwSKU: DellEMC-Z9332f-O32 ASIC: broadcom Serial Number: TH0J5HDGCET009740036A00 Uptime: 05:03:35 up 2 min, 1 user, load average: 0.53, 0.49, 0.21

Oct 12 05:01:30.858087 sonic NOTICE swss#portsyncd: :- main: PortInitDone
Oct 12 05:01:30.889213 sonic WARNING syncd#syncd: [0] SAI_API_HOSTIF:_brcm_sai_set_cpu_queue_shaper:11846 Set CPU Queue 0 shaping: cir 600, cbs 307
Oct 12 05:01:30.911396 sonic WARNING syncd#syncd: [0] SAI_API_HOSTIF:_brcm_sai_set_cpu_queue_shaper:11846 Set CPU Queue 1 shaping: cir 6000, cbs 3072
Oct 12 05:01:30.911975 sonic ERR syncd#syncd: [0] SAI_API_TUNNEL:brcm_sai_create_tunnel_term_table_entry:1848 P2MP tunnel terminator not supported on this platform.
Oct 12 05:01:30.912012 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_TERM_TABLE_ENTRY_ATTR_VR_ID: oid:0x3000000000024
Oct 12 05:01:30.912040 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_TERM_TABLE_ENTRY_ATTR_TYPE: SAI_TUNNEL_TERM_TABLE_ENTRY_TYPE_P2MP
Oct 12 05:01:30.912068 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_TERM_TABLE_ENTRY_ATTR_TUNNEL_TYPE: SAI_TUNNEL_TYPE_IPINIP
Oct 12 05:01:30.912095 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_TERM_TABLE_ENTRY_ATTR_ACTION_TUNNEL_ID: oid:0x2a00000000054e
Oct 12 05:01:30.912122 sonic ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_TERM_TABLE_ENTRY_ATTR_DST_IP: 10.0.0.0
Oct 12 05:01:30.912169 sonic ERR syncd#syncd: :- processQuadEvent: failed to execute api: create, key: SAI_OBJECT_TYPE_TUNNEL_TERM_TABLE_ENTRY:oid:0x2b00000000054f, status: SAI_STATUS_NOT_SUPPORTED
Oct 12 05:01:30.912223 sonic ERR syncd#syncd: :- run: Runtime error: :- processQuadEvent: failed to execute api: create, key: SAI_OBJECT_TYPE_TUNNEL_TERM_TABLE_ENTRY:oid:0x2b00000000054f, status: SAI_STATUS_NOT_SUPPORTED
Oct 12 05:01:30.912374 sonic NOTICE syncd#syncd: :- sendShutdownRequest: sending switch_shutdown_request notification to OA for switch: oid:0x21000000000000
Oct 12 05:01:30.913094 sonic NOTICE syncd#syncd: :- sendShutdownRequestAfterException: notification send successfull
gechiang commented 3 years ago

@srideepDell Please use an image that has the following PR in place to see if you still encounter this issue... https://github.com/Azure/sonic-buildimage/pull/8395