search

sonic-net / SONiC

Landing page for Software for Open Networking in the Cloud (SONiC) - https://sonic-net.github.io/SONiC/
2.22k stars 1.11k forks source link

[SONiC Broadcom AS7816-64X]: Orchagent and syncd crash seen when Asym PFC is enabled in an interface #481

Open afzalashfak opened 4 years ago

afzalashfak commented 4 years ago

I am using AS7816-64X switch. SONiC version is 20190924. Orchagent and syncd crash seen when Asym PFC is enabled in an interface.

CLI: pfc config asymmetric Ethernet0 on

Crash Logs:

orchagent.1478758153.76.core.gz

Nov 10 06:09:13.270432 sonic NOTICE swss#orchagent: :- doPortTask: Set port Ethernet0 asymmetric PFC to on Nov 10 06:09:13.270488 sonic INFO lldp#lldpmgrd: Port name Ethernet0 oper status: up Nov 10 06:09:13.271436 sonic ERR syncd#syncd: brcm_sai_set_port_attribute:1027 Unknown port attribute 74 passed Nov 10 06:09:13.271436 sonic ERR syncd#syncd: :- processEvent: VID: oid:0x1000000000012 RID: oid:0x100000022 Nov 10 06:09:13.271629 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description Nov 10 06:09:13.271744 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1' Nov 10 06:09:13.272216 sonic ERR syncd#syncd: :- processEvent: attr: SAI_PORT_ATTR_PRIORITY_FLOW_CONTROL_MODE: SAI_PORT_PRIORITY_FLOW_CONTROL_MODE_SEPARATE Nov 10 06:09:13.272216 sonic ERR syncd#syncd: :- processEvent: failed to execute api: set, key: SAI_OBJECT_TYPE_PORT:oid:0x1000000000012, status: SAI_STATUS_ATTR_NOT_SUPPORTED_0 Nov 10 06:09:13.272216 sonic ERR syncd#syncd: :- syncd_main: Runtime error: :- processEvent: failed to execute api: set, key: SAI_OBJECT_TYPE_PORT:oid:0x1000000000012, status: SAI_STATUS_ATTR_NOT_SUPPORTED_0 Nov 10 06:09:13.272216 sonic NOTICE syncd#syncd: :- notify_OA_about_syncd_exception: sending switch_shutdown_request notification to OA Nov 10 06:09:13.272508 sonic NOTICE swss#orchagent: :- handle_switch_shutdown_request: switch shutdown request Nov 10 06:09:13.272734 sonic NOTICE swss#portmgrd: :- doTask: Configure Ethernet0 MTU to 9100 Nov 10 06:09:13.272734 sonic NOTICE syncd#syncd: :- notify_OA_about_syncd_exception: notification send successfull Nov 10 06:09:13.275208 sonic INFO swss#supervisord: orchagent terminate called after throwing an instance of 'std::invalid_argument' Nov 10 06:09:13.275208 sonic INFO swss#supervisord: orchagent what(): parse error - unexpected end of input Nov 10 06:09:13.276925 sonic NOTICE swss#portmgrd: :- doTask: Configure Ethernet0 admin status to up Nov 10 06:09:13.283521 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description Nov 10 06:09:13.283521 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1' Nov 10 06:09:13.292829 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description Nov 10 06:09:13.292874 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1' Nov 10 06:09:13.301471 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description Nov 10 06:09:13.301498 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1' Nov 10 06:09:13.309988 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description Nov 10 06:09:13.309988 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1' Nov 10 06:09:13.317396 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description Nov 10 06:09:13.317396 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1' Nov 10 06:09:13.325146 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description Nov 10 06:09:13.325146 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1' Nov 10 06:09:13.332736 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description Nov 10 06:09:13.332736 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1' Nov 10 06:09:13.340707 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description Nov 10 06:09:13.340707 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1' Nov 10 06:09:13.348443 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description Nov 10 06:09:13.348443 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1' Nov 10 06:09:14.460835 sonic INFO swss#supervisor-proc-exit-listener: Process orchagent exited unxepectedly. Terminating supervisor... Nov 10 06:09:20.642954 sonic INFO containerd[512]: time="2016-11-10T06:09:20.642830813Z" level=info msg="shim reaped" id=28a3cfff32772bc530fa245719f439b75623f57893ac82134525a2b75f67e41c Nov 10 06:09:20.652922 sonic INFO dockerd[547]: time="2016-11-10T06:09:20.652784418Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="events.TaskDelete" Nov 10 06:09:20.774943 sonic INFO swss.sh[5221]: 0 Nov 10 06:09:20.782956 sonic NOTICE root: Stopping swss service... Nov 10 06:09:20.786862 sonic NOTICE root: Locking /tmp/swss-syncd-lock from swss service Nov 10 06:09:20.791343 sonic NOTICE root: Locked /tmp/swss-syncd-lock (10) from swss service Nov 10 06:09:21.199234 sonic NOTICE root: Warm boot flag: swss false. Nov 10 06:09:21.244675 sonic INFO swss.sh[22837]: swss Nov 10 06:09:21.249579 sonic NOTICE root: Stopped swss service... Nov 10 06:09:21.253467 sonic NOTICE root: Unlocking /tmp/swss-syncd-lock (10) from swss service Nov 10 06:09:21.260178 sonic INFO systemd[1]: Stopping syncd service... Nov 10 06:09:21.264662 sonic NOTICE root: Stopping syncd service... Nov 10 06:09:21.268833 sonic NOTICE root: Locking /tmp/swss-syncd-lock from syncd service Nov 10 06:09:21.273405 sonic NOTICE root: Locked /tmp/swss-syncd-lock (10) from syncd service Nov 10 06:09:21.695354 sonic NOTICE root: Warm boot flag: syncd false. Nov 10 06:09:21.699772 sonic NOTICE root: cold shutdown syncd process ... Nov 10 06:09:21.839122 sonic NOTICE syncd#syncd_request_shutdown: :- main: requested COLD shutdown Nov 10 06:09:21.839310 sonic NOTICE syncd#syncd: :- syncd_main: is asic queue empty: 0 Nov 10 06:09:21.839310 sonic ERR syncd#syncd: brcm_sai_set_port_attribute:1027 Unknown port attribute 77 passed Nov 10 06:09:21.839310 sonic ERR syncd#syncd: :- processEvent: VID: oid:0x1000000000012 RID: oid:0x100000022 Nov 10 06:09:21.839310 sonic ERR syncd#syncd: :- processEvent: attr: SAI_PORT_ATTR_PRIORITY_FLOW_CONTROL_TX: 120 Nov 10 06:09:21.839310 sonic ERR syncd#syncd: :- processEvent: failed to execute api: set, key: SAI_OBJECT_TYPE_PORT:oid:0x1000000000012, status: SAI_STATUS_ATTR_NOT_SUPPORTED_0 Nov 10 06:09:21.839336 sonic ERR syncd#syncd: :- syncd_main: Runtime error: :- processEvent: failed to execute api: set, key: SAI_OBJECT_TYPE_PORT:oid:0x1000000000012, status: SAI_STATUS_ATTR_NOT_SUPPORTED_0 Nov 10 06:09:21.839336 sonic NOTICE syncd#syncd: :- notify_OA_about_syncd_exception: sending switch_shutdown_request notification to OA Nov 10 06:09:21.839246 sonic INFO syncd.sh[22999]: requested COLD shutdown Nov 10 06:09:21.839477 sonic NOTICE syncd#syncd: :- notify_OA_about_syncd_exception: notification send successfull Nov 10 06:09:21.839477 sonic NOTICE syncd#syncd: :- syncd_main: Removing the switch gSwitchId=0xb970112100000000 Nov 10 06:09:24.003084 sonic WARNING kernel: [159188.373052] linux-kernel-bde (9483): _interrupt_disconnect d 0 Nov 10 06:09:24.003109 sonic WARNING kernel: [159188.373055] linux-kernel-bde (9483): _interrupt_disconnect: isr_active = 1 Nov 10 06:09:24.003111 sonic WARNING kernel: [159188.373055] linux-kernel-bde (9483): disconnect primary isr Nov 10 06:09:24.012107 sonic INFO syncd#supervisord: syncd 0:soc_shutdown: soc_shutdown: all units detached#015 Nov 10 06:09:24.015107 sonic WARNING kernel: [159188.384102] linux-bcm-knet (10179): Next DMA events (0x00000001) Nov 10 06:09:24.015969 sonic NOTICE syncd#syncd: :- syncd_main: remove switch took 1.654644 sec Nov 10 06:09:24.015969 sonic NOTICE syncd#syncd: :- syncd_main: calling api uninitialize Nov 10 06:09:24.015969 sonic NOTICE syncd#syncd: :- syncd_main: uninitialize finished Nov 10 06:09:24.860710 sonic NOTICE syncd#syncd: :- threadFunction: ending timer watchdog thread Nov 10 06:09:24.886226 sonic NOTICE syncd#dsserve: child /usr/bin/syncd exited status: 0 Nov 10 06:09:24.886497 sonic INFO syncd#supervisord: syncd [5] child /usr/bin/syncd exited status: 0 Nov 10 06:09:25.267600 sonic NOTICE root: Finished cold shutdown syncd process ... Nov 10 06:09:26.300595 sonic INFO syncd#supervisord 2016-11-10 06:09:24,886 INFO exited: syncd (exit status 0; expected) Nov 10 06:09:26.300595 sonic INFO syncd#supervisord 2016-11-10 06:09:25,360 WARN received SIGTERM indicating exit request Nov 10 06:09:26.300595 sonic INFO syncd#supervisord 2016-11-10 06:09:25,361 INFO waiting for rsyslogd to die Nov 10 06:09:27.105062 sonic WARNING snmp#snmp-subagent [sonic_ax_impl] WARNING: No managment ports found in b'MGMT_PORT|' Nov 10 06:09:27.539052 sonic INFO containerd[512]: time="2016-11-10T06:09:27.538891458Z" level=info msg="shim reaped" id=6335d8395a21594ccdeaf7c8c2be5a0f619f16fb23e7adfbc992a55abc1d1caf Nov 10 06:09:27.548767 sonic INFO dockerd[547]: time="2016-11-10T06:09:27.548654301Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="events.TaskDelete" Nov 10 06:09:27.634112 sonic INFO syncd.sh[22999]: syncd Nov 10 06:09:27.634655 sonic INFO syncd.sh[9332]: 0 Nov 10 06:09:27.640219 sonic NOTICE root: Stopped syncd service... Nov 10 06:09:27.645312 sonic NOTICE root: Unlocking /tmp/swss-syncd-lock (10) from syncd service Nov 10 06:09:27.651296 sonic INFO systemd[1]: Stopped syncd service. Nov 10 06:09:27.656606 sonic INFO systemd[1]: Stopping DHCP relay container... Nov 10 06:09:27.901079 sonic INFO containerd[512]: time="2016-11-10T06:09:27.900974047Z" level=info msg="shim reaped" id=6574c4e328d49dba89dd78a5f7c1dc0a41ffb7d35784b805e633195548807c85 Nov 10 06:09:27.911610 sonic INFO dockerd[547]: time="2016-11-10T06:09:27.911485369Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="events.TaskDelete" Nov 10 06:09:27.996899 sonic INFO dhcp_relay.sh[23652]: dhcp_relay Nov 10 06:09:27.997703 sonic INFO dhcp_relay.sh[10065]: 0 Nov 10 06:09:28.002436 sonic INFO systemd[1]: Stopped DHCP relay container. Nov 10 06:09:28.003321 sonic INFO systemd[1]: Stopping TEAMD container... Nov 10 06:09:28.292591 sonic INFO containerd[512]: time="2016-11-10T06:09:28.292421480Z" level=info msg="shim reaped" id=cc602baadada95bf7d8cd1aa41a2de72efcbe0338dde255c23ccfe9abec6cfd4 Nov 10 06:09:28.302508 sonic INFO dockerd[547]: time="2016-11-10T06:09:28.302404002Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="events.TaskDelete" Nov 10 06:09:28.404531 sonic INFO teamd.sh[23691]: teamd Nov 10 06:09:28.405110 sonic INFO teamd.sh[8270]: 0 Nov 10 06:09:28.412014 sonic INFO systemd[1]: Stopped TEAMD container. Nov 10 06:09:28.419018 sonic INFO systemd[1]: Stopping Router advertiser container... Nov 10 06:09:29.675716 sonic INFO containerd[512]: time="2016-11-10T06:09:29.675589177Z" level=info msg="shim reaped" id=b767be4ec3eb0a555a2365b5d10bc698011215c4d55a0c0f7ecac96654138b0e Nov 10 06:09:29.685852 sonic INFO dockerd[547]: time="2016-11-10T06:09:29.685727099Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Nov 10 06:09:29.786365 sonic INFO radv.sh[23732]: radv Nov 10 06:09:29.787249 sonic INFO radv.sh[8298]: 0 Nov 10 06:09:29.793899 sonic INFO systemd[1]: Stopped Router advertiser container. [orchagent.1478758153.76.core.gz]

pollyhsu2git commented 4 years ago

@afzalashfak You may try the workaround of the AS7816-64X issue report via https://github.com/Azure/sonic-buildimage/issues/3508.

afzalashfak commented 4 years ago

@pollyhsu2git This issue is particularly observed when we enable asymmetric pfc in an interface. It is not related to sonic-buildimage. It is a configuration issue. FYI, I'd loaded the build using the workaround mentioned in https://github.com/Azure/sonic-buildimage/issues/3508, before trying the configuration.