sonic-net / sonic-swss

SONiC Switch State Service (SwSS)
https://azure.github.io/SONiC
Other
179 stars 537 forks source link

orchagent crash when 20k BGP route advertisement pushd local device #1301

Open tim-rj opened 4 years ago

tim-rj commented 4 years ago

orchagent crash when 20k BGP route advertisement pushd local device.

Steps to reproduce the issue:

  1. Establish IPv4 BGP neighbor with auxiliary testing device
  2. 20K route is printed on the auxiliary test end and passed to the local end

Describe the results you received: May 28 16:17:47.035024 switch2 ERR syncd#syncd: [none] _brcm_sai_l3_route_config:1927 L3 route add failed with error Table full (0xfffffffa). May 28 16:17:47.035024 switch2 ERR syncd#syncd: [none] brcm_sai_create_route_entry:413 L3 route add failed with error -13. May 28 16:17:47.035095 switch2 ERR syncd#syncd: :- processEvent: attr: SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID: oid:0x4000000000a13 May 28 16:17:47.035095 switch2 ERR syncd#syncd: :- processEvent: failed to execute api: create, key: SAI_OBJECT_TYPE_ROUTE_ENTRY:{"dest":"15.12.113.0/24","switch_id":"oid:0x21000000000000","vr":"oid:0x300000000003a"}, status: SAI_STATUS_TABLE_FULL May 28 16:17:47.035117 switch2 ERR syncd#syncd: :- syncd_main: Runtime error: :- processEvent: failed to execute api: create, key: SAI_OBJECT_TYPE_ROUTE_ENTRY:{"dest":"15.12.113.0/24","switch_id":"oid:0x21000000000000","vr":"oid:0x300000000003a"}, status: SAI_STATUS_TABLE_FULL May 28 16:17:47.035117 switch2 NOTICE syncd#syncd: :- notify_OA_about_syncd_exception: sending switch_shutdown_request notification to OA May 28 16:17:47.036315 switch2 NOTICE syncd#syncd: :- notify_OA_about_syncd_exception: notification send successfull May 28 16:17:47.036502 switch2 NOTICE swss#orchagent: :- handle_switch_shutdown_request: switch shutdown request May 28 16:17:47.039008 switch2 INFO swss#supervisord: orchagent terminate called after throwing an instance of 'std::invalid_argument' May 28 16:17:47.039008 switch2 INFO swss#supervisord: orchagent what(): parse error - unexpected end of input

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally): root@switch2:/home/admin# show version

SONiC Software Version: SONiC.HEAD.67-8367dfeb Distribution: Debian 9.12 Kernel: 4.9.0-11-2-amd64 Build commit: 8367dfeb Build date: Wed May 6 13:49:49 UTC 2020 Built by: johnar@jenkins-worker-7

Platform: x86_64-ruijie_b6510-48vs8cq-r0 HwSKU: B6510-48VS8CQ ASIC: broadcom Serial Number: G1W10072 Uptime: 09:05:15 up 1:15, 1 user, load average: 0.38, 0.43, 0.46

Docker images: REPOSITORY TAG IMAGE ID SIZE docker-syncd-brcm HEAD.67-8367dfeb 6902f81e3ca3 430MB docker-syncd-brcm latest 6902f81e3ca3 430MB docker-platform-monitor HEAD.67-8367dfeb 299fc1d87ebf 334MB docker-platform-monitor latest 299fc1d87ebf 334MB docker-router-advertiser HEAD.67-8367dfeb d1f31448c1b9 283MB docker-router-advertiser latest d1f31448c1b9 283MB docker-fpm-frr HEAD.67-8367dfeb c2b1f3dde8c8 328MB docker-fpm-frr latest c2b1f3dde8c8 328MB docker-sflow HEAD.67-8367dfeb 22588d030e81 308MB docker-sflow latest 22588d030e81 308MB docker-lldp-sv2 HEAD.67-8367dfeb 8a9b1ba67160 305MB docker-lldp-sv2 latest 8a9b1ba67160 305MB docker-orchagent HEAD.67-8367dfeb eed753ed4e3b 326MB docker-orchagent latest eed753ed4e3b 326MB docker-dhcp-relay HEAD.67-8367dfeb a91bd90f721a 293MB docker-dhcp-relay latest a91bd90f721a 293MB docker-database HEAD.67-8367dfeb 7665762c9470 283MB docker-database latest 7665762c9470 283MB docker-snmp-sv2 HEAD.67-8367dfeb 893a54916304 340MB docker-snmp-sv2 latest 893a54916304 340MB docker-teamd HEAD.67-8367dfeb 2e77c5e410ba 307MB docker-teamd latest 2e77c5e410ba 307MB docker-nat HEAD.67-8367dfeb d1c78016f845 309MB docker-nat latest d1c78016f845 309MB docker-sonic-mgmt-framework HEAD.67-8367dfeb f1f88830db0c 422MB docker-sonic-mgmt-framework latest f1f88830db0c 422MB docker-sonic-telemetry HEAD.67-8367dfeb 65d82748277a 345MB docker-sonic-telemetry latest 65d82748277a 345MB

root@switch2:/home/admin#

rlhui commented 4 years ago

This is currently expected if routes scale exceeded chip limit. We'll enhance to address this.