search

aristanetworks / sonic

Open source drivers and initialization library for Arista platforms running SONiC
GNU General Public License v2.0
22 stars 30 forks source link

[chassis] syncd busy on all linecards, taking >200% CPU usage #59

Closed wenyiz2021 closed 1 year ago

wenyiz2021 commented 1 year ago 
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                           
1123054 root      20   0 4713628   1.3g 183348 S 262.5   8.6 205:33.71 syncd                                                                                                             
1158282 admin     20   0   11076   3936   3136 R   6.2   0.0   0:00.02 top                                                                                                               
      1 root      20   0  165960  12320   8532 S   0.0   0.1   2:07.43 systemd

syncd process is too busy on all linecards, >200% CPU usage, sup is fine. Ideally it should not exceed 60%

syncd in syslog doesn't seem to be busy:

Dec 14 22:20:22.224405 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 15 ms for 'bulkcreate:SAI_OBJECT_TYPE_NEXT_HOP_GROUP_MEMBER:28'
Dec 14 22:20:41.225689 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 4 ms for 'bulkcreate:SAI_OBJECT_TYPE_NEXT_HOP_GROUP_MEMBER:45'
Dec 14 22:21:20.228142 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 13 ms for 'bulkremove:SAI_OBJECT_TYPE_NEXT_HOP_GROUP_MEMBER:46'
Dec 14 22:21:51.230235 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 0 ms for 'get:SAI_OBJECT_TYPE_PORT:oid:0x1000000000112'
Dec 14 22:22:51.234195 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 0 ms for 'get:SAI_OBJECT_TYPE_PORT:oid:0x1000000000120'
Dec 14 22:23:21.236090 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 0 ms for 'get:SAI_OBJECT_TYPE_PORT:oid:0x100000000013b'
Dec 14 22:24:21.239972 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 0 ms for 'get:SAI_OBJECT_TYPE_PORT:oid:0x1000000000119'
Dec 14 22:25:51.245814 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 10 ms for 'get:SAI_OBJECT_TYPE_PORT:oid:0x100000000011c'
Dec 14 22:31:51.269894 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 0 ms for 'get:SAI_OBJECT_TYPE_PORT:oid:0x1000000000137'
Dec 14 22:33:21.275946 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 0 ms for 'get:SAI_OBJECT_TYPE_PORT:oid:0x100000000013b'
Dec 14 22:33:51.277801 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 0 ms for 'get:SAI_OBJECT_TYPE_PORT:oid:0x100000000010a'
Dec 14 22:34:21.107812 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10000 (calls 10000) Syncd::syncUpdateRedisQuadEvent op took: 20 ms
Dec 14 22:35:51.285673 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 0 ms for 'get:SAI_OBJECT_TYPE_PORT:oid:0x100000000012c'
Dec 14 22:37:51.293389 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 0 ms for 'get:SAI_OBJECT_TYPE_PORT:oid:0x1000000000129'
Dec 14 22:39:21.300984 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 0 ms for 'get:SAI_OBJECT_TYPE_PORT:oid:0x100000000012b'
Dec 14 22:43:51.318966 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 0 ms for 'get:SAI_OBJECT_TYPE_PORT:oid:0x100000000013b'
Dec 14 22:48:51.272151 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10000 (calls 10000) Syncd::syncUpdateRedisQuadEvent op took: 2 ms
Dec 14 23:18:34.266790 str2-7804-lc7-1 NOTICE syncd#syncd: message repeated 2 times: [ :- inc: 10000 (calls 10000) Syncd::syncUpdateRedisQuadEvent op took: 2 ms]
Dec 14 23:18:34.266790 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10776 (calls 312) Syncd::syncUpdateRedisBulkQuadEvent op took: 583 ms
Dec 14 23:18:34.460880 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 6 ms for 'bulkremove:SAI_OBJECT_TYPE_NEXT_HOP_GROUP_MEMBER:2'
Dec 14 23:18:35.461030 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 38 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Dec 14 23:18:36.911942 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10624 (calls 19) Syncd::syncUpdateRedisBulkQuadEvent op took: 304 ms
Dec 14 23:18:39.380058 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10301 (calls 15) Syncd::syncUpdateRedisBulkQuadEvent op took: 266 ms
Dec 14 23:18:39.465615 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 24 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Dec 14 23:18:41.262962 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10331 (calls 12) Syncd::syncUpdateRedisBulkQuadEvent op took: 290 ms
Dec 14 23:18:41.465771 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 46 ms for 'bulkremove:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Dec 14 23:21:37.338501 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10106 (calls 31) Syncd::syncUpdateRedisBulkQuadEvent op took: 350 ms
Dec 14 23:21:38.221308 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10000 (calls 10000) Syncd::processBulkEntry::processEntry(route_entry) CREATE op took: 372 ms
Dec 14 23:21:40.089020 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10651 (calls 19) Syncd::syncUpdateRedisBulkQuadEvent op took: 398 ms
Dec 14 23:21:41.478205 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 14 ms for 'bulkcreate:SAI_OBJECT_TYPE_NEXT_HOP_GROUP_MEMBER:10'
Dec 14 23:21:42.033542 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10000 (calls 10000) Syncd::processBulkEntry::processEntry(route_entry) CREATE op took: 418 ms
Dec 14 23:21:42.493470 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 57 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Dec 14 23:21:42.694115 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10347 (calls 19) Syncd::syncUpdateRedisBulkQuadEvent op took: 332 ms
Dec 14 23:21:45.493713 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 26 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Dec 14 23:21:45.657430 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10900 (calls 20) Syncd::syncUpdateRedisBulkQuadEvent op took: 314 ms
Dec 14 23:21:48.493891 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 0 ms for 'bulkcreate:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Dec 14 23:21:48.953200 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10597 (calls 15) Syncd::syncUpdateRedisBulkQuadEvent op took: 301 ms
Dec 14 23:21:51.253867 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10000 (calls 10000) Syncd::processBulkEntry::processEntry(route_entry) CREATE op took: 318 ms
Dec 14 23:21:51.494170 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 12 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Dec 14 23:21:51.623481 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10954 (calls 26) Syncd::syncUpdateRedisBulkQuadEvent op took: 383 ms
Dec 14 23:30:41.538635 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 20 ms for 'bulkcreate:SAI_OBJECT_TYPE_NEXT_HOP_GROUP_MEMBER:8'
Dec 14 23:30:43.444568 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10627 (calls 22) Syncd::syncUpdateRedisBulkQuadEvent op took: 450 ms
Dec 14 23:30:46.489774 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10185 (calls 15) Syncd::syncUpdateRedisBulkQuadEvent op took: 268 ms
Dec 14 23:30:46.538948 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 45 ms for 'bulkremove:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Dec 14 23:30:47.103624 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10377 (calls 11) Syncd::syncUpdateRedisBulkQuadEvent op took: 256 ms
Dec 14 23:30:47.539070 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 3 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Dec 14 23:30:48.871199 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10719 (calls 17) Syncd::syncUpdateRedisBulkQuadEvent op took: 405 ms
Dec 14 23:33:14.548543 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 1 ms for 'bulkremove:SAI_OBJECT_TYPE_NEXT_HOP_GROUP_MEMBER:16'
Dec 14 23:33:21.136809 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10000 (calls 10000) Syncd::syncUpdateRedisQuadEvent op took: 85 ms
Dec 14 23:33:51.060763 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10034 (calls 47) Syncd::syncUpdateRedisBulkQuadEvent op took: 373 ms
Dec 14 23:33:54.135890 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10000 (calls 10000) Syncd::processBulkEntry::processEntry(route_entry) CREATE op took: 357 ms
Dec 14 23:33:54.551026 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 41 ms for 'bulkcreate:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Dec 14 23:33:54.679095 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10980 (calls 16) Syncd::syncUpdateRedisBulkQuadEvent op took: 384 ms
Dec 14 23:33:57.551247 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 101 ms for 'bulkcreate:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Dec 14 23:33:57.715572 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10000 (calls 10000) Syncd::processBulkEntry::processEntry(route_entry) CREATE op took: 364 ms
Dec 14 23:33:58.341981 str2-7804-lc7-1 NOTICE syncd#syncd: :- inc: 10422 (calls 18) Syncd::syncUpdateRedisBulkQuadEvent op took: 452 ms
Dec 14 23:33:58.551340 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 15 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Dec 14 23:33:59.551432 str2-7804-lc7-1 NOTICE syncd#syncd: :- threadFunction: time span 49 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
wenyiz2021 commented 1 year ago

@Staphylo any ideas? thanks

Staphylo commented 1 year ago

@wenyiz2021 I don't know why the cpu usage of syncd is high like this. My current guess would be counter polling. I'm however not knowledgeable enough about dnx SAI. @kenneth-arista do you know why there is such a big cpu usage of syncd?

wenyiz2021 commented 1 year ago

links are not flapping, this operation seems busy:

Dec 14 23:23:33.981693 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_OUT_CURR_OCCUPANCY_BYTES is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.981754 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_OUT_WATERMARK_BYTES is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.981807 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_OUT_SHARED_CURR_OCCUPANCY_BYTES is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.981857 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_OUT_SHARED_WATERMARK_BYTES is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.981908 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_0_RX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.981958 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_0_TX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982008 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_1_RX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982058 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_1_TX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982111 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_2_RX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982161 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_2_TX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982212 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_3_RX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982262 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_3_TX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982313 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_4_RX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982363 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_4_TX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982413 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_5_RX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982463 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_5_TX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982514 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_6_RX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982564 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_6_TX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982653 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_7_RX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982722 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_7_TX_PAUSE_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982775 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_0_RX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982836 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_0_TX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982890 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_1_RX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982942 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_1_TX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.982994 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_2_RX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983044 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_2_TX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983095 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_3_RX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983147 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_3_TX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983198 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_4_RX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983249 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_4_TX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983300 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_5_RX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983350 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_5_TX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983401 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_6_RX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983451 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_6_TX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983502 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_7_RX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983552 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_7_TX_PAUSE_DURATION_US is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983603 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_DOT3_STATS_ALIGNMENT_ERRORS is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983653 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_DOT3_STATS_FCS_ERRORS is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983704 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_DOT3_STATS_SINGLE_COLLISION_FRAMES is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983755 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_DOT3_STATS_MULTIPLE_COLLISION_FRAMES is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983816 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_DOT3_STATS_SQE_TEST_ERRORS is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983868 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_DOT3_STATS_DEFERRED_TRANSMISSIONS is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983919 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_DOT3_STATS_LATE_COLLISIONS is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.983969 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_DOT3_STATS_EXCESSIVE_COLLISIONS is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984018 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_DOT3_STATS_INTERNAL_MAC_TRANSMIT_ERRORS is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984068 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_DOT3_STATS_CARRIER_SENSE_ERRORS is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984118 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_DOT3_STATS_FRAME_TOO_LONGS is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984167 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_DOT3_STATS_INTERNAL_MAC_RECEIVE_ERRORS is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984218 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_DOT3_STATS_SYMBOL_ERRORS is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984268 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_DOT3_CONTROL_IN_UNKNOWN_OPCODES is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984317 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_EEE_TX_EVENT_COUNT is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984367 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_EEE_RX_EVENT_COUNT is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984417 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_EEE_TX_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984468 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_EEE_RX_DURATION is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984519 str2-7804-lc5-1 INFO syncd0#syncd: [06:00.0] SAI_API_PORT:brcm_sai_get_port_stats:5553 Multi stats get failed with error Feature unavailable (0xfffffff0).
Dec 14 23:23:33.984569 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_IF_IN_FABRIC_DATA_UNITS is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984620 str2-7804-lc5-1 INFO syncd0#syncd: [06:00.0] SAI_API_PORT:brcm_sai_get_port_stats:5553 Multi stats get failed with error Feature unavailable (0xfffffff0).
Dec 14 23:23:33.984670 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_IF_OUT_FABRIC_DATA_UNITS is not supported on port RID oid:0x100000010: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984721 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_GREEN_WRED_DROPPED_PACKETS is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984772 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_GREEN_WRED_DROPPED_BYTES is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984834 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_YELLOW_WRED_DROPPED_PACKETS is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984886 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_YELLOW_WRED_DROPPED_BYTES is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984938 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_RED_WRED_DROPPED_PACKETS is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.984989 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_RED_WRED_DROPPED_BYTES is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985041 str2-7804-lc5-1 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:_brcm_sai_port_wred_stats_get:15101 Hardware failure -16 in getting WRED stat 68 for port 17
Dec 14 23:23:33.985092 str2-7804-lc5-1 ERR syncd0#syncd: [06:00.0] SAI_API_PORT:brcm_sai_get_port_stats:5176 port wred stats get failed with error -2.
Dec 14 23:23:33.985143 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_WRED_DROPPED_PACKETS is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985193 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_IN_CURR_OCCUPANCY_BYTES is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985245 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_IN_WATERMARK_BYTES is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985296 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_IN_SHARED_CURR_OCCUPANCY_BYTES is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985348 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_IN_SHARED_WATERMARK_BYTES is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985418 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_OUT_CURR_OCCUPANCY_BYTES is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985471 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_OUT_WATERMARK_BYTES is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985521 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_OUT_SHARED_CURR_OCCUPANCY_BYTES is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985572 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_OUT_SHARED_WATERMARK_BYTES is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985622 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_0_RX_PAUSE_DURATION is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985673 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_0_TX_PAUSE_DURATION is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985724 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_1_RX_PAUSE_DURATION is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985775 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_1_TX_PAUSE_DURATION is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985825 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_2_RX_PAUSE_DURATION is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985886 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_2_TX_PAUSE_DURATION is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985938 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_3_RX_PAUSE_DURATION is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.985990 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_3_TX_PAUSE_DURATION is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.986040 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_4_RX_PAUSE_DURATION is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Dec 14 23:23:33.986091 str2-7804-lc5-1 NOTICE syncd0#syncd: :- getSupportedPortCounters: Counter SAI_PORT_STAT_PFC_4_TX_PAUSE_DURATION is not supported on port RID oid:0x100000011: SAI_STATUS_NOT_SUPPORTED
Staphylo commented 1 year ago

@wenyiz2021 thanks for the extra logs, our team is currently investigating this issue.

kenneth-arista commented 1 year ago

I have one more change to bring down the syncd MHz to be more similar to another vendor. The change is to set the statistics collection interval in the SDK to 1 sec instead of 1 msec. I'll post a PR for the tweak.

wenyiz2021 commented 1 year ago

@Staphylo @kenneth-arista I'm still seeing syncd CPU usage >100% on all linecards, can you reopen this issue for tracking purposes? thanks