sonic-net / SONiC

Landing page for Software for Open Networking in the Cloud (SONiC) - https://sonic-net.github.io/SONiC/
2.22k stars 1.12k forks source link

syslog errors observed during syncd init process #1248

Open rajneeshaec opened 1 year ago

rajneeshaec commented 1 year ago

Problem Summary:- syslog errors observed during syncd init

Note:- It looks like FDB add request coming from NOS with Vlan (1000) + port (Eth48), but the port is not yet added to vlan. After this error, syncd restarts and in subsequent init, the issue not seen

Steps to reproduce:-

Syslog errors:-

Oct 12 04:17:46.191496 str-marvell-acs-1 ERR teamd#teammgrd: :- checkPortIffUp: Failed to get port Ethernet51 flags
E
E               Oct 12 04:17:46.191496 str-marvell-acs-1 ERR teamd#teammgrd: :- addLagMember: Failed to add Ethernet51 to port channel PortChannel104
E
E               Oct 12 04:17:47.684819 str-marvell-acs-1 ERR teamd#teammgrd: :- checkPortIffUp: Failed to get port Ethernet50 flags
E
E               Oct 12 04:17:47.684819 str-marvell-acs-1 ERR teamd#teammgrd: :- addLagMember: Failed to add Ethernet50 to port channel PortChannel103
E
E               Oct 12 04:17:49.265865 str-marvell-acs-1 ERR teamd#teammgrd: :- checkPortIffUp: Failed to get port Ethernet49 flags
E
E               Oct 12 04:17:49.265865 str-marvell-acs-1 ERR teamd#teammgrd: :- addLagMember: Failed to add Ethernet49 to port channel PortChannel102
E
E               Oct 12 04:17:50.736431 str-marvell-acs-1 ERR teamd#teammgrd: :- checkPortIffUp: Failed to get port Ethernet48 flags
E
E               Oct 12 04:17:50.736774 str-marvell-acs-1 ERR teamd#teammgrd: :- addLagMember: Failed to add Ethernet48 to port channel PortChannel101
E
E               Oct 12 04:18:21.330508 str-marvell-acs-1 ERR syncd#syncd: :- listFailedAttributes: SAI_FDB_ENTRY_ATTR_PACKET_ACTION = SAI_PACKET_ACTION_FORWARD
E
E               Oct 12 04:18:21.330711 str-marvell-acs-1 ERR syncd#syncd: :- listFailedAttributes: SAI_FDB_ENTRY_ATTR_TYPE = SAI_FDB_ENTRY_TYPE_DYNAMIC
E
E               Oct 12 04:18:21.330914 str-marvell-acs-1 ERR syncd#syncd: :- listFailedAttributes: SAI_FDB_ENTRY_ATTR_BRIDGE_PORT_ID = oid:0x3a000000000030
E
E               Oct 12 04:18:21.331152 str-marvell-acs-1 ERR syncd#syncd: :- processFdbs: failed to create_fdb_entry {"bvid":"oid:0x2600000000076b","mac":"AC:16:2D:76:20:2F","switch_id":"oid:0x21000000000000"}: SAI_STATUS_INVALID_PARAMETER
E
E               Oct 12 04:18:21.364074 str-marvell-acs-1 ERR syncd#syncd: :- run: Runtime error during syncd init: :- processFdbs: failed to create_fdb_entry {"bvid":"oid:0x2600000000076b","mac":"AC:16:2D:76:20:2F","switch_id":"oid:0x21000000000000"}: SAI_STATUS_INVALID_PARAMETER

sairedis logs:-

root@str-marvell-acs-1:/var/log/swss# zgrep "AC:16:2D:76:20:1A" *
sairedis.rec:2022-10-17.13:35:14.083459|n|fdb_event|[{"fdb_entry":"{\"bvid\":\"oid:0x2600000000076c\",\"mac\":\"AC:16:2D:76:20:1A\",\"switch_id\":\"oid:0x21000000000000\"}","fdb_event":"SAI_FDB_EVENT_LEARNED","list":[{"id":"SAI_FDB_ENTRY_ATTR_TYPE","value":"SAI_FDB_ENTRY_TYPE_DYNAMIC"},{"id":"SAI_FDB_ENTRY_ATTR_BRIDGE_PORT_ID","value":"oid:0x3a000000000793"},{"id":"SAI_FDB_ENTRY_ATTR_PACKET_ACTION","value":"SAI_PACKET_ACTION_FORWARD"}]}]|
sairedis.rec.16.gz:2022-10-17.13:14:57.247181|n|fdb_event|[{"fdb_entry":"{\"bvid\":\"oid:0x2600000000076c\",\"mac\":\"AC:16:2D:76:20:1A\",\"switch_id\":\"oid:0x21000000000000\"}","fdb_event":"SAI_FDB_EVENT_LEARNED","list":[{"id":"SAI_FDB_ENTRY_ATTR_TYPE","value":"SAI_FDB_ENTRY_TYPE_DYNAMIC"},{"id":"SAI_FDB_ENTRY_ATTR_BRIDGE_PORT_ID","value":"oid:0x3a000000000793"},{"id":"SAI_FDB_ENTRY_ATTR_PACKET_ACTION","value":"SAI_PACKET_ACTION_FORWARD"}]}]|

Image:-

root@str-marvell-acs-1:/home/admin# show version

SONiC Software Version: SONiC.202205.179361-7c1015b45
Distribution: Debian 11.5
Kernel: 5.10.0-12-2-armmp
Build commit: 7c1015b45
Blueve commented 1 year ago

@zhangyanzhao can you help triage this issue? Marvell is plan to provide a mitigation in their implementation to eliminate these error logs, however that is hidden the true issue in actual. The owner of this issue should be saireids