search

platinasystems / go

Other
9 stars 68 forks source link

mmu continuously drops mcast after storm #153

Closed rondv closed 5 years ago

rondv commented 5 years ago

misconfigured vlan with a loop which created a packet storm, system did not recover after breaking the loop

ping packets continue to drop "mmu multicast tx cos0 drop packets"

this happened a few times during svi bringup, I don't have exact repro steps

         Name                  Address         Link    Counter                                      Count

xeth1 02:46:8a:00:01:98 up port rx packets 1 port rx bytes 68 port rx 65 to 127 byte packets 1 port rx good packets 1 port rx broadcast packets 1 port rx 1tag vlan packets 1 rx pipe port table packets 1 rx pipe port table bytes 64 xeth3 02:46:8a:00:01:9a up tx pipe vlan tagged packets 1 tx pipe unicast queue cos0 packets 1 tx pipe unicast queue cos0 bytes 64 tx pipe port table packets 1 tx pipe port table bytes 64 port tx packets 1 port tx bytes 68 port tx 65 to 127 byte packets 1 port tx good packets 1 port tx broadcast packets 1 port tx 1tag vlan packets 1 xeth5 02:46:8a:00:01:9c up mmu multicast tx cos0 drop packets 1 mmu multicast tx cos0 drop bytes 64 xeth7 02:46:8a:00:01:9e up mmu multicast tx cos0 drop packets 1 mmu multicast tx cos0 drop bytes 64 meth-0 00:00:00:00:00:00 up mmu multicast tx cos0 drop packets 1 mmu multicast tx cos0 drop bytes 64 meth-1 00:00:00:00:00:00 up port rx packets 1 port rx bytes 64 port rx 64 byte packets 1 port rx good packets 1 port rx broadcast packets 1 rx pipe port table packets 1 rx pipe port table bytes 60 fe1-cpu 00:00:00:00:00:00 up tx pipe cpu vlan-redirect packets 1 tx pipe cpu vlan-redirect bytes 64 tx pipe port table packets 1 tx pipe port table bytes 64

mj073 commented 5 years ago

Please find the logs to reproduce the issue in the attachment. Congestion state of DUT is not consistently established, but once we establish a congestion state issue can be reproduced consistently. mcast_issue_screen_logs.txt

Also I observed that, after setting multicast related buffers on DUT using cli "sudo goes vnet set fe1 mmu-mc-re-lossless-config", packets are not getting dropped.

mj073 commented 5 years ago

Previously I observed that, while congestion state is still present and when I execute the cli to set mcast thresholds, I was able to see drops on at-least one of the interface on DUT when I send incremental packets from pktgen.

Today I modified the cli for setting missing mcast thresholds. Now after executing this cli on DUT while congestion state is still present, I don't see any drops on any interface on DUT when I send incremental packets from pktgen.

Please find the attached logs in the attachment. mcast_issue_screen_logs_2.txt

rondv commented 5 years ago

tracking integration task here: GOES Development Status (1.3 release)