Closed Ndancejic closed 2 months ago
Please link the master PR in description
Please test on 202305 image and confirm the results. @StormLiangMS for viz
Please test on 202305 image and confirm the results. @StormLiangMS for viz
tested on 202305 dualtor testbed:
w/out fix:
Jun 9 06:05:46.642266 svcstr-7050-acs-4 NOTICE swss#orchagent: :- setState: [Ethernet12] Set MUX state from active to standby
Jun 9 06:05:46.642559 svcstr-7050-acs-4 NOTICE swss#orchagent: :- nbrHandler: Processing neighbors for mux Ethernet12, enable 0, state 2
Jun 9 06:05:46.642719 svcstr-7050-acs-4 NOTICE swss#orchagent: :- updateRoute: Updating route 11.11.11.11/32 pointing to Mux nexthops 192.168.0.2@Vlan1000,192.168.100.4@Vlan1000
Jun 9 06:05:46.644840 svcstr-7050-acs-4 NOTICE swss#orchagent: :- updateRoute: setting route 11.11.11.11/32 with nexthop 192.168.100.4@Vlan1000 400000000081d
Jun 9 06:05:46.648190 svcstr-7050-acs-4 NOTICE swss#orchagent: :- create_route: Created tunnel route to 192.168.0.4/32
Jun 9 06:05:46.648287 svcstr-7050-acs-4 NOTICE swss#orchagent: :- disableNeighbor: Neighbor disable request for 192.168.0.4
Jun 9 06:05:46.650488 svcstr-7050-acs-4 NOTICE swss#orchagent: :- removeNeighbor: Removed next hop 192.168.0.4 on Vlan1000
Jun 9 06:05:46.654161 svcstr-7050-acs-4 NOTICE swss#orchagent: :- removeNeighbor: Removed neighbor aa:f4:6e:25:d3:03 on Vlan1000
Jun 9 06:05:46.662224 svcstr-7050-acs-4 NOTICE swss#orchagent: :- create_route: Created tunnel route to 192.168.100.4/32
Jun 9 06:05:46.662224 svcstr-7050-acs-4 NOTICE swss#orchagent: :- disableNeighbor: Neighbor disable request for 192.168.100.4
Jun 9 06:05:46.662224 svcstr-7050-acs-4 ERR swss#orchagent: :- meta_generic_validation_remove: object 0x400000000081d reference count is 1, can't remove
Jun 9 06:05:46.662224 svcstr-7050-acs-4 ERR swss#orchagent: :- removeNeighbor: Failed to remove next hop 192.168.100.4 on Vlan1000, rv:-17
Jun 9 06:05:46.662282 svcstr-7050-acs-4 ERR swss#orchagent: :- handleSaiRemoveStatus: Encountered failure in remove operation, exiting orchagent, SAI API: SAI_API_NEXT_HOP, status: SAI_STATUS_OBJECT_IN_USE
w/fix:
Jun 9 06:03:08.374440 svcstr-7050-acs-3 NOTICE swss#orchagent: :- setState: [Ethernet12] Set MUX state from active to standby
Jun 9 06:03:08.374785 svcstr-7050-acs-3 INFO swss#orchagent: :- stateStandby: Set state to Standby for Ethernet12
Jun 9 06:03:08.375067 svcstr-7050-acs-3 NOTICE swss#orchagent: :- nbrHandler: Processing neighbors for mux Ethernet12, enable 0, state 2
Jun 9 06:03:08.375210 svcstr-7050-acs-3 INFO swss#orchagent: :- updateRoutes: Updating routes pointing to multiple mux nexthops
Jun 9 06:03:08.375553 svcstr-7050-acs-3 NOTICE swss#orchagent: :- updateRoute: Updating route 11.11.11.11/32 pointing to Mux nexthops 192.168.0.2@Vlan1000,192.168.100.4@Vlan1000
Jun 9 06:03:08.375553 svcstr-7050-acs-3 INFO swss#orchagent: :- updateRoute: No Active neighbors found, setting route 11.11.11.11 to point to tun
Jun 9 06:03:08.377446 svcstr-7050-acs-3 INFO swss#orchagent: :- disable: Disabling neigh 192.168.100.4 on Vlan1000
Jun 9 06:03:08.377446 svcstr-7050-acs-3 INFO swss#orchagent: :- updateNextHopRoutes: Route 11.11.11.11/32 is mux multi nexthop route, skipping.
Jun 9 06:03:08.382395 svcstr-7050-acs-3 INFO swss#orchagent: :- addTunnelRoute: Add tunnel route DB 'Vlan1000:192.168.100.4/32'
Jun 9 06:03:08.389626 svcstr-7050-acs-3 NOTICE swss#orchagent: :- create_route: Created tunnel route to 192.168.100.4/32
Jun 9 06:03:08.389626 svcstr-7050-acs-3 NOTICE swss#orchagent: :- disableNeighbor: Neighbor disable request for 192.168.100.4
Jun 9 06:03:08.392097 svcstr-7050-acs-3 NOTICE swss#orchagent: :- removeNeighbor: Removed next hop 192.168.100.4 on Vlan1000
Jun 9 06:03:08.394777 svcstr-7050-acs-3 INFO swss#orchagent: :- decreaseRouterIntfsRefCount: Router interface Vlan1000 ref count is decreased to 93
Jun 9 06:03:08.395137 svcstr-7050-acs-3 INFO swss#orchagent: :- decreaseRouterIntfsRefCount: Router interface Vlan1000 ref count is decreased to 92
Jun 9 06:03:08.395313 svcstr-7050-acs-3 NOTICE swss#orchagent: :- removeNeighbor: Removed neighbor aa:f4:6e:25:d3:03 on Vlan1000
Jun 9 06:03:08.395491 svcstr-7050-acs-3 INFO swss#orchagent: :- disable: Disabling neigh fc02:1000:100::4 on Vlan1000
Jun 9 06:03:08.395645 svcstr-7050-acs-3 INFO swss#orchagent: :- updateNextHopRoutes: No routes found for NH fc02:1000:100::4
Jun 9 06:03:08.395793 svcstr-7050-acs-3 INFO swss#orchagent: :- addTunnelRoute: Add tunnel route DB 'Vlan1000:fc02:1000:100::4/128'
Jun 9 06:03:08.400434 svcstr-7050-acs-3 NOTICE swss#orchagent: :- create_route: Created tunnel route to fc02:1000:100::4/128
Jun 9 06:03:08.400908 svcstr-7050-acs-3 NOTICE swss#orchagent: :- disableNeighbor: Neighbor disable request for fc02:1000:100::4
Jun 9 06:03:08.403559 svcstr-7050-acs-3 NOTICE swss#orchagent: :- removeNeighbor: Removed next hop fc02:1000:100::4 on Vlan1000
Jun 9 06:03:08.406318 svcstr-7050-acs-3 INFO swss#orchagent: :- decreaseRouterIntfsRefCount: Router interface Vlan1000 ref count is decreased to 91
Jun 9 06:03:08.406318 svcstr-7050-acs-3 INFO swss#orchagent: :- decreaseRouterIntfsRefCount: Router interface Vlan1000 ref count is decreased to 90
Jun 9 06:03:08.406318 svcstr-7050-acs-3 NOTICE swss#orchagent: :- removeNeighbor: Removed neighbor aa:f4:6e:25:d3:03 on Vlan1000
Jun 9 06:03:08.409735 svcstr-7050-acs-3 INFO swss#orchagent: :- createMuxAclTable: ACL table IngressTableDrop exists, reuse the same
Jun 9 06:03:08.409735 svcstr-7050-acs-3 NOTICE swss#orchagent: :- MuxAclHandler: Binding port 1000000000005
Jun 9 06:03:08.411635 svcstr-7050-acs-3 INFO swss#orchagent: :- setState: Changed state to standby
@StormLiangMS for viz
mux neighbors that were not the configured mux ip were being treated as active.
cherry-pick of: https://github.com/sonic-net/sonic-swss/pull/3187
What I did use isNeighborActive to determine if neighbor is active for updateRoute logic
Why I did it bug was causing crash when neighbor was not the same as the mux configured neighbor
How I verified it added new vstests and ran locally tests passed on master PR tested on 202305 dualtor testbed: