search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
722 stars 1.38k forks source link

[202311][interface]: Failed to remove router port when IP mask is provided in a full notation #18766

Open nazariig opened 4 months ago

nazariig commented 4 months ago

Signed-off-by: Nazarii Hnydyn nazariig@nvidia.com

Description

CLI/SWSS doesn't have a way to normalize the IP mask notation regardless of what was provided by user. Due to this issue, the router port can't be successfully removed because it is still being referenced.

Steps to reproduce the issue:

  1. Add neighbors 
    root@r-leopard-32:/home/admin# ip -4 neigh replace 10.0.0.1 lladdr 52:54:00:8c:b2:01 nud permanent dev PortChannel102
root@r-leopard-32:/home/admin# ip -6 neigh replace fc00::2 lladdr 52:54:00:8c:b2:01 nud permanent dev PortChannel102
  2. Remove IP 
    root@r-leopard-32:/home/admin# config interface ip remove PortChannel102 10.0.0.0/255.255.255.254
root@r-leopard-32:/home/admin# config interface ip remove PortChannel102 fc00::1/126

Describe the results you received:

Apr 22 17:16:14.040884 r-leopard-58 DEBUG bgp#staticroutebfd: Received message : '('PortChannel102|10.0.0.0/31', 'DEL', ())'
Apr 22 17:16:14.041086 r-leopard-58 DEBUG bgp#staticroutebfd: get ip from intf key: valid True is_ipv4 True, if_name PortChannel102 ip 10.0.0.0
Apr 22 17:16:14.041205 r-leopard-58 DEBUG bgp#bgpcfgd: Received message : '('PortChannel102|10.0.0.0/31', 'DEL', ())'
Apr 22 17:16:14.046610 r-leopard-58 DEBUG bgp#bgpcfgd: Received message : '('PortChannel102|10.0.0.0/31', 'DEL', ())'
Apr 22 17:16:14.046610 r-leopard-58 WARNING bgp#bgpcfgd: Delete key 'PortChannel102|10.0.0.0/31' is not supported for 'zebra set src' templates
Apr 22 17:16:14.047261 r-leopard-58 NOTICE syncd#SDK: [SAI_ROUTE.NOTICE] ./src/mlnx_sai_route.c[875]- mlnx_remove_route: Remove ROUTE_ENTRY 10.0.0.0 255.255.255.255
Apr 22 17:16:14.048293 r-leopard-58 INFO swss#supervisord: message repeated 4 times: [ orchagent ]
Apr 22 17:16:14.048293 r-leopard-58 NOTICE swss#orchagent: :- removeIp2MeRoute: Remove packet action trap route ip:10.0.0.0
Apr 22 17:16:14.055373 r-leopard-58 NOTICE bgp#fpmsyncd: :- onRouteMsg: RouteTable del msg for route with only one nh on eth0/docker0: fd00::/80 :: docker0
Apr 22 17:16:14.056700 r-leopard-58 NOTICE syncd#SDK: [SAI_ROUTE.NOTICE] ./src/mlnx_sai_route.c[1795]- mlnx_route_bulk_api_impl: Remove ROUTE_ENTRY 10.0.0.0 255.255.255.254
Apr 22 17:16:14.056700 r-leopard-58 NOTICE syncd#SDK: [SAI_ROUTE.NOTICE] ./src/mlnx_sai_route.c[875]- mlnx_remove_route: Remove ROUTE_ENTRY 10.0.0.0 255.255.255.254
Apr 22 17:16:14.057271 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_REMOVE] 1 Routes: 1 success, 0 not executed, 0 failed
Apr 22 17:16:14.168576 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[1334]- mlnx_create_next_hop_group: Create NEXT_HOP_GROUP: #0 TYPE=DYNAMIC_ORDERED_ECMP
Apr 22 17:16:14.168576 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[1369]- mlnx_create_next_hop_group: Created NEXT_HOP_GROUP [OID:0x8800000005] [ID:2]
Apr 22 17:16:14.169828 r-leopard-58 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group 10.0.0.5@PortChannel105,10.0.0.9@PortChannel108,10.0.0.13@PortChannel1011
Apr 22 17:16:14.171211 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[4745]- mlnx_create_next_hop_group_members: Create NEXT_HOP_GROUP_MEMBER: #0 NEXT_HOP_GROUP_ID=NEXT_HOP_GROUP [OID:0x8800000005] [ID:2] #1 NEXT_HOP_ID=NEXT_HOP [OID:0x1300000004] [ecmp:0x19] #2 WEIGHT=1 #3 SEQUENCE_ID=1
Apr 22 17:16:14.171282 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[4745]- mlnx_create_next_hop_group_members: Create NEXT_HOP_GROUP_MEMBER: #0 NEXT_HOP_GROUP_ID=NEXT_HOP_GROUP [OID:0x8800000005] [ID:2] #1 NEXT_HOP_ID=NEXT_HOP [OID:0x3200000004] [ecmp:0x50] #2 WEIGHT=1 #3 SEQUENCE_ID=2
Apr 22 17:16:14.171296 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[4745]- mlnx_create_next_hop_group_members: Create NEXT_HOP_GROUP_MEMBER: #0 NEXT_HOP_GROUP_ID=NEXT_HOP_GROUP [OID:0x8800000005] [ID:2] #1 NEXT_HOP_ID=NEXT_HOP [OID:0x3000000004] [ecmp:0x48] #2 WEIGHT=1 #3 SEQUENCE_ID=3
Apr 22 17:16:14.171320 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[4795]- mlnx_create_next_hop_group_members: Created NEXT_HOP_GROUP_MEMBER [OID:0x1090000002D] [ID:4]
Apr 22 17:16:14.171341 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[4795]- mlnx_create_next_hop_group_members: Created NEXT_HOP_GROUP_MEMBER [OID:0x1490000002D] [ID:5]
Apr 22 17:16:14.171362 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[4795]- mlnx_create_next_hop_group_members: Created NEXT_HOP_GROUP_MEMBER [OID:0x1890000002D] [ID:6]
Apr 22 17:16:14.190495 r-leopard-58 NOTICE syncd#SDK: :- threadFunction: time span 19 ms for 'bulkcreate:SAI_OBJECT_TYPE_NEXT_HOP_GROUP_MEMBER:3'
Apr 22 17:16:14.276475 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 169 Routes: 169 success, 0 not executed, 0 failed
Apr 22 17:16:15.036219 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 1000 Routes: 1000 success, 0 not executed, 0 failed
Apr 22 17:16:15.190657 r-leopard-58 NOTICE syncd#SDK: :- threadFunction: time span 103 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:534'
Apr 22 17:16:15.361147 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 534 Routes: 534 success, 0 not executed, 0 failed
Apr 22 17:16:16.136844 r-leopard-58 NOTICE syncd#SDK: [SAI_ROUTE.NOTICE] ./src/mlnx_sai_route.c[1795]- mlnx_route_bulk_api_impl: Remove ROUTE_ENTRY 100.1.0.1 255.255.255.255
Apr 22 17:16:16.136844 r-leopard-58 NOTICE syncd#SDK: [SAI_ROUTE.NOTICE] ./src/mlnx_sai_route.c[875]- mlnx_remove_route: Remove ROUTE_ENTRY 100.1.0.1 255.255.255.255
Apr 22 17:16:16.137804 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_REMOVE] 1 Routes: 1 success, 0 not executed, 0 failed
Apr 22 17:16:16.190801 r-leopard-58 NOTICE syncd#SDK: :- threadFunction: time span 32 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Apr 22 17:16:16.725900 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 1000 Routes: 1000 success, 0 not executed, 0 failed
Apr 22 17:16:17.190945 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 1000 Routes: 1000 success, 0 not executed, 0 failed
Apr 22 17:16:17.190945 r-leopard-58 NOTICE syncd#SDK: :- threadFunction: time span 3 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Apr 22 17:16:17.663273 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 1000 Routes: 1000 success, 0 not executed, 0 failed
Apr 22 17:16:17.745227 r-leopard-58 NOTICE syncd#SDK: :- inc: 10453 (calls 17) Syncd::syncUpdateRedisBulkQuadEvent op took: 541 ms
Apr 22 17:16:18.191056 r-leopard-58 NOTICE syncd#SDK: :- threadFunction: time span 436 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Apr 22 17:16:18.325186 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 1000 Routes: 1000 success, 0 not executed, 0 failed
Apr 22 17:16:18.793577 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[1929]- set_dispatch_attrib_handler: Set PACKET_ACTION, key:ROUTE_ENTRY 0.0.0.0 0.0.0.0, val:FORWARD
Apr 22 17:16:18.794890 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 667 Routes: 667 success, 0 not executed, 0 failed
Apr 22 17:16:18.922260 r-leopard-58 NOTICE swss#orchagent: :- addRoutePost: Update Nexthop Group 10.0.0.1@PortChannel102,10.0.0.5@PortChannel105,10.0.0.9@PortChannel108,10.0.0.13@PortChannel1011
Apr 22 17:16:18.922260 r-leopard-58 NOTICE swss#orchagent: :- removeNextHopGroup: Delete next hop group 10.0.0.1@PortChannel102,10.0.0.5@PortChannel105,10.0.0.9@PortChannel108,10.0.0.13@PortChannel1011
Apr 22 17:16:18.923750 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[5042]- mlnx_remove_next_hop_group_members: Remove NEXT_HOP_GROUP_MEMBER [OID:0x2890000002D] [ID:10]
Apr 22 17:16:18.923750 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[5042]- mlnx_remove_next_hop_group_members: Remove NEXT_HOP_GROUP_MEMBER [OID:0x2490000002D] [ID:9]
Apr 22 17:16:18.923884 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[5042]- mlnx_remove_next_hop_group_members: Remove NEXT_HOP_GROUP_MEMBER [OID:0x490000002D] [ID:1]
Apr 22 17:16:18.923884 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[5042]- mlnx_remove_next_hop_group_members: Remove NEXT_HOP_GROUP_MEMBER [OID:0x90000002D] [ID:0]
Apr 22 17:16:18.925400 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[1502]- mlnx_remove_next_hop_group: Remove NEXT_HOP_GROUP [OID:0x800000005] [ID:0]
Apr 22 17:16:18.932185 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP.NOTICE] ./src/mlnx_sai_nexthop.c[1413]- mlnx_remove_next_hop: Remove NEXT_HOP [OID:0x1000000004] [ecmp:0x16]
Apr 22 17:16:18.933335 r-leopard-58 NOTICE swss#orchagent: :- removeNeighbor: Removed next hop 10.0.0.1 on PortChannel102
Apr 22 17:16:18.933720 r-leopard-58 NOTICE syncd#SDK: [SAI_NEIGHBOR.NOTICE] ./src/mlnx_sai_neighbor.c[309]- mlnx_remove_neighbor_entry: Remove NEIGHBOR_ENTRY 10.0.0.1RIF:0x30100000006
Apr 22 17:16:18.934648 r-leopard-58 NOTICE swss#orchagent: :- removeNeighbor: Removed neighbor 4a:12:0f:ff:7a:a1 on PortChannel102
Apr 22 17:16:18.935253 r-leopard-58 INFO swss#supervisord: orchagent
Apr 22 17:16:22.609717 r-leopard-58 NOTICE rsyslog_plugin: :- publish: EVENT_PUBLISHED: {"sonic-events-bgp:notification":{"ip":"10.0.0.1","is_sent":"true","major_code":"4","minor_code":"0","timestamp":"2024-04-22T17:16:22.608515Z"}}

Apr 22 17:16:35.434614 r-leopard-58 DEBUG bgp#staticroutebfd: Received message : '('PortChannel102|FC00::1/126', 'DEL', ())'
Apr 22 17:16:35.434614 r-leopard-58 DEBUG bgp#bgpcfgd: Received message : '('PortChannel102|FC00::1/126', 'DEL', ())'
Apr 22 17:16:35.434687 r-leopard-58 DEBUG bgp#staticroutebfd: get ip from intf key: valid True is_ipv4 False, if_name PortChannel102 ip FC00::1
Apr 22 17:16:35.441490 r-leopard-58 DEBUG bgp#bgpcfgd: Received message : '('PortChannel102|FC00::1/126', 'DEL', ())'
Apr 22 17:16:35.441490 r-leopard-58 WARNING bgp#bgpcfgd: Delete key 'PortChannel102|FC00::1/126' is not supported for 'zebra set src' templates
Apr 22 17:16:35.442028 r-leopard-58 NOTICE syncd#SDK: [SAI_ROUTE.NOTICE] ./src/mlnx_sai_route.c[875]- mlnx_remove_route: Remove ROUTE_ENTRY fc00::1 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
Apr 22 17:16:35.442836 r-leopard-58 INFO swss#supervisord: orchagent
Apr 22 17:16:35.442836 r-leopard-58 NOTICE swss#orchagent: :- removeIp2MeRoute: Remove packet action trap route ip:fc00::1
Apr 22 17:16:35.443303 r-leopard-58 DEBUG bgp#staticroutebfd: Received message : '('PortChannel102', 'DEL', ())'
Apr 22 17:16:35.443303 r-leopard-58 DEBUG bgp#bgpcfgd: Received message : '('PortChannel102', 'DEL', ())'
Apr 22 17:16:35.447337 r-leopard-58 NOTICE bgp#fpmsyncd: :- onRouteMsg: RouteTable del msg for route with only one nh on eth0/docker0: fd00::/80 :: docker0
Apr 22 17:16:35.449115 r-leopard-58 NOTICE syncd#SDK: [SAI_ROUTE.NOTICE] ./src/mlnx_sai_route.c[1795]- mlnx_route_bulk_api_impl: Remove ROUTE_ENTRY fc00:: ffff:ffff:ffff:ffff:ffff:ffff:ffff:fffc
Apr 22 17:16:35.449115 r-leopard-58 NOTICE syncd#SDK: [SAI_ROUTE.NOTICE] ./src/mlnx_sai_route.c[875]- mlnx_remove_route: Remove ROUTE_ENTRY fc00:: ffff:ffff:ffff:ffff:ffff:ffff:ffff:fffc
Apr 22 17:16:35.449729 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_REMOVE] 1 Routes: 1 success, 0 not executed, 0 failed
Apr 22 17:16:35.451735 r-leopard-58 DEBUG bgp#bgpcfgd: Received message : '('PortChannel102', 'DEL', ())'
Apr 22 17:16:35.451735 r-leopard-58 WARNING bgp#bgpcfgd: Delete key 'PortChannel102' is not supported for 'zebra set src' templates
Apr 22 17:16:35.451972 r-leopard-58 NOTICE swss#orchagent: :- removeRouterIntfs: Router interface PortChannel102 is still referenced with ref count 2
Apr 22 17:16:35.572637 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[1334]- mlnx_create_next_hop_group: Create NEXT_HOP_GROUP: #0 TYPE=DYNAMIC_ORDERED_ECMP
Apr 22 17:16:35.572637 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[1369]- mlnx_create_next_hop_group: Created NEXT_HOP_GROUP [OID:0x800000005] [ID:0]
Apr 22 17:16:35.573480 r-leopard-58 NOTICE swss#orchagent: message repeated 4 times: [ :- removeRouterIntfs: Router interface PortChannel102 is still referenced with ref count 2]
Apr 22 17:16:35.573480 r-leopard-58 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group fc00::6@PortChannel105,fc00::a@PortChannel108,fc00::e@PortChannel1011
Apr 22 17:16:35.593149 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[4745]- mlnx_create_next_hop_group_members: Create NEXT_HOP_GROUP_MEMBER: #0 NEXT_HOP_GROUP_ID=NEXT_HOP_GROUP [OID:0x800000005] [ID:0] #1 NEXT_HOP_ID=NEXT_HOP [OID:0x2B00000004] [ecmp:0x43] #2 WEIGHT=1 #3 SEQUENCE_ID=1
Apr 22 17:16:35.593149 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[4745]- mlnx_create_next_hop_group_members: Create NEXT_HOP_GROUP_MEMBER: #0 NEXT_HOP_GROUP_ID=NEXT_HOP_GROUP [OID:0x800000005] [ID:0] #1 NEXT_HOP_ID=NEXT_HOP [OID:0x3300000004] [ecmp:0x51] #2 WEIGHT=1 #3 SEQUENCE_ID=2
Apr 22 17:16:35.593149 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[4745]- mlnx_create_next_hop_group_members: Create NEXT_HOP_GROUP_MEMBER: #0 NEXT_HOP_GROUP_ID=NEXT_HOP_GROUP [OID:0x800000005] [ID:0] #1 NEXT_HOP_ID=NEXT_HOP [OID:0x1700000004] [ecmp:0x23] #2 WEIGHT=1 #3 SEQUENCE_ID=3
Apr 22 17:16:35.593149 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[4795]- mlnx_create_next_hop_group_members: Created NEXT_HOP_GROUP_MEMBER [OID:0x90000002D] [ID:0]
Apr 22 17:16:35.593149 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[4795]- mlnx_create_next_hop_group_members: Created NEXT_HOP_GROUP_MEMBER [OID:0x490000002D] [ID:1]
Apr 22 17:16:35.593149 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[4795]- mlnx_create_next_hop_group_members: Created NEXT_HOP_GROUP_MEMBER [OID:0x1C90000002D] [ID:7]
Apr 22 17:16:35.681402 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 156 Routes: 156 success, 0 not executed, 0 failed
Apr 22 17:16:35.688857 r-leopard-58 NOTICE swss#orchagent: :- removeRouterIntfs: Router interface PortChannel102 is still referenced with ref count 2
Apr 22 17:16:35.710038 r-leopard-58 NOTICE swss#orchagent: :- doTask: Removed pending neighbor DEL operation for PortChannel102:fc00::2 after SET operation
Apr 22 17:16:35.710038 r-leopard-58 NOTICE swss#orchagent: :- removeRouterIntfs: Router interface PortChannel102 is still referenced with ref count 2
Apr 22 17:16:36.192416 r-leopard-58 NOTICE syncd#SDK: :- threadFunction: time span 88 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Apr 22 17:16:36.556865 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 1000 Routes: 1000 success, 0 not executed, 0 failed
Apr 22 17:16:36.982021 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 788 Routes: 788 success, 0 not executed, 0 failed
Apr 22 17:16:37.031345 r-leopard-58 NOTICE phc_ctl: [7269.540] set clock time to 1713795397.030922012 or Mon Apr 22 14:16:37 2024
Apr 22 17:16:37.031555 r-leopard-58 INFO supervisord: phcsync phc_ctl[7269.540]: set clock time to 1713795397.030922012 or Mon Apr 22 14:16:37 2024
Apr 22 17:16:37.033098 r-leopard-58 NOTICE phc_ctl: [7269.542] set clock time to 1713795397.032784328 or Mon Apr 22 14:16:37 2024
Apr 22 17:16:37.033298 r-leopard-58 INFO supervisord: phcsync phc_ctl[7269.542]: set clock time to 1713795397.032784328 or Mon Apr 22 14:16:37 2024
Apr 22 17:16:37.755686 r-leopard-58 NOTICE syncd#SDK: [SAI_ROUTE.NOTICE] ./src/mlnx_sai_route.c[1795]- mlnx_route_bulk_api_impl: Remove ROUTE_ENTRY 2064:100::1 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
Apr 22 17:16:37.755686 r-leopard-58 NOTICE syncd#SDK: [SAI_ROUTE.NOTICE] ./src/mlnx_sai_route.c[875]- mlnx_remove_route: Remove ROUTE_ENTRY 2064:100::1 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
Apr 22 17:16:37.756662 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_REMOVE] 1 Routes: 1 success, 0 not executed, 0 failed
Apr 22 17:16:38.192496 r-leopard-58 NOTICE syncd#SDK: :- threadFunction: time span 417 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:1000'
Apr 22 17:16:38.198338 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 1000 Routes: 1000 success, 0 not executed, 0 failed
Apr 22 17:16:39.032021 r-leopard-58 NOTICE syncd#SDK: message repeated 2 times: [ [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 1000 Routes: 1000 success, 0 not executed, 0 failed]
Apr 22 17:16:39.032021 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[1929]- set_dispatch_attrib_handler: Set PACKET_ACTION, key:ROUTE_ENTRY :: ::, val:FORWARD
Apr 22 17:16:39.134936 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 1000 Routes: 1000 success, 0 not executed, 0 failed
Apr 22 17:16:39.192578 r-leopard-58 NOTICE syncd#SDK: :- threadFunction: time span 25 ms for 'bulkset:SAI_OBJECT_TYPE_ROUTE_ENTRY:426'
Apr 22 17:16:39.281579 r-leopard-58 NOTICE syncd#SDK: [SAI_UTILS.NOTICE] ./src/mlnx_sai_utils.c[2337]- mlnx_bulk_statuses_print: [BULK_SET] 426 Routes: 426 success, 0 not executed, 0 failed
Apr 22 17:16:39.364569 r-leopard-58 NOTICE swss#orchagent: message repeated 3 times: [ :- removeRouterIntfs: Router interface PortChannel102 is still referenced with ref count 2]
Apr 22 17:16:39.364569 r-leopard-58 NOTICE swss#orchagent: :- addRoutePost: Update Nexthop Group fc00::2@PortChannel102,fc00::6@PortChannel105,fc00::a@PortChannel108,fc00::e@PortChannel1011
Apr 22 17:16:39.364735 r-leopard-58 NOTICE swss#orchagent: :- removeNextHopGroup: Delete next hop group fc00::2@PortChannel102,fc00::6@PortChannel105,fc00::a@PortChannel108,fc00::e@PortChannel1011
Apr 22 17:16:39.365748 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[5042]- mlnx_remove_next_hop_group_members: Remove NEXT_HOP_GROUP_MEMBER [OID:0x3890000002D] [ID:14]
Apr 22 17:16:39.365795 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[5042]- mlnx_remove_next_hop_group_members: Remove NEXT_HOP_GROUP_MEMBER [OID:0x3490000002D] [ID:13]
Apr 22 17:16:39.365817 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[5042]- mlnx_remove_next_hop_group_members: Remove NEXT_HOP_GROUP_MEMBER [OID:0x3090000002D] [ID:12]
Apr 22 17:16:39.365817 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[5042]- mlnx_remove_next_hop_group_members: Remove NEXT_HOP_GROUP_MEMBER [OID:0x2C90000002D] [ID:11]
Apr 22 17:16:39.367596 r-leopard-58 NOTICE syncd#SDK: [SAI_NEXT_HOP_GROUP.NOTICE] ./src/mlnx_sai_nexthopgroup.c[1502]- mlnx_remove_next_hop_group: Remove NEXT_HOP_GROUP [OID:0x10800000005] [ID:4]
Apr 22 17:16:39.373216 r-leopard-58 NOTICE swss#orchagent: :- removeRouterIntfs: Router interface PortChannel102 is still referenced with ref count 2
Apr 22 17:16:39.501425 r-leopard-58 NOTICE swss#orchagent: message repeated 1578 times: [ :- removeRouterIntfs: Router interface PortChannel102 is still referenced with ref count 2]
Apr 22 17:16:39.501425 r-leopard-58 INFO swss#supervisord: orchagent
Apr 22 17:16:39.501425 r-leopard-58 NOTICE swss#orchagent: :- removeRouterIntfs: Router interface PortChannel102 is still referenced with ref count 2
Apr 22 17:16:44.638087 r-leopard-58 NOTICE rsyslog_plugin: :- publish: EVENT_PUBLISHED: {"sonic-events-bgp:notification":{"ip":"fc00::2","is_sent":"true","major_code":"4","minor_code":"0","timestamp":"2024-04-22T17:16:44.636990Z"}}
Apr 22 17:16:49.838252 r-leopard-58 NOTICE swss#orchagent: message repeated 4816 times: [ :- removeRouterIntfs: Router interface PortChannel102 is still referenced with ref count 2]
Apr 22 17:16:49.838252 r-leopard-58 INFO swss#supervisord: orchagent
Apr 22 17:16:50.501500 r-leopard-58 NOTICE swss#orchagent: :- removeRouterIntfs: Router interface PortChannel102 is still referenced with ref count 2

Describe the results you expected:

CLI/SWSS should either normalize the mask or raise a configuration error

Output of show version:

Output of show techsupport:

Additional information you deem important (e.g. issue happens only occasionally):

bingwang-ms commented 4 months ago

As discussed in the meeting, the issue happened when full netmask is provided when removing the assigned IP.

bingwang-ms commented 4 months ago

@prsunny Can you please help check this issue?

prsunny commented 4 months ago

@nazariig , looks like this could be fixed by this PR - https://github.com/sonic-net/sonic-utilities/pull/3281. Can you check?

@saiarcot895 for viz

nazariig commented 4 months ago

@nazariig , looks like this could be fixed by this PR - sonic-net/sonic-utilities#3281. Can you check?

@saiarcot895 for viz

@prsunny It doesn't solve the issue. I would expect to have a mask validation (normalization) in two places: CLI and SWSS (in case config is pushed via controller)