Target cannot ping the proxy bridge IP

Describe the bug Trying to ping the bridge IP from the target was not working, I could not see anything with tcpdump in the proxy. But pinging the target from the proxy was working. While it pinging the target from the proxy, I tried again pinging the proxy from the target and it worked. I then stopped all pings, and wait 5 seconds, tried again to ping the bridge IP from the target, and it was not working.

Arp from target to bridge IP works fine, even if ping is not working. Ping from target to the peer interface in the proxy works. Ping from the target to another target works.

The problem might be related to the vpp forwarder.

To Reproduce Run these e2e several times: https://github.com/Nordix/Meridio/issues/318 First e2e run: 968970349 Second e2e run: 267521934

Specific test order new-attractor-nsm-vlan -> open-second-stream-second-conduit

Expected behavior Target should be able to ping the bridge IP and should get its nsm interface removed when disconnecting a conduit (https://github.com/Nordix/Meridio/issues/344)

Context

Kernel: 5.15.0
Kubernetes: 1.26.0
Network Service Mesh: v1.7.1 (previous version might have the same problem)
Meridio: v1.0.1

Logs

Target arp table:

? (10.244.1.1) at d6:80:7e:0c:11:11 [ether] on eth0
? (172.16.0.1) at 02:fe:84:8c:f1:46 [ether] on nsm-0

Target interface:

3: nsm-0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN group default qlen 1000
    link/ether 02:fe:26:3c:c8:2c brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.6/24 brd 172.16.0.255 scope global nsm-0
       valid_lft forever preferred_lft forever
    inet 20.0.0.1/32 scope global nsm-0
       valid_lft forever preferred_lft forever
    inet6 2000::1/128 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fd00::6/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::fe:8aff:fee3:1b16/64 scope link 
       valid_lft forever preferred_lft forever

Proxy arp table:

? (172.16.0.6) at 02:fe:26:3c:c8:2c [ether] on bridge0
? (10.244.1.1) at 7e:9e:d4:c6:6c:7a [ether] on eth0

Proxy interface:

3: bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1446 qdisc noqueue state UP group default 
    link/ether 02:fe:84:8c:f1:46 brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.1/24 brd 172.16.0.255 scope global bridge0
       valid_lft forever preferred_lft forever
    inet6 fd00::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::2444:c6ff:fef5:153b/64 scope link 
       valid_lft forever preferred_lft forever
9: proxy.cond-9928: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge0 state UNKNOWN group default qlen 1000
    link/ether 02:fe:8a:5c:e8:08 brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.7/24 brd 172.16.0.255 scope global proxy.cond-9928
       valid_lft forever preferred_lft forever
    inet6 fd00::7/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::fe:8aff:fe5c:e808/64 scope link 
       valid_lft forever preferred_lft forever

vppctl show interface addr:

tap23 (up):
  L2 xconnect tap9
tap9 (up):
  L2 xconnect tap23

vppctl show mode:

l2 xconnect tap23 tap9

TAP interface on the proxy side:

Interface: tap9 (ifindex 13)
  name "proxy.cond-9928"
  host-ns "/proc/1/fd/98"
  host-mac-addr: 02:fe:8a:5c:e8:08
  host-carrier-up: 1
  vhost-fds 60
  tap-fds 59
  gso-enabled 0
  csum-enabled 0
  packet-coalesce 0
  packet-buffering 0
  Mac Address: 02:fe:64:55:48:df
  Device instance: 9
  flags 0x1
    admin-up (0)
  features 0x110008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_F_VERSION_1 (32)
  remote-features 0x33d008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_F_NOTIFY_ON_EMPTY (24)
    VHOST_F_LOG_ALL (26)
    VIRTIO_F_ANY_LAYOUT (27)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_RING_F_EVENT_IDX (29)
    VIRTIO_F_VERSION_1 (32)
    VIRTIO_F_IOMMU_PLATFORM (33)
  Number of RX Virtqueue  1
  Number of TX Virtqueue  1
  Virtqueue (RX) 0
    qsz 1024, last_used_idx 25124, desc_next 448, desc_in_use 924
    avail.flags 0x0 avail.idx 26048 used.flags 0x1 used.idx 25124
    kickfd 62, callfd 61
  Virtqueue (TX) 1
    qsz 1024, last_used_idx 14362, desc_next 27, desc_in_use 1
    avail.flags 0x1 avail.idx 14363 used.flags 0x0 used.idx 14363
    kickfd 63, callfd -1

TAP interface on the target side:

Interface: tap23 (ifindex 33)
  name "nsm-0"
  host-ns "/proc/1/fd/58"
  host-mac-addr: 02:fe:26:3c:c8:2c
  host-carrier-up: 1
  vhost-fds 130
  tap-fds 129
  gso-enabled 0
  csum-enabled 0
  packet-coalesce 0
  packet-buffering 0
  Mac Address: 02:fe:2a:a3:0a:73
  Device instance: 23
  flags 0x1
    admin-up (0)
  features 0x110008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_F_VERSION_1 (32)
  remote-features 0x33d008000
    VIRTIO_NET_F_MRG_RXBUF (15)
    VIRTIO_F_NOTIFY_ON_EMPTY (24)
    VHOST_F_LOG_ALL (26)
    VIRTIO_F_ANY_LAYOUT (27)
    VIRTIO_RING_F_INDIRECT_DESC (28)
    VIRTIO_RING_F_EVENT_IDX (29)
    VIRTIO_F_VERSION_1 (32)
    VIRTIO_F_IOMMU_PLATFORM (33)
  Number of RX Virtqueue  1
  Number of TX Virtqueue  1
  Virtqueue (RX) 0
    qsz 1024, last_used_idx 14363, desc_next 960, desc_in_use 933
    avail.flags 0x0 avail.idx 15296 used.flags 0x1 used.idx 14363
    kickfd 132, callfd 131
  Virtqueue (TX) 1
    qsz 1024, last_used_idx 25123, desc_next 548, desc_in_use 1
    avail.flags 0x1 avail.idx 25124 used.flags 0x0 used.idx 25124
    kickfd 133, callfd -1

Nordix / Meridio

Target cannot ping the proxy bridge IP #372