search

ligato / vpp-agent

⚡️ Control plane management agent for FD.io's VPP
https://docs.ligato.io/
Apache License 2.0
252 stars 125 forks source link

configurator.ConfiguratorClient.Dump()/configurator.ConfiguratorClient.Get() returns strange results #1525

Closed denis-tingaikin closed 4 years ago

denis-tingaikin commented 5 years ago

Steps to reproduce:

Case 1

  1. Use vppagent version 2.3.0 and use startup-config: https://github.com/networkservicemesh/networkservicemesh/blob/master/dataplane/vppagent/conf/vpp/startup.conf
  2. Create configurator.ConfiguratorClient with grpc connection via TCP localhost:9111
  3. Do client.Update(config) where config is 
    vpp_config: <
interfaces: <
name: "SRC-1"
type: TAP
enabled: true
tap: <
  version: 2
>
>
interfaces: <
name: "DST-1"
type: TAP
enabled: true
tap: <
  version: 2
>
>
xconnect_pairs: <
receive_interface: "SRC-1"
transmit_interface: "DST-1"
>
xconnect_pairs: <
receive_interface: "DST-1"
transmit_interface: "SRC-1"
>
>
linux_config: <
interfaces: <
name: "SRC-1"
type: TAP_TO_VPP
namespace: <
  type: FD
  reference: "/proc/7477/ns/net"
>
host_if_name: "nsm0"
enabled: true
ip_addresses: "172.16.1.1/30"
tap: <
  vpp_tap_if_name: "SRC-1"
>
>
interfaces: <
name: "DST-1"
type: TAP_TO_VPP
namespace: <
  type: FD
  reference: "/proc/7192/ns/net"
>
host_if_name: "nsm9QjSeyg9"
enabled: true
ip_addresses: "172.16.1.2/30"
phys_address: "2a:42:a5:73:90:97"
tap: <
  vpp_tap_if_name: "DST-1"
>
>
routes: <
outgoing_interface: "SRC-1"
scope: GLOBAL
dst_network: "8.8.8.8/30"
gw_addr: "172.16.1.2"
>
>

Notice: Interface DST-1 has phys_address: 2a:42:a5:73:90:97

  1. Do client.Dump() 
    <interfaces:<type:SOFTWARE_LOOPBACK > interfaces:<name:\"mgmt\" type:AF_PACKET enabled:true phys_address:\"08:00:27:24:6d:d8\" ip_addresses:\"172.28.128.35/24\" rx_modes:<mode:INTERRUPT > rx_placements:<main_thread:true > afpacket:<host_if_name:\"eth1\" > > interfaces:<name:\"DST-1\" type:TAP enabled:true phys_address:\"02:fe:b0:95:4e:b9\" rx_modes:<mode:POLLING > rx_placements:<main_thread:true > tap:<version:2 host_if_name:\"tap-1649917878\" rx_ring_size:256 tx_ring_size:256 > > interfaces:<name:\"SRC-1\" type:TAP enabled:true phys_address:\"02:fe:e6:f0:e5:e8\" rx_modes:<mode:POLLING > rx_placements:<main_thread:true > tap:<version:2 host_if_name:\"tap-1015135623\" rx_ring_size:256 tx_ring_size:256 > > acls:<name:\"NSMmgmtInterfaceACL\" rules:<action:PERMIT ip_rule:<ip:<destination_network:\"172.28.128.35/32\" source_network:\"0.0.0.0/0\" > udp:<destination_port_range:<lower_port:4789 upper_port:4789 > source_port_range:<upper_port:65535 > > > > interfaces:<ingress:\"mgmt\" > > xconnect_pairs:<receive_interface:\"SRC-1\" transmit_interface:\"DST-1\" > xconnect_pairs:<receive_interface:\"DST-1\" transmit_interface:\"SRC-1\" > routes:<type:DROP dst_network:\"::/0\" next_hop_addr:\"::\" weight:1 > routes:<dst_network:\"fe80::/10\" next_hop_addr:\"::\" weight:1 > routes:<dst_network:\"0.0.0.0/0\" next_hop_addr:\"10.0.2.2\" outgoing_interface:\"mgmt\" weight:1 > routes:<type:DROP dst_network:\"240.0.0.0/4\" next_hop_addr:\"0.0.0.0\" weight:1 > routes:<type:DROP dst_network:\"224.0.0.0/4\" next_hop_addr:\"0.0.0.0\" weight:1 > routes:<dst_network:\"172.28.128.35/24\" next_hop_addr:\"0.0.0.0\" outgoing_interface:\"mgmt\" weight:1 > routes:<dst_network:\"10.32.0.6/32\" next_hop_addr:\"10.32.0.6\" outgoing_interface:\"mgmt\" weight:1 > routes:<dst_network:\"10.44.0.5/32\" next_hop_addr:\"10.44.0.5\" outgoing_interface:\"mgmt\" weight:1 > routes:<type:DROP dst_network:\"172.28.128.0/32\" next_hop_addr:\"0.0.0.0\" weight:1 > routes:<dst_network:\"10.32.0.5/32\" next_hop_addr:\"10.32.0.5\" outgoing_interface:\"mgmt\" weight:1 > routes:<dst_network:\"10.32.0.4/32\" next_hop_addr:\"10.32.0.4\" outgoing_interface:\"mgmt\" weight:1 > routes:<dst_network:\"172.28.128.1/32\" next_hop_addr:\"172.28.128.1\" outgoing_interface:\"mgmt\" weight:1 > routes:<dst_network:\"10.32.0.3/32\" next_hop_addr:\"10.32.0.3\" outgoing_interface:\"mgmt\" weight:1 > routes:<dst_network:\"10.44.0.2/32\" next_hop_addr:\"10.44.0.2\" outgoing_interface:\"mgmt\" weight:1 > routes:<type:DROP dst_network:\"0.0.0.0/32\" next_hop_addr:\"0.0.0.0\" weight:1 > routes:<dst_network:\"172.28.128.2/32\" next_hop_addr:\"172.28.128.2\" outgoing_interface:\"mgmt\" weight:1 > routes:<dst_network:\"10.32.0.2/32\" next_hop_addr:\"10.32.0.2\" outgoing_interface:\"mgmt\" weight:1 > routes:<type:DROP dst_network:\"255.255.255.255/32\" next_hop_addr:\"0.0.0.0\" weight:1 > routes:<dst_network:\"172.28.128.35/32\" next_hop_addr:\"172.28.128.35\" outgoing_interface:\"mgmt\" weight:1 > routes:<dst_network:\"10.0.2.2/32\" next_hop_addr:\"10.0.2.2\" outgoing_interface:\"mgmt\" weight:1 > routes:<dst_network:\"10.32.0.7/32\" next_hop_addr:\"10.32.0.7\" outgoing_interface:\"mgmt\" weight:1 > routes:<dst_network:\"172.28.128.36/32\" next_hop_addr:\"172.28.128.36\" outgoing_interface:\"mgmt\" weight:1 > routes:<type:DROP dst_network:\"172.28.128.255/32\" next_hop_addr:\"0.0.0.0\" weight:1 > routes:<dst_network:\"10.0.2.3/32\" next_hop_addr:\"10.0.2.3\" outgoing_interface:\"mgmt\" weight:1 > routes:<dst_network:\"10.44.0.6/32\" next_hop_addr:\"10.44.0.6\" outgoing_interface:\"mgmt\" weight:1 > arps:<interface:\"mgmt\" ip_address:\"172.28.128.1\" phys_address:\"0a:00:27:00:00:00\" > arps:<interface:\"mgmt\" ip_address:\"172.28.128.2\" phys_address:\"08:00:27:d6:53:87\" > arps:<interface:\"mgmt\" ip_address:\"172.28.128.36\" phys_address:\"08:00:27:0a:e4:5d\" > arps:<interface:\"mgmt\" ip_address:\"10.0.2.2\" phys_address:\"52:54:00:12:35:02\" > arps:<interface:\"mgmt\" ip_address:\"10.0.2.3\" phys_address:\"52:54:00:12:35:03\" > arps:<interface:\"mgmt\" ip_address:\"10.32.0.2\" phys_address:\"16:5d:7c:cc:3a:ef\" > arps:<interface:\"mgmt\" ip_address:\"10.32.0.3\" phys_address:\"9a:8b:00:b7:86:36\" > arps:<interface:\"mgmt\" ip_address:\"10.32.0.4\" phys_address:\"9a:a5:ec:88:0b:99\" > arps:<interface:\"mgmt\" ip_address:\"10.32.0.5\" phys_address:\"56:03:29:c4:8f:b7\" > arps:<interface:\"mgmt\" ip_address:\"10.32.0.6\" phys_address:\"6e:91:bd:c4:ff:90\" > arps:<interface:\"mgmt\" ip_address:\"10.32.0.7\" phys_address:\"2e:f8:fa:fa:60:0f\" > arps:<interface:\"mgmt\" ip_address:\"10.44.0.2\" phys_address:\"e6:7d:4a:90:3a:00\" > arps:<interface:\"mgmt\" ip_address:\"10.44.0.5\" phys_address:\"b2:b7:f3:5a:9e:3f\" > arps:<interface:\"mgmt\" ip_address:\"10.44.0.6\" phys_address:\"fe:3b:2e:bc:ed:fd\" > nat44_global:<virtual_reassembly:<timeout:2 max_reassemblies:1024 max_fragments:5 > > > linux_config:<> netalloc_config:<>

    Notice: interfaces: DST-1 has phys_address:\"02:fe:b0:95:4e:b9\",

Actual: Dump request returns strange phys_address. Expected: Dump request returns actual phys_address.

Case 2

  1. Use vppagnet version 2.3.0 and use startup-config: https://github.com/networkservicemesh/networkservicemesh/blob/master/dataplane/vppagent/conf/vpp/startup.conf 2) Create configurator.ConfiguratorClient with grpc connection via TCP localhost:9111 3) Do client.Update(config) where config is 

    vpp_config:<interfaces:<name:\"SRC-1\" type:TAP enabled:true tap:<version:2 > > interfaces:<name:\"DST-1\" type:TAP enabled:true tap:<version:2 > > xconnect_pairs:<receive_interface:\"SRC-1\" transmit_interface:\"DST-1\" > xconnect_pairs:<receive_interface:\"DST-1\" transmit_interface:\"SRC-1\" > > linux_config:<interfaces:<name:\"SRC-1\" type:TAP_TO_VPP namespace:<type:FD reference:\"/proc/7477/ns/net\" > host_if_name:\"nsm0\" enabled:true ip_addresses:\"172.16.1.1/30\" tap:<vpp_tap_if_name:\"SRC-1\" > > interfaces:<name:\"DST-1\" type:TAP_TO_VPP namespace:<type:FD reference:\"/proc/7192/ns/net\" > host_if_name:\"nsm9QjSeyg9\" enabled:true ip_addresses:\"172.16.1.2/30\" tap:<vpp_tap_if_name:\"DST-1\" > > routes:<outgoing_interface:\"SRC-1\" scope:GLOBAL dst_network:\"8.8.8.8/30\" gw_addr:\"172.16.1.2\" > >

    Notice: Interface DST-1 has not phys_address. Let vpp generate it.

    4) Do client.Dump() 5) Check phys address of the interface in target container.

Actual: Dump request returns strange phys_address. Expected: Dump request returns actual phys_address.

Logs

1) Get response for case 1 is

Get response: config:<vpp_config:<interfaces:<name:\"SRC-1\" type:TAP enabled:true tap:<version:2 > > interfaces:<name:\"DST-1\" type:TAP enabled:true tap:<version:2 > > interfaces:<name:\"mgmt\" type:AF_PACKET enabled:true phys_address:\"08:00:27:24:6d:d8\" ip_addresses:\"172.28.128.35/24\" afpacket:<host_if_name:\"eth1\" > > acls:<name:\"NSMmgmtInterfaceACL\" rules:<action:PERMIT ip_rule:<ip:<destination_network:\"172.28.128.35/32\" source_network:\"0.0.0.0/0\" > udp:<destination_port_range:<lower_port:4789 upper_port:4789 > source_port_range:<upper_port:65535 > > > > interfaces:<ingress:\"mgmt\" > > xconnect_pairs:<receive_interface:\"DST-1\" transmit_interface:\"SRC-1\" > xconnect_pairs:<receive_interface:\"SRC-1\" transmit_interface:\"DST-1\" > routes:<type:INTER_VRF dst_network:\"0.0.0.0/0\" next_hop_addr:\"10.0.2.2\" outgoing_interface:\"mgmt\" weight:1 > arps:<interface:\"mgmt\" ip_address:\"10.32.0.3\" phys_address:\"9a:8b:00:b7:86:36\" > arps:<interface:\"mgmt\" ip_address:\"10.44.0.2\" phys_address:\"e6:7d:4a:90:3a:00\" > arps:<interface:\"mgmt\" ip_address:\"10.32.0.6\" phys_address:\"6e:91:bd:c4:ff:90\" > arps:<interface:\"mgmt\" ip_address:\"10.32.0.5\" phys_address:\"56:03:29:c4:8f:b7\" > arps:<interface:\"mgmt\" ip_address:\"10.32.0.7\" phys_address:\"2e:f8:fa:fa:60:0f\" > arps:<interface:\"mgmt\" ip_address:\"10.0.2.2\" phys_address:\"52:54:00:12:35:02\" > arps:<interface:\"mgmt\" ip_address:\"172.28.128.36\" phys_address:\"08:00:27:0a:e4:5d\" > arps:<interface:\"mgmt\" ip_address:\"10.44.0.5\" phys_address:\"b2:b7:f3:5a:9e:3f\" > arps:<interface:\"mgmt\" ip_address:\"10.0.2.3\" phys_address:\"52:54:00:12:35:03\" > arps:<interface:\"mgmt\" ip_address:\"172.28.128.2\" phys_address:\"08:00:27:d6:53:87\" > arps:<interface:\"mgmt\" ip_address:\"10.32.0.4\" phys_address:\"9a:a5:ec:88:0b:99\" > arps:<interface:\"mgmt\" ip_address:\"10.44.0.6\" phys_address:\"fe:3b:2e:bc:ed:fd\" > arps:<interface:\"mgmt\" ip_address:\"172.28.128.1\" phys_address:\"0a:00:27:00:00:00\" > arps:<interface:\"mgmt\" ip_address:\"10.32.0.2\" phys_address:\"16:5d:7c:cc:3a:ef\" > > linux_config:<interfaces:<name:\"DST-1\" type:TAP_TO_VPP namespace:<type:FD reference:\"/proc/7192/ns/net\" > host_if_name:\"nsm9QjSeyg9\" enabled:true ip_addresses:\"172.16.1.2/30\" phys_address:\"2a:42:a5:73:90:97\" tap:<vpp_tap_if_name:\"DST-1\" > > interfaces:<name:\"SRC-1\" type:TAP_TO_VPP namespace:<type:FD reference:\"/proc/7477/ns/net\" > host_if_name:\"nsm0\" enabled:true ip_addresses:\"172.16.1.1/30\" tap:<vpp_tap_if_name:\"SRC-1\" > > routes:<outgoing_interface:\"SRC-1\" scope:GLOBAL dst_network:\"8.8.8.8/30\" gw_addr:\"172.16.1.2\" > > netalloc_config:<> >
  1. Full logs of container with vpp: nsmd-dataplane-kube-master-vppagent-dataplane.log
edwarnicke commented 5 years ago

@ondrej-fabry Could you have a look?

rewenset commented 5 years ago

So I did a little bit of parsing for Case 1.

3. Do client.Update(config) where config is

``` vpp_config:< interfaces: > interfaces: > xconnect_pairs: xconnect_pairs: > linux_config:< interfaces:< name:\"SRC-1\" type:TAP_TO_VPP namespace: host_if_name:\"nsm0\" enabled:true ip_addresses:\"172.16.1.1/30\" tap: > interfaces:< name:\"DST-1\" type:TAP_TO_VPP namespace: host_if_name:\"nsm9QjSeyg9\" enabled:true ip_addresses:\"172.16.1.2/30\" phys_address:\"2a:42:a5:73:90:97\" tap: > routes:< outgoing_interface:\"SRC-1\" scope:GLOBAL dst_network:\"8.8.8.8/30\" gw_addr:\"172.16.1.2\" > > ```

4. Do client.Dump()

``` < interfaces: interfaces:< name:\"mgmt\" type:AF_PACKET enabled:true phys_address:\"08:00:27:24:6d:d8\" ip_addresses:\"172.28.128.35/24\" rx_modes: rx_placements: afpacket: > interfaces:< name:\"DST-1\" type:TAP enabled:true phys_address:\"02:fe:b0:95:4e:b9\" rx_modes: rx_placements: tap: > interfaces:< name:\"SRC-1\" type:TAP enabled:true phys_address:\"02:fe:e6:f0:e5:e8\" rx_modes: rx_placements: tap: > acls:< name:\"NSMmgmtInterfaceACL\" rules:< action:PERMIT ip_rule:< ip:< destination_network:\"172.28.128.35/32\" source_network:\"0.0.0.0/0\" > udp:< destination_port_range: source_port_range: > > > interfaces: > xconnect_pairs: xconnect_pairs: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: routes: arps: arps: arps: arps: arps: arps: arps: arps: arps: arps: arps: arps: arps: arps: nat44_global: > > linux_config:<> netalloc_config:<> ```

And it is looks like you are expecting to see phys_address set for linux side of TAP interface to be also phys_address of VPP side of TAP. Or am I missing something here?

rewenset commented 5 years ago

yeah, I think, it is not working this way. In client.Dump() response you have vpp_config which will tell you about VPP side of TAP and linux_config which contains info about Linux side of TAP.

If you are setting phys_address for Linux side, then you may check it in linux_config and in the container (or whatever this interface was put) and expect it to match.

For the Case 1, you are setting 2a:42:a5:73:90:97 for linux interface, and not for vpp side, but in config you see VPP generated address 02:fe:b0:95:4e:b9 for interface in VPP and it's ok that they are different. (I'm not sure why linux_config is empty though)

And as for the Case 2, I think you've compared address of interface in VPP with address of interface in target container and again it's ok that they are different.

One more thing, I think, it would be more practical to give TAP interfaces names like awesomeLinuxTap and awesomeVPPTap, so no confusion in future will happen.

ondrej-fabry commented 5 years ago

@rewenset is correct here.

(I'm not sure why linux_config is empty though)

Unfortunately Linux dump is not implemented in Configurator. Opened issue for this: #1528

denis-tingaikin commented 5 years ago

@rewenset yes, you are right. The problem is not absolutely correct. The main problem of dump response is it has not linux config.

denis-tingaikin commented 5 years ago

@rewenset , @ondrej-fabry got it, thanks for answers.

ondrej-fabry commented 5 years ago

@rewenset yes, you are right. The problem is not absolutely correct. The main problem of dump response is it has not linux config.

Is this currently a blocker for you?

denis-tingaikin commented 5 years ago

Is this currently a blocker for you?

Yes, it blocks me.

I need to get a generated mac address for linux interfaces right after creation. It is needed for updating arp table in some cases for SRC interface(for example DST interface was recreated). We have few scenarios where the container with SRC interface has wrong arp entry (means that entry has correct IP, but wrong MAC).

Do you know some workaround without using dump requests to fix arp table? For example, can we somehow clear arp entries for SRC interface via vpp-agent?

denis-tingaikin commented 5 years ago

@ondrej-fabry ^^^

milanlenco commented 5 years ago

Do you know some workaround without using dump requests to fix arp table? For example, can we somehow clear arp entries for SRC interface via vpp-agent?

Unfortunately it is not possible to clear dynamically created ARP entries via VPP-Agent. One workaround (until linux dump is available) could be to generate MAC address for the VETH/TAP on the CNF side and create static ARP entries on VPP yourselves - then if you remember the generated MAC address, you will know what to remove once it is re-created. Btw., do you not have this issue also on CNFs that were connected to a restarted client/endpoint? I would expect their ARP table to contain an invalid entry for some time too.

edwarnicke commented 4 years ago

@ondrej-fabry Progress on this?

ondrej-fabry commented 4 years ago

@edwarnicke should be fixed in #1574

Please re-test to confirm.

denis-tingaikin commented 4 years ago

looks like fixed