search

ligato / vpp-agent

⚡️ Control plane management agent for FD.io's VPP
https://docs.ligato.io/
Apache License 2.0
252 stars 126 forks source link

Update/Delete/Update didn't work if same names are passed #1285

Closed haiodo closed 5 years ago

haiodo commented 5 years ago

Steps performed:

  1. Create Connection between source/destination.
  2. Let destination die, namespace is not available anymore.
  3. Request Delete on vpp-agent with connection information passed, got error in logs.
  4. Request new Connection betwrrn source and new destination but use same interface names.
  5. Expect to see connection created but receive error. 
    
time="2019-04-12 17:38:14.97837" level=error msg="failed to get link for interface veth-1649917878: Link not found" loc="descriptor/interface_veth.go(71)" logger=linux-ifplugin.if-descriptor

More details with logs:

### 1. Init connection between two pods in kubernetes (source/destination):

Update with following Config Object:

vpp_config:<interfaces:<name:\"SRC-1\" type:AF_PACKET enabled:true afpacket:<host_if_name:\"SRC-1-veth\" > > interfaces:<name:\"DST-1\" type:AF_PACKET enabled:true afpacket:<host_if_name:\"DST-1-veth\" > > xconnect_pairs:<receive_interface:\"SRC-1\" transmit_interface:\"DST-1\" > xconnect_pairs:<receive_interface:\"DST-1\" transmit_interface:\"SRC-1\" > > linux_config:<interfaces:<name:\"SRC-1-veth\" type:VETH host_if_name:\"SRC-1-veth\" enabled:true ip_addresses:\"10.20.1.1/30\" veth:<peer_if_name:\"SRC-1\" > > interfaces:<name:\"SRC-1\" type:VETH namespace:<type:FD reference:\"/proc/85977/ns/net\" > host_if_name:\"nsm\" enabled:true ip_addresses:\"10.20.1.1/30\" veth:<peer_if_name:\"SRC-1-veth\" > > interfaces:<name:\"DST-1-veth\" type:VETH host_if_name:\"DST-1-veth\" enabled:true ip_addresses:\"10.20.1.2/30\" veth:<peer_if_name:\"DST-1\" > > interfaces:<name:\"DST-1\" type:VETH namespace:<type:FD reference:\"/proc/85844/ns/net\" > host_if_name:\"nsmXEhBAHC5Y\" enabled:true ip_addresses:\"10.20.1.2/30\" veth:<peer_if_name:\"DST-1-veth\" > > arp_entries:<interface:\"SRC-1\" ip_address:\"10.40.1.135\" hw_address:\"0a:58:0a:28:01:87\" > arp_entries:<interface:\"SRC-1\" ip_address:\"fe80::d892:1bff:fe75:a2c3\" hw_address:\"0a:58:0a:28:01:87\" > routes:<outgoing_interface:\"SRC-1\" scope:LINK dst_network:\"8.8.8.8/30\" > > "


Got following in vpp-agent logs:

+======================================================================================================================+ | Transaction #2 NB Transaction | +======================================================================================================================+

2 Delete destination POD, send Cancel to vpp-agent.

Sending Delete with following Config object: 

vpp_config:<interfaces:<name:\"SRC-1\" type:AF_PACKET enabled:true afpacket:<host_if_name:\"SRC-1-veth\" > > interfaces:<name:\"DST-1\" type:AF_PACKET enabled:true afpacket:<host_if_name:\"DST-1-veth\" > > xconnect_pairs:<receive_interface:\"SRC-1\" transmit_interface:\"DST-1\" > xconnect_pairs:<receive_interface:\"DST-1\" transmit_interface:\"SRC-1\" > > linux_config:<interfaces:<name:\"SRC-1-veth\" type:VETH host_if_name:\"SRC-1-veth\" enabled:true ip_addresses:\"10.20.1.1/30\" veth:<peer_if_name:\"SRC-1\" > > interfaces:<name:\"SRC-1\" type:VETH namespace:<type:FD reference:\"/proc/85977/ns/net\" > host_if_name:\"nsm\" enabled:true ip_addresses:\"10.20.1.1/30\" veth:<peer_if_name:\"SRC-1-veth\" > > interfaces:<name:\"DST-1-veth\" type:VETH host_if_name:\"DST-1-veth\" enabled:true ip_addresses:\"10.20.1.2/30\" veth:<peer_if_name:\"DST-1\" > > interfaces:<name:\"DST-1\" type:VETH namespace:<type:FD > host_if_name:\"nsmXEhBAHC5Y\" enabled:true ip_addresses:\"10.20.1.2/30\" veth:<peer_if_name:\"DST-1-veth\" > > arp_entries:<interface:\"SRC-1\" ip_address:\"10.40.1.135\" hw_address:\"0a:58:0a:28:01:87\" > arp_entries:<interface:\"SRC-1\" ip_address:\"fe80::d892:1bff:fe75:a2c3\" hw_address:\"0a:58:0a:28:01:87\" > routes:<outgoing_interface:\"SRC-1\" scope:LINK dst_network:\"8.8.8.8/30\" > > "

Got following in logs:

2019-04-12 17:38:14,749 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.74899" level=debug msg="Pushing data with 11 KV pairs (source: grpc)" loc="orchestrator/dispatcher.go(73)" logger=orchestrator.dispatcher

2019-04-12 17:38:14,749 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.74907" level=debug msg=" - DELETE: \"config/vpp/v2/interfaces/SRC-1\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74914" level=debug msg=" - DELETE: \"config/vpp/v2/interfaces/DST-1\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74920" level=debug msg=" - DELETE: \"config/vpp/l2/v2/xconnect/SRC-1\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74925" level=debug msg=" - DELETE: \"config/vpp/l2/v2/xconnect/DST-1\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74930" level=debug msg=" - DELETE: \"config/linux/interfaces/v2/interface/SRC-1-veth\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74936" level=debug msg=" - DELETE: \"config/linux/interfaces/v2/interface/SRC-1\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74941" level=debug msg=" - DELETE: \"config/linux/interfaces/v2/interface/DST-1-veth\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher

2019-04-12 17:38:14,749 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.74951" level=debug msg=" - DELETE: \"config/linux/interfaces/v2/interface/DST-1\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74960" level=debug msg=" - DELETE: \"config/linux/l3/v2/arp/SRC-1/10.40.1.135\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
time="2019-04-12 17:38:14.74967" level=debug msg=" - DELETE: \"config/linux/l3/v2/arp/SRC-1/fe80::d892:1bff:fe75:a2c3\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher

2019-04-12 17:38:14,750 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.74973" level=debug msg=" - DELETE: \"config/linux/l3/v2/route/8.8.8.8/30/SRC-1\"" loc="orchestrator/dispatcher.go(96)" logger=orchestrator.dispatcher
+======================================================================================================================+
| Transaction #6                                                                                        NB Transaction |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 6
      - type: NB Transaction
      - values:
          - key: config/linux/interfaces/v2/interface/DST-1
            val: <NIL>
          - key: config/linux/interfaces/v2/interface/DST-1-veth
            val: <NIL>
          - key: config/linux/interfaces/v2/interface/SRC-1
            val: <NIL>
          - key: config/linux/interfaces/v2/interface/SRC-1-veth
            val: <NIL>
          - key: config/linux/l3/v2/arp/SRC-1/10.40.1.135
            val: <NIL>
          - key: config/linux/l3/v2/arp/SRC-1/fe80::d892:1bff:fe75:a2c3
            val: <NIL>
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
            val: <NIL>
          - key: config/vpp/l2/v2/xconnect/DST-1
            val: <NIL>
          - key: config/vpp/l2/v2/xconnect/SRC-1
            val: <NIL>
          - key: config/vpp/v2/interfaces/DST-1
            val: <NIL>
          - key: config/vpp/v2/interfaces/SRC-1
            val: <NIL>

2019-04-12 17:38:14,750 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.75036" level=error msg="failed to remove IP address 10.20.1.2/30 from linux interface DST-1-veth: Link not found" loc="descriptor/interface.go(417)" logger=linux-ifplugin.if-descriptor

2019-04-12 17:38:14,750 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.75062" level=error msg="failed to get or create namespace (type:FD reference:\"/proc/85844/ns/net\" ): failed to get file /proc/85844/ns/net from path: no such file or directory" loc="descriptor/interface.go(399)" logger=linux-ifplugin.if-descriptor

2019-04-12 17:38:14,790 DEBG 'vpp' stdout output:
/usr/bin/vpp[85783]: linux_epoll_file_update:120: epoll_ctl: Bad file descriptor (errno 9)

2019-04-12 17:38:14,803 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.80333" level=debug msg="publish del to ifplugin_ifstate{{SRC-1 true false vpp-interface-index} 0xc00025cea0}" loc="mem/inmemory_name_mapping.go(323)" logger=vpp-ifplugin.if-descriptor

2019-04-12 17:38:14,805 DEBG 'agent' stdout output:
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-04-12 17:38:14.75 +0000 UTC -> 2019-04-12 17:38:14.804 +0000 UTC, dur: 54ms):
      1. DELETE [PROPERTY]:
          - key: linux/interface/address/DST-1/10.20.1.2/30
          - value: <EMPTY> 
      2. DELETE [PROPERTY]:
          - key: linux/interface/state/DST-1/UP
          - value: <EMPTY> 
      3. DELETE [PROPERTY]:
          - key: linux/interface/address/DST-1-veth/10.20.1.2/30
          - value: <EMPTY> 
      4. DELETE [PROPERTY]:
          - key: linux/interface/state/DST-1-veth/UP
          - value: <EMPTY> 
      5. DELETE [FAILED]:
          - key: config/linux/interfaces/v2/interface/DST-1-veth
          - value: { name:"DST-1-veth" type:VETH host_if_name:"DST-1-veth" enabled:true ip_addresses:"10.20.1.2/30" veth:<peer_if_name:"DST-1" >  } 
          - error: failed to remove IP address 10.20.1.2/30 from linux interface DST-1-veth: Link not found
      6. DELETE [FAILED]:
          - key: config/linux/interfaces/v2/interface/DST-1
          - value: { name:"DST-1" type:VETH namespace:<type:FD reference:"/proc/85844/ns/net" > host_if_name:"nsmXEhBAHC5Y" enabled:true ip_addresses:"10.20.1.2/30" veth:<peer_if_name:"DST-1-veth" >  } 
          - error: failed to get or create namespace (type:FD reference:"/proc/85844/ns/net" ): failed to get file /proc/85844/ns/net from path: no such file or directory
      7. DELETE:
          - key: config/linux/l3/v2/arp/SRC-1/10.40.1.135
          - value: { interface:"SRC-1" ip_address:"10.40.1.135" hw_address:"0a:58:0a:28:01:87"  } 
      8. DELETE:
          - key: config/linux/l3/v2/arp/SRC-1/fe80::d892:1bff:fe75:a2c3
          - value: { interface:"SRC-1" ip_address:"fe80::d892:1bff:fe75:a2c3" hw_address:"0a:58:0a:28:01:87"  } 
      9. DELETE [PROPERTY]:
          - key: linux/link-local-route/SRC-1/8.8.8.8/30
          - value: <EMPTY> 
      10. DELETE:
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
          - value: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  } 
      11. DELETE [PROPERTY]:
          - key: linux/interface/address/SRC-1/10.20.1.1/30
          - value: <EMPTY> 
      12. DELETE [PROPERTY]:
          - key: linux/interface/state/SRC-1/UP
          - value: <EMPTY> 
      13. DELETE [PROPERTY]:
          - key: linux/interface/address/SRC-1-veth/10.20.1.1/30
          - value: <EMPTY> 
      14. DELETE [PROPERTY]:
          - key: linux/interface/state/SRC-1-veth/UP
          - value: <EMPTY> 
      15. DELETE:
          - key: config/linux/interfaces/v2/interface/SRC-1-veth
          - value: { name:"SRC-1-veth" type:VETH host_if_name:"SRC-1-veth" enabled:true ip_addresses:"10.20.1.1/30" veth:<peer_if_name:"SRC-1" >  } 
      16. DELETE:
          - key: config/linux/interfaces/v2/interface/SRC-1
          - value: { name:"SRC-1" type:VETH namespace:<type:FD reference:"/proc/85977/ns/net" > host_if_name:"nsm" enabled:true ip_addresses:"10.20.1.1/30" veth:<peer_if_name:"SRC-1-veth" >  } 
      17. DELETE [NOOP WAS-PENDING]:
          - key: config/vpp/l2/v2/xconnect/DST-1
          - value: { receive_interface:"DST-1" transmit_interface:"SRC-1"  } 
      18. DELETE [NOOP WAS-PENDING]:
          - key: config/vpp/l2/v2/xconnect/SRC-1
          - value: { receive_interface:"SRC-1" transmit_interface:"DST-1"  } 
      19. DELETE [NOOP WAS-PENDING]:
          - key: config/vpp/v2/interfaces/DST-1
          - value: { name:"DST-1" type:AF_PACKET enabled:true afpacket:<host_if_name:"DST-1-veth" >  } 
      20. DELETE:
          - key: config/vpp/v2/interfaces/SRC-1
          - value: { name:"SRC-1" type:AF_PACKET enabled:true afpacket:<host_if_name:"SRC-1-veth" >  } 
x----------------------------------------------------------------------------------------------------------------------x
| #6                                                                                                       took 53.7ms |
x----------------------------------------------------------------------------------------------------------------------x

time="2019-04-12 17:38:14.80428" level=debug msg="Global interface state data updated: internal_name:\"host-SRC-1-veth\" index:3 status:\"DELETED\" mac_address:\"02:fe:43:f1:81:d8\" " loc="statuscheck/plugin_impl_statuscheck.go(251)" logger=status-check
time="2019-04-12 17:38:14.80480" level=debug msg="linux-interface descriptor retrieved 0 items: " loc="kvscheduler/refresh.go(129)" logger=kvscheduler
+======================================================================================================================+
| Transaction #7                                                                                       SB Notification |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 7
      - type: SB Notification
      - values:
          - key: linux/interface/host-name/SRC-1-veth
            val: <NIL>

o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-04-12 17:38:14.805 +0000 UTC -> 2019-04-12 17:38:14.805 +0000 UTC, dur: 0s):
      1. DELETE [WAS-OBTAINED]:
          - key: linux/interface/host-name/SRC-1-veth
          - value: <EMPTY> 
x----------------------------------------------------------------------------------------------------------------------x
| #7                                                                                                        took 100µs |
x----------------------------------------------------------------------------------------------------------------------x

time="2019-04-12 17:38:14.80524" level=error msg="Transaction #6 finished with 2 errors" loc="orchestrator/dispatcher.go(119)" logger=orchestrator.dispatcher
 -   1. error (DELETE) config/linux/interfaces/v2/interface/DST-1-veth - failed to remove IP address 10.20.1.2/30 from linux interface DST-1-veth: Link not found
 -   2. error (DELETE) config/linux/interfaces/v2/interface/DST-1 - failed to get or create namespace (type:FD reference:"/proc/85844/ns/net" ): failed to get file /proc/85844/ns/net from path: no such file or directory

time="2019-04-12 17:38:14.80529" level=info msg="Transaction 7 successful!" loc="kvscheduler/txn_process.go(487)" logger=kvscheduler

3 trying to configure new connection but with same interface names, I'm assume previous delete was ok.

 vpp_config:<interfaces:<name:\"SRC-1\" type:AF_PACKET enabled:true afpacket:<host_if_name:\"SRC-1-veth\" > > interfaces:<name:\"DST-1\" type:AF_PACKET enabled:true afpacket:<host_if_name:\"DST-1-veth\" > > xconnect_pairs:<receive_interface:\"SRC-1\" transmit_interface:\"DST-1\" > xconnect_pairs:<receive_interface:\"DST-1\" transmit_interface:\"SRC-1\" > > linux_config:<interfaces:<name:\"SRC-1-veth\" type:VETH host_if_name:\"SRC-1-veth\" enabled:true ip_addresses:\"10.20.1.1/30\" veth:<peer_if_name:\"SRC-1\" > > interfaces:<name:\"SRC-1\" type:VETH namespace:<type:FD reference:\"/proc/85977/ns/net\" > host_if_name:\"nsm\" enabled:true ip_addresses:\"10.20.1.1/30\" veth:<peer_if_name:\"SRC-1-veth\" > > interfaces:<name:\"DST-1-veth\" type:VETH host_if_name:\"DST-1-veth\" enabled:true ip_addresses:\"10.20.1.2/30\" veth:<peer_if_name:\"DST-1\" > > interfaces:<name:\"DST-1\" type:VETH namespace:<type:FD reference:\"/proc/86609/ns/net\" > host_if_name:\"nsmRMj5VM6fj\" enabled:true ip_addresses:\"10.20.1.2/30\" veth:<peer_if_name:\"DST-1-veth\" > > arp_entries:<interface:\"SRC-1\" ip_address:\"10.40.1.137\" hw_address:\"0a:58:0a:28:01:89\" > arp_entries:<interface:\"SRC-1\" ip_address:\"fe80::6cf6:b9ff:fe91:1fe8\" hw_address:\"0a:58:0a:28:01:89\" > routes:<outgoing_interface:\"SRC-1\" scope:LINK dst_network:\"8.8.8.8/30\" > > "

And following in vpp-agent logs:

+======================================================================================================================+
| Transaction #8                                                                                        NB Transaction |
+======================================================================================================================+
  * transaction arguments:
      - seq-num: 8
      - type: NB Transaction
      - values:
          - key: config/linux/interfaces/v2/interface/DST-1
            val: { name:"DST-1" type:VETH namespace:<type:FD reference:"/proc/86609/ns/net" > host_if_name:"nsmRMj5VM6fj" enabled:true ip_addresses:"10.20.1.2/30" veth:<peer_if_name:"DST-1-veth" >  }
          - key: config/linux/interfaces/v2/interface/DST-1-veth
            val: { name:"DST-1-veth" type:VETH host_if_name:"DST-1-veth" enabled:true ip_addresses:"10.20.1.2/30" veth:<peer_if_name:"DST-1" >  }
          - key: config/linux/interfaces/v2/interface/SRC-1
            val: { name:"SRC-1" type:VETH namespace:<type:FD reference:"/proc/85977/ns/net" > host_if_name:"nsm" enabled:true ip_addresses:"10.20.1.1/30" veth:<peer_if_name:"SRC-1-veth" >  }
          - key: config/linux/interfaces/v2/interface/SRC-1-veth
            val: { name:"SRC-1-veth" type:VETH host_if_name:"SRC-1-veth" enabled:true ip_addresses:"10.20.1.1/30" veth:<peer_if_name:"SRC-1" >  }
          - key: config/linux/l3/v2/arp/SRC-1/10.40.1.137
            val: { interface:"SRC-1" ip_address:"10.40.1.137" hw_address:"0a:58:0a:28:01:89"  }
          - key: config/linux/l3/v2/arp/SRC-1/fe80::6cf6:b9ff:fe91:1fe8
            val: { interface:"SRC-1" ip_address:"fe80::6cf6:b9ff:fe91:1fe8" hw_address:"0a:58:0a:28:01:89"  }
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
            val: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  }
          - key: config/vpp/l2/v2/xconnect/DST-1
            val: { receive_interface:"DST-1" transmit_interface:"SRC-1"  }
          - key: config/vpp/l2/v2/xconnect/SRC-1
            val: { receive_interface:"SRC-1" transmit_interface:"DST-1"  }
          - key: config/vpp/v2/interfaces/DST-1
            val: { name:"DST-1" type:AF_PACKET enabled:true afpacket:<host_if_name:"DST-1-veth" >  }
          - key: config/vpp/v2/interfaces/SRC-1
            val: { name:"SRC-1" type:AF_PACKET enabled:true afpacket:<host_if_name:"SRC-1-veth" >  }

2019-04-12 17:38:14,978 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.97837" level=error msg="failed to get link for interface veth-1649917878: Link not found" loc="descriptor/interface_veth.go(71)" logger=linux-ifplugin.if-descriptor

2019-04-12 17:38:14,978 DEBG 'agent' stdout output:
time="2019-04-12 17:38:14.97880" level=error msg="failed to get link for interface veth-2364841676: Link not found" loc="descriptor/interface_veth.go(71)" logger=linux-ifplugin.if-descriptor

2019-04-12 17:38:15,022 DEBG 'agent' stdout output:
o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2019-04-12 17:38:14.977 +0000 UTC -> 2019-04-12 17:38:15.022 +0000 UTC, dur: 45ms):
      1. CREATE [STILL-FAILING]:
          - key: config/linux/interfaces/v2/interface/DST-1
          - value: { name:"DST-1" type:VETH namespace:<type:FD reference:"/proc/86609/ns/net" > host_if_name:"nsmRMj5VM6fj" enabled:true ip_addresses:"10.20.1.2/30" veth:<peer_if_name:"DST-1-veth" >  } 
          - prev-error: failed to get or create namespace (type:FD reference:"/proc/85844/ns/net" ): failed to get file /proc/85844/ns/net from path: no such file or directory
          - error: failed to get link for interface veth-1649917878: Link not found
      2. CREATE [STILL-FAILING]:
          - key: config/linux/interfaces/v2/interface/DST-1-veth
          - value: { name:"DST-1-veth" type:VETH host_if_name:"DST-1-veth" enabled:true ip_addresses:"10.20.1.2/30" veth:<peer_if_name:"DST-1" >  } 
          - prev-error: failed to remove IP address 10.20.1.2/30 from linux interface DST-1-veth: Link not found
          - error: failed to get link for interface veth-2364841676: Link not found
      3. CREATE:
          - key: config/linux/interfaces/v2/interface/SRC-1-veth
          - value: { name:"SRC-1-veth" type:VETH host_if_name:"SRC-1-veth" enabled:true ip_addresses:"10.20.1.1/30" veth:<peer_if_name:"SRC-1" >  } 
      4. CREATE:
          - key: config/linux/interfaces/v2/interface/SRC-1
          - value: { name:"SRC-1" type:VETH namespace:<type:FD reference:"/proc/85977/ns/net" > host_if_name:"nsm" enabled:true ip_addresses:"10.20.1.1/30" veth:<peer_if_name:"SRC-1-veth" >  } 
      5. CREATE [PROPERTY]:
          - key: linux/interface/address/SRC-1/10.20.1.1/30
          - value: <EMPTY> 
      6. CREATE [PROPERTY]:
          - key: linux/interface/state/SRC-1/UP
          - value: <EMPTY> 
      7. CREATE [PROPERTY]:
          - key: linux/interface/address/SRC-1-veth/10.20.1.1/30
          - value: <EMPTY> 
      8. CREATE [PROPERTY]:
          - key: linux/interface/state/SRC-1-veth/UP
          - value: <EMPTY> 
      9. CREATE:
          - key: config/linux/l3/v2/arp/SRC-1/10.40.1.137
          - value: { interface:"SRC-1" ip_address:"10.40.1.137" hw_address:"0a:58:0a:28:01:89"  } 
      10. CREATE:
          - key: config/linux/l3/v2/arp/SRC-1/fe80::6cf6:b9ff:fe91:1fe8
          - value: { interface:"SRC-1" ip_address:"fe80::6cf6:b9ff:fe91:1fe8" hw_address:"0a:58:0a:28:01:89"  } 
      11. CREATE:
          - key: config/linux/l3/v2/route/8.8.8.8/30/SRC-1
          - value: { outgoing_interface:"SRC-1" scope:LINK dst_network:"8.8.8.8/30"  } 
      12. CREATE [PROPERTY]:
          - key: linux/link-local-route/SRC-1/8.8.8.8/30
          - value: <EMPTY> 
      13. CREATE [NOOP IS-PENDING]:
          - key: config/vpp/l2/v2/xconnect/DST-1
          - value: { receive_interface:"DST-1" transmit_interface:"SRC-1"  } 
      14. CREATE [NOOP IS-PENDING]:
          - key: config/vpp/l2/v2/xconnect/SRC-1
          - value: { receive_interface:"SRC-1" transmit_interface:"DST-1"  } 
      15. CREATE [NOOP IS-PENDING]:
          - key: config/vpp/v2/interfaces/DST-1
          - value: { name:"DST-1" type:AF_PACKET enabled:true afpacket:<host_if_name:"DST-1-veth" >  } 
      16. CREATE [NOOP IS-PENDING]:
          - key: config/vpp/v2/interfaces/SRC-1
          - value: { name:"SRC-1" type:AF_PACKET enabled:true afpacket:<host_if_name:"SRC-1-veth" >  } 
x----------------------------------------------------------------------------------------------------------------------x
| #8                                                                                                         took 45ms |
x----------------------------------------------------------------------------------------------------------------------x

2019-04-12 17:38:15,026 DEBG 'agent' stdout output:
time="2019-04-12 17:38:15.02471" level=debug msg="linux-interface descriptor retrieved 2 items: \n -  1. [linux-interface]: \"config/linux/interfaces/v2/interface/SRC-1-veth\" (from-NB)\n   { name:\"SRC-1-veth\" type:VETH host_if_name:\"SRC-1-veth\" enabled:true ip_addresses:\"10.20.1.1/30\" phys_address:\"52:63:e8:21:49:fc\" mtu:1500 veth:<peer_if_name:\"SRC-1\" >  }\n   Metadata: &{LinuxIfIndex:9 VPPTapName: Namespace:<nil>}\n -  2. [linux-interface]: \"config/linux/interfaces/v2/interface/SRC-1\" (from-NB)\n   { name:\"SRC-1\" type:VETH namespace:<type:FD reference:\"/proc/85977/ns/net\" > host_if_name:\"nsm\" enabled:true ip_addresses:\"10.20.1.1/30\" phys_address:\"56:bd:c3:63:70:9a\" mtu:1500 veth:<peer_if_name:\"SRC-1-veth\" >  }\n   Metadata: &{LinuxIfIndex:8 VPPTapName: Namespace:type:FD reference:\"/proc/85977/ns/net\" }" loc="kvscheduler/refresh.go(129)" logger=kvscheduler

2019-04-12 17:38:15,027 DEBG 'agent' stdout output:
time="2019-04-12 17:38:15.02722" level=error msg="Transaction #8 finished with 2 errors" loc="orchestrator/dispatcher.go(119)" logger=orchestrator.dispatcher

2019-04-12 17:38:15,028 DEBG 'vppagent-dataplane' stdout output:
2019/04/12 17:38:15 Reporting span 7e2120e855ea193a:7e2120e855ea193a:0:1

2019-04-12 17:38:15,028 DEBG 'agent' stdout output:
 -   1. error (CREATE) config/linux/interfaces/v2/interface/DST-1 - failed to get link for interface veth-1649917878: Link not found
 -   2. error (CREATE) config/linux/interfaces/v2/interface/DST-1-veth - failed to get link for interface veth-2364841676: Link not found
milanlenco commented 5 years ago

Could you please check if your version of the vpp-agent includes the following commit: https://github.com/ligato/vpp-agent/pull/1259/commits/0022a5ef3c98bb8fddfa6fa9ca6198f5cceea97d

edwarnicke commented 5 years ago

@milanlenco We are using ligato/vpp-agent:v2.0.0 ? Which version has the fix?

haiodo commented 5 years ago

Confirming vpp-agent v2.0.1 fixes issue in Google cloud engine.