Open kmaloor opened 3 years ago
cjnolan
commented
3 years ago Hi @kmaloor,
The interfaceservice reads the MAC address of the interfaces on the edgenode from the system itself via the Golang net package. Looking at the OVS documentation for ovs-vsctl, it looks for the interface ID and MAC address of an interface as well when it is adding a new interface to an OVS bridge.
Could you check if both interfaces on the edgenode have MAC addresses associated with them please?
kmaloor
commented
3 years ago Hi @cjnolan
Thank you for the response. It appears that the two NICs have valid MAC addresses assigned:
[root@node01 ~]# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1460
inet 10.128.0.14 netmask 255.255.255.255 broadcast 10.128.0.14
inet6 fe80::4001:aff:fe80:e prefixlen 64 scopeid 0x20<link>
ether 42:01:0a:80:00:0e txqueuelen 1000 (Ethernet)
RX packets 1561 bytes 1176344 (1.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1568 bytes 259782 (253.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@node01 ~]# ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1460
inet 10.1.128.2 netmask 255.255.255.255 broadcast 10.1.128.2
inet6 fe80::4001:aff:fe01:8002 prefixlen 64 scopeid 0x20<link>
ether 42:01:0a:01:80:02 txqueuelen 1000 (Ethernet)
RX packets 6 bytes 1310 (1.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12 bytes 1512 (1.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@node01 ~]# cat /sys/class/net/eth0/address
42:01:0a:80:00:0e
[root@node01 ~]# cat /sys/class/net/eth1/address
42:01:0a:01:80:02
[root@node01 ~]# It appears that the framework queries the machine's network interfaces through here: https://github.com/open-ness/edgenode/blob/f4d4f9afe37ba8d1f9d44e1e35fe57de2ddb4656/pkg/ela/helpers/network_interfaces.go#L103
Do you think something may be going wrong in this block? https://github.com/open-ness/edgenode/blob/f4d4f9afe37ba8d1f9d44e1e35fe57de2ddb4656/pkg/ela/helpers/network_interfaces.go#L116
kmaloor
commented
3 years ago Hi @cjnolan
Just a quick follow-up. By manually reading the uevent as over here: https://github.com/open-ness/edgenode/blob/f4d4f9afe37ba8d1f9d44e1e35fe57de2ddb4656/pkg/ela/helpers/network_interfaces.go#L117 from inside the interfaceservice container, I see the following contents:
[root@controller ~]# kubectl exec -it -n openness interfaceservice-dxb5n /bin/sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD]
-- [COMMAND] instead.
sh-4.2$ cat /var/host_net_devices/eth1/device/uevent
DRIVER=virtio_net
MODALIAS=virtio:d00000001v00001AF4It appears that the interfaceservice code wants to extract the PCI address corresponding to network interfaces (by interface name) so as to collate all the metadata (which includes the MAC address) into devs []NetworkDevice. As the uevent excludes this information, the subsequent logic fails.
As such, I'm wondering if the interfaceservice code needs to use an alternative method to collect the PCI address. As an example, using the output of ethtool:
[root@node01 ~]# ethtool -i eth1
driver: virtio_net
version: 1.0.0
firmware-version:
expansion-rom-version:
bus-info: 0000:00:05.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: noIf the above output were placed into the content variable in
https://github.com/open-ness/edgenode/blob/f4d4f9afe37ba8d1f9d44e1e35fe57de2ddb4656/pkg/ela/helpers/network_interfaces.go#L119
then I reckon that the rest of the logic would work as expected.
cjnolan
commented
3 years ago Hi @kmaloor
The cause of the failure with the interfaceservice is that the uevent is not being populated with the PCI addresses of the network interfaces. This appears to be due to the edgenode being deployed on a virtual machine with two virtual network interfaces. This configuration of the edgenode on a virtual machine was not tested with OpenNESS however the interfaceservice has been tested and validated with the edgenode on baremetal.
kmaloor
commented
3 years ago Hi @cjnolan
Thank you for your response. Yes, my Edge node has two (virtio) Ethernet controllers with distinct PCI addresses.
[root@node01 ~]# lshw -c network
*-network:0
description: Ethernet controller
product: Virtio network device
vendor: Red Hat, Inc.
physical id: 4
bus info: pci@0000:00:04.0
version: 00
width: 32 bits
clock: 33MHz
capabilities: msix bus_master cap_list
configuration: driver=virtio-pci latency=0
resources: irq:10 ioport:c040(size=64) memory:c0001000-c000107f
*-virtio1
description: Ethernet interface
physical id: 0
bus info: virtio@1
logical name: eth0
serial: 42:01:0a:80:00:0e
capabilities: ethernet physical
configuration: broadcast=yes driver=virtio_net driverversion=1.0.0 ip=10.128.0.14 link=yes multicast=yes
*-network:1
description: Ethernet controller
product: Virtio network device
vendor: Red Hat, Inc.
physical id: 5
bus info: pci@0000:00:05.0
version: 00
width: 32 bits
clock: 33MHz
capabilities: msix bus_master cap_list
configuration: driver=virtio-pci latency=0
resources: irq:10 ioport:c000(size=64) memory:c0000000-c000007f
*-virtio2
description: Ethernet interface
physical id: 0
bus info: virtio@2
logical name: eth1
serial: 42:01:0a:01:80:02
capabilities: ethernet physical
configuration: broadcast=yes driver=virtio_net driverversion=1.0.0 ip=10.1.128.2 link=yes multicast=yes
...I am currently testing an OpenNESS edge deployment cluster on Google Cloud. I understand that the interfaceservice may be working as expected when run on baremetal.
I was wondering if there's any scope for altering the code such that it works everywhere? There could be use cases where an OpenNESS cluster is deployed over virtualized infrastructure on the edge.
cjnolan
commented
3 years ago Hi @kmaloor Yes, we can look to alter the interfaceservice to work with deployments on virtualized infrastructure as part of a future OpenNESS release.
Hello,
I have built a network edge deployment (OpenNESS 20.09.01) cluster between two QEMU/KVM machines, and have successfully tested producer/consumer apps. The edge node (worker) is provisioned with two network interfaces (eth0 and eth1) with the idea of dedicating one to management/control and the 2nd to data. The worker connects to the master via eth0 and eth1 is on a different subnet.
To route traffic from an external machine to the OpenVINO apps, I am unable to bridge the 2nd (data) network interface on the Edge node to the OVS as suggested in the documentation.
Specifically, a listing of interfaces on the worker node returns two
kernel interfaces(which is expected-eth0 and eth1 respectively for control and data), but the output does not include the MAC addresses of the interfaces.Further, attaching the interface fails as follows:
Is anything misconfigured on my Edge node? Please advise.
If it helps, below is a listing of all PCI devices on the Edge/worker node which indicates the two Ethernet controllers.
Also, below is the status of all pods as seen from the master: