open-vm-tools not recognizing IP addresses from network namespaces

[FlorinAsavoaie]

Use case:

I have 2 VMWare Virtual Machines with Ubuntu 16.04.3. VMWare Tools version 10.0.7.52125.

I am trying to use the Docker IpVlan network driver. What that driver does is to create a slave device (one for each container) to one of the interfaces of the VM, put that slave device in a new network namespace and assign a routable IP address to it.

The problem comes when VMWare IP Spoof Guard is active at VmWare/NSX layer because open-vm-tools doesn't report the IPs in the secondary namespaces to VmWare.

Basically it is needed that open-vm-tools loops through the network namespaces on the system and also reports the IP addresses that are assigned to slave devices present in other network namespaces.

[oliverkurth]

Thank you for feedback - unfortunately I am not familiar with IP Spoof Guard, so I delegated that question to people who know.

[FlorinAsavoaie]

Well, meanwhile I discovered that it seems to be generally valid for NSX networks, it is not due to Spoof Guard. The problem is in lib/nicInfo, where the IPs are not visible to getifaddrs since the vmtoolsd daemon is running, obviously, in the default network namespace.

[oliverkurth]

Thank you for that additional feedback - I think I understand the issue better now. So this needs to be addressed inside tools, and they need to report on interfaces in other namespaces inside the VM.

[FlorinAsavoaie]

Exactly.

[FlorinAsavoaie]

This problem apparently happens when you don't have ARP or DHCP Spoofing in NSX and NSX only relies on the data read by the open-vm-tools to know what IPs are assigned to each VM. It also happens if you use Virtual IPs that don't reside on an interface and other similar cases.

Looking and studying more into this, as well as asking some more brilliant minds, I don't think there's a real way to implement a "loop through namespaces" directly, since you would actually need to move the vmtoolsd process around in each namespace to read that information or spawn a process in each namespace that reports back the IPs list. At least these are the only ways I found to do it programatically. Getting the list of namespaces is also a totally different story since containerization tools don't always use the /var/run/netns path to link to the namespaces created. For example Docker creates its own in /var/run/docker/netns, as such even getting a list of namespaces would be kind of problematic. I couldn't find any API to expose this to userspace.

I thought of alternatives and I think that the safest and best one would be to let the user add a list of Virtual IPs to tools.conf (or create a separate file like /etc/vmware-tools/virtual-ips.conf). Such as:

[network.ips]
MAC1=IP1,IP2,IP3
MAC2=IP4,IP5,IP6,IP7

Then it is at the discretion of the administrator to populate with the proper information, and it seems a better alternative than executing a script or something as often as NSX appears to request those information.

If my proposal is ok, I can start working on a PR myself.

Thanks.

[FlorinAsavoaie]

I looked at the NetworkManager code and the way it does it is to remember the namespace where the process (vmtoolsd itself in this case) is running by storing the FD at /proc/self/ns/net, then uses setns() to cycle through the available network namespaces. This could be done in lib/nicInfo/nicInfoPosix.c by renaming the function GuestInfoGetNicInfo to something like RealGuestInfoGetNicInfo and re-creating GuestInfoGetNicInfo as a loop through the available namespaces and calling RealGuestInfoGetNicInfo in each namespace, then restore the process in its own namespace.

The only problem that remains is how to list namespaces. And for this, I would think that using /var/run/netns is sufficient - maybe adding the option to specify alternate netns path(s) from the configuration. When using Docker, one can then either bind mount /var/run/docker/netns to /var/run/netns, either have a tool create symlinks in /var/run/netns, either reconfigure vmtools where to get the netns list from.

WORKAROUND Since I don't have the time now to prepare this PR, I found a workaround, in case someone gets into the same trouble: you can just configure all container IPs as alias interfaces in both namespaces (primary and container). Honestly, I didn't check exactly why it works but I guess that in the container namespace you get a slave link which is processed before the alias interfaces in the primary namespace. The only "downside" is that if you try to access the IP from the host, you end up on the host, when it should actually drop the packets, but I don't see any issues with that, at least not for my use case.

[FlorinAsavoaie]

@oliverkurth so glad that you guys are taking everything so serious and help your customers.

[jasonking3]

I am running into this issue with an OVF-packaged virtual network function (VNF). In this case the management interface of the VNF is put into its own network namespace, and in order to further automate the configuration of this VNF I need to pull the DHCP-assigned address of the management interface via VM tools.