Lingering macVlans and permanent resource busy when re creating new container with macvlan

threefoldtecharchive / 0-core

Multi Node OS which requires no install.

Apache License 2.0

3 stars 2 forks source link

Lingering macVlans and permanent resource busy when re creating new container with macvlan #84

Closed LeeSmet closed 5 years ago

LeeSmet commented 6 years ago

After successfully creating a container with a macVlan interface, the service can not be restarted. When the container is re created, a failed to setup hw address: device or resource busy error is returned.

Possibly related, there appear to be a log of lingering macvlan interfaces on the node:

Note that only containers with id's 1, 132 and 135 exist, meaning that the interfaces referring to containers 137 and up are not referencing anything at all

LeeSmet commented 6 years ago

For what its worth: the macaddress which was assigned was configured in the dhcp to get a static ipv4 address. It didnt receive any ipv4 address however, only a random ipv6.

muhamadazmy commented 6 years ago

@LeeSmet

Macvlan does not create visible veth on the host. So I suppose all the listed in the screenshot are veth ends of the default network
Weirdly enough, the error happens on setting the MAC address of the macvlan device. Not on creation!

I really can't reproduce so if it possible to give me access to this machine? and also the mac address u trying to use?

LeeSmet commented 6 years ago

Additional info: I managed to restart (stop/start) another container using a macvlan interface on the same node without issues. Every time I try to start this container and get the device busy error, a new interface appears in this list of interfaces

muhamadazmy commented 6 years ago

I couldn't manage to reproduce locally
I looked into the affected node, and it seems this error only happens with a specific mac address.
The mac address I can use on other nodes with no issue
The lingering macvlan links on the host is due to a bug in the clean up process on failure which should be fixed in case this happened again. But it does not explain or cause the issue.
Cleaning all the lingering interfaces did not fix the problem.
No dmesg errors or other logs other than "device or resource busy"

Conclusion: A chance that this is a kernel bug that put this machine on this weird state. I will add a fix for auto clean up anyway to not leave any macvlan links behind.

LeeSmet commented 6 years ago

After rebooting the node, I ran into the same problem. After checking, it seemed that the IP address was already leased to another device (which explains why the IPv4 address was not aquired in the first place). However after the initial container was stopped, the same issue occurred where the container can't be started anymore with a hw or resource busy error