macvlan driver doesn't work in MacOS

[ajoldham]

Continuation of : https://github.com/docker/for-mac/issues/3447 as it was auto-closed. This isn't fixed in the latest Docker Edge 19.03.2 with MacOS 10.14.6.

---

$ ifconfig en9 en9: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500 options=4 ether 58:ef:68:7d:06:05 inet6 fe80::1028:71b1:5694:b4bf%en9 prefixlen 64 secured scopeid 0x12 inet 10.42.34.91 netmask 0xfffffc00 broadcast 10.42.35.255 nd6 options=201<PERFORMNUD,DAD> media: autoselect (1000baseT ) status: active $ docker network create -d macvlan --subnet=10.42.32.0/22 --gateway=10.42.32.1 -o parent=en9 pub_net Error response from daemon: invalid subinterface vlan name en9, example formatting is eth0.10

[TimXL]

Same issue :-(

[dmnkdmnt]

Would really love to see this working, too.

[aalejoz25]

I have the same issue on fedora 38, i don't know why

[aalejoz25@Nitro5-Alvaro ~]$ docker network create -d macvlan \ --subnet=192.168.0.0/24 \ --gateway=192.168.0.1 \ -o parent=wlp0s20f3 \ macvlan1 Error response from daemon: invalid subinterface vlan name wlp0s20f3, example formatting is eth0.10

[miamilabs]

I have the same issue on fedora 38, i don't know why

[aalejoz25@Nitro5-Alvaro ~]$ docker network create -d macvlan \ --subnet=192.168.0.0/24 \ --gateway=192.168.0.1 \ -o parent=wlp0s20f3 \ macvlan1 Error response from daemon: invalid subinterface vlan name wlp0s20f3, example formatting is eth0.10

this is open for 4 years now and still broken. I did end up order mini computer to install linux on it..

[autumnjolitz]

Well, I wish I had Google'd for this nonsense before wasting an hour trying to get macvlan to work on OSX.

I will note that docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh will grant access to the internal docker VM for OSX. In theory, it should be possible to add a custom bridge to allow for a container to acquire an ip address from the host's DHCP network.

However, it is unclear how to add a viable bridge as explained in devplayer0/docker-net-dhcp as one needs to successfully route packets sent to my-bridge to the external OSX network. One or more iptables rules need to be adjusted to forward packets so a DHCP discover on the host (accomplished by udhcpc my-bridge) will succeed.

I rather detest network nonsense because of how undebuggable it often is. Perhaps someone else can build upon my work here.

[miamilabs]

Well, I wish I had Google'd for this nonsense before wasting an hour trying to get macvlan to work on OSX.

I will note that docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh will grant access to the internal docker VM for OSX. In theory, it should be possible to add a custom bridge to allow for a container to acquire an ip address from the host's DHCP network.

However, it is unclear how to add a viable bridge as explained in devplayer0/docker-net-dhcp as one needs to successfully route packets sent to my-bridge to the external OSX network. One or more iptables rules need to be adjusted to forward packets so a DHCP discover on the host (accomplished by udhcpc my-bridge) will succeed.

I rather detest network nonsense because of how undebuggable it often is. Perhaps someone else can build upon my work here.

Thank you for your work!! I will try this when I get home.

It took me days but I end up spinning up a VM to solve this.

[michaelprescott]

Well, I landed here again, wanting this feature to work and re-discovering it still does not. Heavier weight, manually configured, VMs are still the solution I guess.

[miamilabs]

Well, I landed here again, wanting this feature to work and re-discovering it still does not. Heavier weight, manually configured, VMs are still the solution I guess.

Well it’s a shame that developers ignore the power of new macs. Imagine a Mac mini runs with 2 watts all my stuff which run before on intel linux with 50 watts.

I did end up mostly spin up VMs to solve all the issues.

[enzofrnt]

+1

[enzofrnt]

@christophermclellan hi, any news.. one year later ? We need it to use the power of the new Mac, we can make a lot of thing with a lot less power.. Please, help us.

[enzofrnt]

Well, I wish I had Google'd for this nonsense before wasting an hour trying to get macvlan to work on OSX. I will note that docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh will grant access to the internal docker VM for OSX. In theory, it should be possible to add a custom bridge to allow for a container to acquire an ip address from the host's DHCP network. However, it is unclear how to add a viable bridge as explained in devplayer0/docker-net-dhcp as one needs to successfully route packets sent to my-bridge to the external OSX network. One or more iptables rules need to be adjusted to forward packets so a DHCP discover on the host (accomplished by udhcpc my-bridge) will succeed. I rather detest network nonsense because of how undebuggable it often is. Perhaps someone else can build upon my work here.

Thank you for your work!! I will try this when I get home.

It took me days but I end up spinning up a VM to solve this.

Hi, the thing with "the internal docker VM for OSX" didn't work ?

[miamilabs]

@christophermclellan hi, any news.. one year later ? We need it to use the power of the new Mac, we can make a lot of thing with a lot less power.. Please, help us.

There are for sure other prios and if you are not content creator or some famous youtuber like Linus there wont be any help or so. Provably in 10 years we might get some hotfix

[enzofrnt]

Hi, @miamilabs, this might interest you. After extensive research, I believe I'm close to finding the solution.

I disagree with the notion that our Docker isn't functioning directly on MacOS. It seems to be running on a subsystem similar to Windows' WSL, possibly called "Linux Kit." I'm not entirely certain, but that's my current understanding.

This discovery stems from here.

With the help of this GitHub Repo, I managed to establish a connection to the subsystem using the command:

nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock

I suspect the file located at /Users/--YOUR USER--/Library/Containers/com.docker.docker/Data/vms/0/data could very well be the raw file for the subsystem, akin to a VM's raw file.

From there, navigating here :

reveals all the interfaces that might serve as parent interfaces for IPVLAN or MACVLAN because docker can access directly to those.

My current understanding is that Docker developers might have implemented a MacOS solution by merely integrating a Linux-based subsystem. This approach might have allowed them to sidestep MacOS-centric development, relying instead on the Linux system. However, it seems there's a shortfall in configuration, as eth0 fails to interact seamlessly with the host (your Mac).

If my observations hold water, my next course of action will be to devise a more efficient networking configuration between MacOS and LinuxKit.

But how and where.. And can we do it ?

[SteveMarshall]

@enzofrnt I'm afraid what you've discovered… is actually just what is very well documented across this and other threads (notably, moby/hyperkit#45, and throughout this thread, if you read back).

It's well known that Docker for Mac uses LinuxKit under the hood, and LinuxKit says in the README that it runs on macOS in a hypervisor. For Docker for Mac, that hypervisor is Virtualization.framework. This is even noted in the blog post from 2020 announcing the preview of Docker Desktop for M1.

All of what you've described are parts of how Virtualization.framework runs virtual machines on macOS in as minimal and transparent a fashion as possible.

The actual solution to this issue appears to require a change to the way Docker for Mac interacts with HyperKit/VPNKit, as described in moby/hyperkit#45.

---

My current workaround, for anyone who's after workarounds, is to go back to the old way of doing things: running containers in VMs in Virtualbox, which can run on Apple Silicon and can bridge the VMs' network interface to the host Mac. It's not perfect, but it works well enough.

[enzofrnt]

@SteveMarshall Thanks for your kindness with a French student, who is trying to understand what's going on ! If you know all this stuff really nice, I think there is a better way to explain it, and maybe try to find other workaround. ;) Have a nice day, Steve

[expresspotato]

Still docker team? Its been years and once again I find my self looking for the Mac VLAN driver in Docker... Nope.

[vschwaberow]

The never-ending story

[miamilabs]

Hi, @miamilabs, this might interest you. After extensive research, I believe I'm close to finding the solution.

I disagree with the notion that our Docker isn't functioning directly on MacOS. It seems to be running on a subsystem similar to Windows' WSL, possibly called "Linux Kit." I'm not entirely certain, but that's my current understanding.

This discovery stems from here.

With the help of this GitHub Repo, I managed to establish a connection to the subsystem using the command:

nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock

I suspect the file located at /Users/--YOUR USER--/Library/Containers/com.docker.docker/Data/vms/0/data could very well be the raw file for the subsystem, akin to a VM's raw file.

From there, navigating here : reveals all the interfaces that might serve as parent interfaces for IPVLAN or MACVLAN because docker can access directly to those.

My current understanding is that Docker developers might have implemented a MacOS solution by merely integrating a Linux-based subsystem. This approach might have allowed them to sidestep MacOS-centric development, relying instead on the Linux system. However, it seems there's a shortfall in configuration, as eth0 fails to interact seamlessly with the host (your Mac).

If my observations hold water, my next course of action will be to devise a more efficient networking configuration between MacOS and LinuxKit.

But how and where.. And can we do it ?

Unfortunately, my knowledge of Linux and networking is not extensive. I bring my developer skills with me, but I have yet to find any hacks or solutions to the problem. Using a workaround like running items on a virtual machine on the Mac Mini still seems inefficient and resource-intensive.

As i see Last week some one got assigned to that issue. Lets hope some movement will be done after 5 years on that issue.

[encbladexp]

No Idea why this Ticket is that long open, but:

The macvlan networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server.

From the Official Docs

[enzofrnt]

No Idea why this Ticket is that long open, but:

The macvlan networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server.

From the Official Docs

Not sure when it was update but in some documentation it was not clear that only linux support it. But any way, because we ask to get support of it on MacOS especially with the new ARM that are freaking efficient

[encbladexp]

It is a hard technical limitation of using virtualisation on macOS and Windows. You just don't have access to a host's network interface through the used virtualisation, neither Hyper-V (Windows) nor HyperKit (macOS) support to be attached directly to a hosts interface. And that is what you would need.

I created a repository, just for fun reasons: Docker Test Network on Github.

The efficiency of ARM is related to ARM stuff, as soon as you need an x86/amd64 image, things get much less efficient, by design.

Guidance: Close this ticket, and open one for HyperKit, because that is the place where things would need to be "fixed", but be aware: HyperKit is based von Hypervisor.Framework from Apple, its possible that this Framework just lacks features needed, which would result in a call towards the Apple Support. Good luck. 🍀

[Neo-OuO]

+1

[CaptainRedHat]

Well this is unfortunate news as I was looking to using a Mac mini as my homelab’s master development workstation and docker host for things that don’t work well in k8s. However, I might try a hacky workaround involving a WireGuard vpn. (I’m thinking if I use WireGuard to bridge the Linuxkit vm’s network to my physical network, I could theoretically make the docker internal network routable from the physical network).

[enzofrnt]

Well this is unfortunate news as I was looking to using a Mac mini as my homelab’s master development workstation and docker host for things that don’t work well in k8s. However, I might try a hacky workaround involving a WireGuard vpn. (I’m thinking if I use WireGuard to bridge the Linuxkit vm’s network to my physical network, I could theoretically make the docker internal network routable from the physical network).

Yes, if you do it. Please provide us the solution. Thanks !

[michaelprescott]

It is a hard technical limitation of using virtualisation on macOS and Windows. You just don't have access to a host's network interface through the used virtualisation, neither Hyper-V (Windows) nor HyperKit (macOS) support to be attached directly to a hosts interface. And that is what you would need.

I created a repository, just for fun reasons: Docker Test Network on Github.

The efficiency of ARM is related to ARM stuff, as soon as you need an x86/amd64 image, things get much less efficient, by design.

Guidance: Close this ticket, and open one for HyperKit, because that is the place where things would need to be "fixed", but be aware: HyperKit is based von Hypervisor.Framework from Apple, its possible that this Framework just lacks features needed, which would result in a call towards the Apple Support. Good luck. 🍀

How it is achieved and how it runs so well on both Windows and Linux platforms is magic to me, but for each, the performance is amazing and good enough for my usage. There are a lot of us that use macOS, and the new Mac Mini's are absolutely amazing. At the present I'm still working around the matter by just using VMWare VMs, and built my own IaC tool for ramping up and configuring new machines. One Mac mini is now running about 6 VMs with my current setup, however, I suspect Docker containers would be even more performant. So, I would hope they don't close this ticket and get it resolved one day. For that reason, I come back here and checkup on status from time to time.

[encbladexp]

Just in case: AFAIK is macvlan also not working on Windows, because as I said: It is a VM, sitting on a Bridge, on every OS except on Linux where Docker has its home. And even on Linux, its possible to install a mess called Docker Desktop, which nobody asked for and nobody really needed there.

[jkneer]

The situation is a usability nightmare. Docker is offered on multiple platforms, and I have actually used docker on Linux, macos and windows for years, before running into this issue. I spend hours debugging, until I found the hint in the documentation that what I wanted to do was unsupported.

I do understand the technical reasoning why some features are not implemented on macos and windows. But that was - at least not my complaint - the problem is the silent failure. The CLI parameters are there but they fail silently, no warning nothing. As you can see in this thread, this has probably wasted thousands of hours.

The solution is quite simple: There should be either a very prominent warning upon usage of the CLI parameters, or they should be removed on the unsupported platforms. I would prefer the removal of the parameters, especially as there seems no larger interest in ever supporting the features on those platforms.

[encbladexp]

In that case @jkneer the server/daemon would need to inform the CLI/client that this feature is unsupported. The parameter itself could not be remove from the client, as you could use any Docker daemon by using either docker context or setting DOCKER_HOST explicitly.

[g10ck]

I've resolved my issue. It's not regarding to macvlan, but I've found this issue after searching an ability to connect to my containers directly. If you are here for the same reason then my solution can be helpful for you.

What was the problem: I was testing dev environment and my app has the same ports in each container. I didn't want to do port mapping for each service like: 22 ssh in container ONE to 23 local, 22 ssh on container TWO to 24 local, ... And ssh was not the case, other services had also the same ports in each container. So I wanted to reach my containers by their ip-addresses and ports directly.

Solution:

1. Get subnet that is bridget to host network. In my case bridge name in MacOS was "bridge100": bridge_name="bridge100" brnet=$(ifconfig $bridge_name | egrep -o 'inet.*netmask' | cut -d' ' -f 2 | awk -F'.' '{print $1"."$2"."$3}') && echo $brnet
2. Get iface address in docker VM in MacOS: gateway=$(echo "ifconfig | grep $brnet && exit" | nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock 2>/dev/null | egrep -o 'inet.*netmask' | cut -d' ' -f 2) && echo $gateway 3, Get docker containers network address: containers_net=$(echo "ifconfig docker0 && exit" | nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock 2>/dev/null | egrep -o 'inet.*netmask' | cut -d' ' -f 2 | awk -F'.' '{print $1"."$2".0.0/16"}') && echo $containers_net
3. Add route from MacOS to docker container network (which is NATed) via bridget interface on docker VM side: sudo route add $containers_net $gateway
4. Enjoy

[encbladexp]

@g10ck that is a different use case. You are basically just routing the container networks which exist within the VM from the host into the VM, that will (most likely as you showed) work.

However, it is not a solution to provide containers IPs on the hosts network itself.

[matthew-ewing]

+1

[NickFall]

posting in an epic thread

Hi all - we'll investigate this in our next sprint, commencing Monday 15-August. I'll post back here with an update shortly.

@christophermclellan did said discussion ever happen? the promised update sure didn't @akerouanton please lead us out of this darkness

[enzofrnt]

posting in an epic thread

Hi all - we'll investigate this in our next sprint, commencing Monday 15-August. I'll post back here with an update shortly. @christophermclellan did said discussion ever happen? the promised update sure didn't @akerouanton please lead us out of this darkness

Does that mean that they are working on it ?!

[spamshaker]

6.10.2024 Error response from daemon: invalid subinterface vlan name en0, example formatting is eth0.10 Docker version 27.3.1, build ce1223035a

[akerouanton]

Does that mean that they are working on it ?!

No, not yet. We probably need to use an under-documented macOS feature which is gated behind an entitlement that need to be granted by Apple Developer Support on a case-by-case basis. We've filled the required paperwork for that and are currently waiting for Apple's approval. Afterward, we'll need some time to do exploratory work, etc...

We'll also need to weigh up and prioritize this feature request against other features we've in our backlog. But we hear you, and we know this issue ranks first in terms of reactions / comments.

[enzofrnt]

Does that mean that they are working on it ?!

No, not yet. We probably need to use an under-documented macOS feature which is gated behind an entitlement that need to be granted by Apple Developer Support on a case-by-case basis. We've filled the required paperwork for that and are currently waiting for Apple's approval. Afterward, we'll need some time to do exploratory work, etc...

We'll also need to weigh up and prioritize this feature request against other features we've in our backlog. But we hear you, and we know this issue ranks first in terms of reactions / comments.

Nice to read that, thanks you ! Don't forget that new mac are pretty impressive in energy efficiency and unlocking those capabilities will be a key to use those Mac as "Server".