docker / roadmap

Welcome to the Public Roadmap for All Things Docker! We welcome your ideas.
https://github.com/orgs/docker/projects/51
Creative Commons Zero v1.0 Universal
1.81k stars 266 forks source link

Support Host Network (--network host) mode on Docker Desktop for Mac and Windows #238

Closed hexcowboy closed 7 months ago

hexcowboy commented 3 years ago

Tell us about your request

Right now, docker run --network host <image> is only available to Linux docker hosts. From the official documentation:

The host networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server.

Which service(s) is this request for?

Docker Desktop for Mac, Docker Desktop for Windows

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Making a docker container with dynamic port mappings is not possible right now.

For example, a container that allows a user to accept connections on arbitrary ports through a netcat listener. During a network penetration test, the user may be testing outbound firewall rules and needs to dynamically accept connections on different ports in the container, eg 53, 80, 443, 1337, 9001, or any other arbitary port. The user will not know these ports before starting the container, so it's not possible for the user to run the container with docker run -p <ports> <image>.

With docker run --network=host, this could be accomplished by just using the same network schema as localhost.

Are you currently working around the issue?

The "best" workaround I know of would to add an EXPOSE 1-1000 in the Dockerfile and run the container with docker run -P <image>, which is very slow and does still limits the range of open ports. Using the full port range of EXPOSE 1-65535 hangs my terminal, so I don't know how effective it is.

The -P flag is documented here: https://docs.docker.com/network/links/#connect-using-network-port-mapping

This is not a sustainable workaround.

Additional context

Amondale commented 3 years ago

Although documentation stats not supported on any O/S but Linux, it also doesn't seem to work correctly (although it doesn't throw error) on Linux flavors on WSL2. The swarm configurations likewise don't function as they do in "bare metal" Linux, which can make for a ton of confusion in the docs.

There is a large and growing population of Mac/Windows users that are not fully supported by the extant documentation, and issues like this (many other networking issues exist on those platforms too).

I plan to follow this thread closely, and would welcome the chance to help refine docs so they apply across the board, or exceptions are noted where they exist.

gramss commented 2 years ago

What is basically needed to make this work? Where would one need to start looking to make this work on MacOS? Any basic information on how to start that would be much appreciated. I think technically this should be doable, also from the community. It's currently just not on the roadmap..?

zfil commented 2 years ago

Having this supported could maybe help with the horrible network performance I see on macos docker ...

This makes for instance using postgres in a container very painful. Running our test suite using a postgres in a container and with docker 4.6: virtualization.framework: 16min hypervisor.framework: 10min locally installed postgres: 3min

christophermclellan commented 2 years ago

Hi - thanks for this issue and sorry for the delayed reply. We'll do some scoping on this and circle back on this thread shortly.

wiredwiz commented 2 years ago

This is certainly a feature I would love to see support for. lacking host network support makes it impossible for me to run services that need to be network aware on anything other than linux, since otherwise they only see the network bridge.

hexcowboy commented 2 years ago

Another limitation I've found - not being able to run a local Redis cluster https://redis.io/docs/manual/scaling/#redis-cluster-and-docker

ReubenFrankel commented 2 years ago

Related to https://github.com/docker/desktop-linux/issues/87

gramss commented 2 years ago

Now with the new release 4.14.0 of Docker Desktop on Mac utilizing the Apple Virtualization Framework, it should be possible to add another network adapter featuring the bridged network aka. network_mode = host ?

https://developer.apple.com/documentation/virtualization/vzvirtualmachineconfiguration/3656724-networkdevices

Is this a big challenge? Can somebody point out the hypervisor configuration for the Apple Virtualization Framework here? :)

Also thanks for putting it in the "Considering" space of the docker-roadmap ! 👍

logical-and commented 1 year ago

+1 - would be really nice to get this implemented as it works on Linux!

jwoodrow commented 1 year ago

Now with the new release 4.14.0 of Docker Desktop on Mac utilizing the Apple Virtualization Framework, it should be possible to add another network adapter featuring the bridged network aka. network_mode = host ?

developer.apple.com/documentation/virtualization/vzvirtualmachineconfiguration/3656724-networkdevices

Is this a big challenge? Can somebody point out the hypervisor configuration for the Apple Virtualization Framework here? :)

Also thanks for putting it in the "Considering" space of the docker-roadmap ! 👍

Is there any way to get more updates on this ? I think this would be an extremely appreciated feature for mac users

davidthornton commented 1 year ago

Here's the link to the filter to track its progress!

https://github.com/docker/roadmap/projects/1?card_filter_query=network+host

dewdrinker19 commented 1 year ago

Yes please. Very much please.

ghost commented 1 year ago

Here's the link to the filter to track its progress!

https://github.com/docker/roadmap/projects/1?card_filter_query=network+host

It doesn't seem to be getting traction or is it too early to tell?

alikhanich commented 1 year ago

I've been waiting for 5 years for that. Can you please do it? If the security is main concern please disable this feature by default.

BlaiseOfGlory commented 1 year ago

bump. I have several tools that I would love to be able to containerize that need layer2 access.

ShutdownRepo commented 1 year ago

For everyone's bummed about this lack, the solution may very well be to switch to a more active project 🤷 It's been five years. For instance, https://orbstack.dev/, while not open-source, seems very promising and already supporting the --network=host mode (among other things). I'm amazed to see independent projects like this one surpass Docker Desktop (for mac in this case) performance and feature-wise.

ghost commented 1 year ago

https://orbstack.dev/

Website has been registered for only 1 week. How well do you know it?

ShutdownRepo commented 1 year ago

https://orbstack.dev/

Website has been registered for only 1 week. How well do you know it?

Not well myself, but I work with 2 people that tested it and that raised excellent feedback.

reececomo commented 1 year ago

Any update on this? Host networking is a huge enabler for local development of WebRTC apps & games.

jupadev commented 1 year ago

it would be great have this flag for mac and windows in order to run some container against host machine for early testing

dontcrash commented 1 year ago

+1 to this

jwoodrow commented 1 year ago

https://orbstack.dev/

Website has been registered for only 1 week. How well do you know it?

Not well myself, but I work with 2 people that tested it and that raised excellent feedback.

I've been using orbstack for the past 2 months and have had 0 complaints apart from details like the fact I can't access logs from the gui or images. It's faster, lighter and has more important features than docker desktop ironically.

Docker has been sitting on this issue for months among many other issues and I just feel like they don't care at this point...

dontcrash commented 1 year ago

OrbStack has issues as well, I tried to run a Traefik container and bind port 53, no matter what I did, it would not bind. Other ports were fine.

kdrag0n commented 1 year ago

OrbStack has issues as well, I tried to run a Traefik container and bind port 53, no matter what I did, it would not bind. Other ports were fine.

@dontcrash This is probably because you already have a service bound to port 53 on macOS. To identify the process:

lsof -Pwni | grep :53

Let me know if you find anything!

dontcrash commented 1 year ago

OrbStack has issues as well, I tried to run a Traefik container and bind port 53, no matter what I did, it would not bind. Other ports were fine.

@dontcrash This is probably because you already have a service bound to port 53 on macOS. To identify the process:

lsof -Pwni | grep :53

Let me know if you find anything!

Binds to OrbStack yes, but it does not work. I have Traefik hosting a UDP tunnel to a DNS server, the request hits OrbStack and nothing further, does not go to the container, another issue I am seeing is that even with other services the source IP is 127.0.0.1, this is not how host networking should behave. This problem would be solved if the Docker team would push this from considering after several years and just do it.

chrisdoherty4 commented 1 year ago

Lets try and keep the conversation focused on the issue (as opposed to OrbStack troubleshooting).

I've been watching the issue for quite some time so I'll add my +1 - we could really use this feature, thanks Docker team.

ilya-girman commented 1 year ago

One more vote for this. We are not able to setup dev environment with our sip proxy inside linux container on windows. The reason is in NAT which stands between host machine and container. If host send UDP packet to the proxy it is important that SRC port stays the same, because proxy is configured to determine routing path based on SRC port where message came from. But with default network "bridge" mode source port is substituted by random value. I tried to setup network_mode: "host" in compose file and came to strange situation. I see that process listens on port 35060 inside container:

netstat -tulpn | grep 35060
tcp        0      0 192.168.65.4:35060      0.0.0.0:*               LISTEN      -                   
udp        0      0 127.0.0.1:35060         0.0.0.0:*                           -                   
udp        0      0 192.168.65.4:35060      0.0.0.0:*                           - 

But I can't find it anywhere at the host machine. Host machine doesn't have IP in subnet 192.168.65.0 at all.

I see that this container still uses separate network 192.168.65.X like it was with default networking.

ping host.docker.internal
PING host.docker.internal (192.168.65.254) 56(84) bytes of data.
64 bytes from 192.168.65.254 (192.168.65.254): icmp_seq=1 ttl=63 time=5.75 ms

But docker ps shows that container has no exposed ports at all (column PORTS is empty) as it should be with network mode "host". I tried to disable firewall on host machine with no success. I read about "transparent" network mode but it is available only in "windows containers" mode.

kcognata commented 1 year ago

Would love to see this feature. It's been a linux exclusive for far too long.

RedNo7 commented 1 year ago

+1 for this.

Orbstack does not have any timing for this either; feedback was there is not enough support to prioritise it. https://github.com/orbstack/orbstack/issues/183

leosamuele221 commented 1 year ago

+1 Cant see client IP on traefik

nrjohnstone commented 1 year ago

Would this also restore the old functionality we used to have with the NAT router way back in 19.x that allowed us to route traffic from our windows host to static IPs on containers? That was excellent functionality that allows you to mimic a deployed environment plus have routable host names in your localhosts file. Having to port map everything and access it via "127.0.0.1" is a very poor substitute, especially if you have quite a number of dependencies you need running in containers locally.

thaJeztah commented 1 year ago

In host-mode networking, the container does not have a networking namespace, and as such doesn't have an IP-address.

nrjohnstone commented 1 year ago

@thaJeztah ah of course, what functionality would it take then to restore the ability to have NAT routing from the host to container IPs running on Windows ? I'm assuming with WSL2 there is some sort of dependency on what MS have done under the hood for WSL2 ?

sourabhjaiswal commented 1 year ago

+1. Would love to see this feature as in some cases we need to run a linux container on windows system and want the IP of host to be used inside container.

andvrz commented 1 year ago

+1 facing the same issue. Found this workaround on Medium but for me it doesn't work.

enzofrnt commented 1 year ago

I'd like to see the inclusion of this feature, and I also have other recommendations to mention.

Is it possible to add the ability to bind the interface from LinuxKit to MacOS concurrently ? This would facilitate the utilization of IPVLAN and MACVLAN with an operational parent interface, or any other mechanism that would allow us to use these types of networks and expose IPs on the local network.

There's a workaround that worked previously on x86, but not on Apple Silicon Mac : docker-tuntap-osx

There is a big lack of networking capabilities on the Mac Version of Docker.

danielfinke commented 1 year ago

@enzofrnt I am using docker-mac-net-connect as a workaround on my M1 Mac. I have a container that listens on a wide port range so I need to route traffic to it directly.

enzofrnt commented 1 year ago

Hi, @danielfinke thanks ! I will try it because I find nothing else, but that will be way better to has it implemented directly in docker…

enzofrnt commented 1 year ago

@danielfinke i'm that will make me able to do make MacVlan or ipvlan directly to my local network. As my computer is 192.168.1.45 i want container in this range and available for other devices.. Any help there ?

danielfinke commented 1 year ago

@enzofrnt It only gets you as far as docker-tuntap-osx in that you can route traffic from the host directly to containers. I think you would need to add a proxy on top if you wanted to use the host's IP, but I am not sure.

enzofrnt commented 1 year ago

@danielfinke i'm currently working on a Mac with M1 chip and they use linux kit instead of HyperKit.. So the tuntap thing didn't work for me. Maybe with some modifications that will work but i'm not able do this kind of modifications. So, i'm juste in front of a wall.

enzofrnt commented 1 year ago

@danielfinke there is what I wonder : image

nikjuice commented 1 year ago

+1

miamilabs commented 1 year ago

I was trying to run home assistant in host mode which did not work for me. For my understanding the mdns traffic was not forwarded correctly. From the reply which i got its because of the host mode is missing on mac os.

quiquelhappy commented 11 months ago

wen windows

outsideryn commented 11 months ago

+1

vadimen commented 8 months ago

any ideas how to https://devops.stackexchange.com/questions/19117/bind-dedicated-ip-to-docker-container-in-windows-server-2022

4-FLOSS-Free-Libre-Open-Source-Software commented 7 months ago

Version Docker Desktop 4.29.0 Settings ⇒ Features in development ⇒ Enable host networking Host networking allows containers that are started with --net=host to use localhost to connect to TCP and UDP services on the host. It will automatically allow software on the host to use localhost to connect to TCP and UDP services in the container. Sign in required.

Enable host networking

vadimen commented 7 months ago

Version Docker Desktop 4.29.0 Settings ⇒ Features in development ⇒ Enable host networking Host networking allows containers that are started with --net=host to use localhost to connect to TCP and UDP services on the host. It will automatically allow software on the host to use localhost to connect to TCP and UDP services in the container. Sign in required.

Enable host networking

that is very cool, can we activate it using powershell on windows server core ?

4-FLOSS-Free-Libre-Open-Source-Software commented 7 months ago

C:\Users\$env:USERNAME\AppData\Roaming\Docker\settings.json

"hostNetworkingEnabled": true,

Version Docker Desktop 4.29.0 Settings ⇒ Features in development ⇒ Enable host networking Host networking allows containers that are started with --net=host to use localhost to connect to TCP and UDP services on the host. It will automatically allow software on the host to use localhost to connect to TCP and UDP services in the container. Sign in required. Enable host networking

that is very cool, can we activate it using powershell on windows server core ?

Try my script:


# Read the JSON file
$path = 'C:\Users\{0}\AppData\Roaming\Docker\settings.json' -f $env:USERNAME
$json = Get-Content -Raw -Path $path | ConvertFrom-Json

# Check if the property exists
if ($json.PSObject.Properties.Name -notcontains 'hostNetworkingEnabled') {
    # Add the property if it does not exist
    $json | Add-Member -Type NoteProperty -Name hostNetworkingEnabled -Value $true
} else {
    # Change the value
    $json.hostNetworkingEnabled = $true
}

# Save the changes back to the file
$json | ConvertTo-Json -Depth 100 | Set-Content -Path $path