Closed Davasny closed 3 years ago
Confirming.
Specifying network_mode
in the job config makes docker use specified bridge. While network stanza configures iptables rules like if the container is in nomad
bridge. So port mapping doesn't work. We'd like to have both ability to specify the bridge (network_mode
) and the interface for portmapping (host_network
). I guess cni bridge plugin is able to use multiple bridges, I guess it even possible to use native docker bridges for this purpose. Example configuration with the bridge created with the command:
docker network create --subnet 172.19.0.0/16 --gateway=172.19.0.1 --ip-range 172.19.0.0/24 --driver=bridge -ocom.docker.network.bridge.name=br-test test
Might look like this:
# job group config
network {
mode = "bridge"
bridge_name = "br-test"
port ....
}
# nomad client config
cni_bridge "br-test" {
device = "br-test"
subnet = "172.19.0.1/16"
range = "172.19.1.0/24"
}
Such configuration would allow using cni-bridge with the docker's bridge, since address ranges from the docker and from the nomad(cni) don't overlap (172.19.0.0/24
vs 172.19.1.0/24
). This would allow keeping containers with port-maping and without port-mapping in the same subnet (network_mode
). From the other side - if we could specify multiple cni bridges we would not need docker native bridges.
Hi @Davasny and @urusha!
I've verified this as well. On my test machine here I've configured eth1
as the public
network for Nomad. Then I created a network bridge via docker network create --subnet=192.168.17.0/24 test
, which results in the following bridge configurations:
$ ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
eth0 UP 10.0.2.15/24 fe80::a00:27ff:fec5:bc64/64
eth1 UP 10.199.0.200/24 fe80::a00:27ff:fe56:7ad1/64
docker0 DOWN 172.17.0.1/16
br-d7837cb43544 UP 192.168.17.1/24 fe80::42:53ff:fe79:4d38/64
nomad UP 172.26.64.1/20 fe80::6cf4:33ff:fe3b:710c/64
veth6e107668@eth1 UP fe80::b81d:44ff:fece:fee2/64
vethc6e08f0@if12 UP fe80::5c4f:f5ff:fe37:4ab0/64
I ran @Davasny's job and got the following output of docker inspect :id | jq '.[0].NetworkSettings
. Which gives us an address on the test
network as we'd expect, but that's not what Nomad is configuring for its network.
However I'm sorry to say I think you've run into a known limitation. From the network docs on host networks:
Note: host_network does not currently support task-based mapped ports such as the Docker driver's port_map configuration.
The example there could be more comprehensive but this looks like another case of the same issue. I'd have thought we'd have an issue open for it, but other than other reports like https://github.com/hashicorp/nomad/issues/10001 and https://github.com/hashicorp/nomad/issues/9006 it doesn't look like it. I'll circle up with the original author of this feature to make sure I understand whether the limitation is inherent or just "not done yet". And in the meanwhile, I'll try to make sure this gets surfaced for our roadmapping to fix.
Ok, had a chat with some folks internally here and this is indeed a known limitation. When you set the network_mode
in the Docker configuration, you're asking Docker to define the network namespace, but that's also what the network
block is doing. With Docker tasks, Nomad creates a pause
container and sets the network namespace associated with the network
block to that of the container (ex. container:abcde1134
).
So there's a documentation bug here to fix. The snippet of documentation I quoted above is also wrong when it comes to port mapping; that should all work now. Going to mark this as a docs bug and will ship a PR to clean this all up.
Documentation fixes are merged and will go out with the next website push.
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
If filing a bug please include the following:
Nomad version
Operating system and Environment details
Issue
When using multi-interface feature it's possible to access container port via public IP, but after enabling
network_mode
in task config, container becomes unaccessible.Reproduction steps
network_mode = "test"
network_mode = "test"
and rerun the jobNetwork config
Job file (if appropriate)
Nomad config
Alloc status after step 5 in reproduction
Clone of: https://discuss.hashicorp.com/t/question-how-to-run-task-in-multi-interface-configuration-with-access-to-docker-network/20768
Same issue had @urusha in #8432