Open jhaezebr opened 3 years ago
lgfa29
commented
3 years ago Thanks for the report @jhaezebr.
I wonder if this is related to https://github.com/hashicorp/nomad/issues/11033, more specifically the last item:
with a
network.dnsblock, bothdockerandexectasks get the Nomad-managedresolv.confgenerated byGenerateDNSMount
Could you try changing your host DNS configuration to see if it solves your problem?
jhaezebr
commented
3 years ago @lgfa29 I'm not sure if this was what you were asking, but it works when i add the following to the job configs:
network {
dns {
searches = ["service.consul"]
}
}Using the servers options the containers both contain the resolv.conf search option from the host
network {
dns {
servers = ["10.0.0.2"]
}
}resolv.conf in both of the containers:
search home
nameserver 10.0.0.2So it seems that it is indeed the same problem?
jrasell
commented
3 years ago Hi @jhaezebr and thanks for raising this. I've taken a look into this today and wanted to share some my findings for you and other readers.
I firstly confirmed that just using vanilla Docker results in an expected resolv.conf file:
$ docker run --net=bridge --dns-search=service.consul busybox sleep 3600 &
$ docker exec 1494764cc5bd cat /etc/resolv.conf
search service.consul
nameserver 10.0.2.3When running your example which uses bridge networking and inspecting the resolv.conf file, the below is detailed. Interestingly, the container ID detailed in the path, is not that of the busybox container, but that of the pause container that is used to secure the network namespace. This indicates to me that the container inherits the DNS configuration from the pause container, and therefore the dns_search_domains is ignored/overwritten by Docker when running in this configuration mode. It's worth noting, this setup is different to the initial Docker test which doesn't utilise network namespaces nor pause containers.
$ docker inspect 4f7c8ab8362c |jq '.[0].ResolvConfPath'
"/var/lib/docker/containers/9c877407dc33a2e84829faf04a0e72c74732df55ff7b4d25e4610ec83b14a624/resolv.conf"
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4f7c8ab8362c busybox:1.28 "sleep 3600" 7 minutes ago Up 7 minutes bridged-0a72140d-6151-3506-ed85-c787a01b9959
9c877407dc33 gcr.io/google_containers/pause-amd64:3.1 "/pause" 7 minutes ago Up 7 minutes nomad_init_0a72140d-6151-3506-ed85-c787a01b9959The Nomad job specification allows for setting of DNS configuration via the network stanza. Utilising this in the job specification included below, results in the desired resolv.conf.
job "bridged" {
datacenters = ["dc1"]
group "bridged" {
network {
mode = "bridge"
dns {
searches = ["service.consul"]
}
}
task "bridged" {
driver = "docker"
config {
image = "busybox:1.28"
command = "sleep"
args = ["3600"]
}
}
}
}$ nomad alloc exec cb1b2d32 cat /etc/resolv.conf
search service.consul
nameserver 10.0.2.3I will discuss this internally and would be interested in your feedback on the approach; but I feel using the network.dns stanza is the preferred method when using Nomad with Docker in bridge mode and network namespaces. We could mark the dns_search_domains Docker configuration parameter as such, indicating the exact use cases and alternatives. An alternative approach to a resolution would be to patch the creation of the pause container so that the dns_search_domains is passed to this container creation. I feel this is a suboptimal fix though as dns_search_domains can be set per Docker task in bridge mode, however, the pause container is set once per network namespace and thus Nomad job task group.
jhaezebr
commented
3 years ago Hi @jrasell ,
I just tried and apparently the network.dns also works in the nonbridged docker container, even with multiple task per group. So to me it seems that network.dns would be the preferred choice to configure any job, and keep dns_search_domains to the special cases where some containers need to have a special configuration? As you suggested the documentation could then indicate that this dns_search_domains option does not work in bridged network mode.
In any case, I'd like to thank you and @lgfa29 for the help on this problem.
tgross
commented
2 years ago
Nomad version
Nomad v1.1.5 (117a23d2cdf26a1837b21c84f30c8c0f3441e927)
Operating system and Environment details
Vagrant with ubuntu/focal64 CNI bridge plugin v1.0.1 Consul v1.9.4, Revision 10bb6cb3b
Issue
When running a job using the bridge network mode, the _dns_searchdomains parameter is not applied.
Reproduction steps
Run nomad and consul as root, both in dev mode Use bind to redirect DNS to consul
Expected Result
Actual Result
Job file (if appropriate)
Bridged job:
Non bridged job:
Nomad logs
Click to expand