Database hostname lookup fails on Kubernetes

[nsmith5]

Classification

• [X] Minor bug

Reproducibility

• [X] Always

Docker information

Running on kubernetes so probably not as relevant, but here it is:

$ docker info
Containers: 74
 Running: 43
 Paused: 0
 Stopped: 31
Images: 37
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: oci runc
Default Runtime: oci
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: 63758e0305bf576fecada317156f4347bc7a2c03 (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: bdb8293590f930c81becb72496aad1a9a6c6abd4-dirty (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
 selinux
Kernel Version: 4.17.3-200.fc28.x86_64
Operating System: Fedora 28 (Cloud Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 2
Total Memory: 3.852 GiB
Name: {{ redacted }}
ID: {{ redacted }}
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true
Registries: docker.io (secure), registry.fedoraproject.org (secure), quay.io (secure), registry.access.redhat.com (secure), registry.centos.org (secure), docker.io (secure)

$ docker images hardware/mailserver --digests --filter "dangling=false"
REPOSITORY                      TAG                 DIGEST                                                                    IMAGE ID            CREATED             SIZE
docker.io/hardware/mailserver   1.1-stable          sha256:293e2a37d49ed7c2ccba12ab38e88e63057d46598832f4eaa0b29b745bee47c5   66477618802b        7 days ago          364 MB

Description

If DBHOST is not specified, dns lookup of the default DBHOST (mariadb) fails in a kubernetes environment.

Steps to reproduce

1. Make a kubernetes deployment for hardware/mailserver
2. Make a mariadb service named 'mariadb'
3. Deploy mailserver deployment

Expected results

Finds mariadb service

Actual results

Does find service, DNS lookup fails

Debugging information

kubectl logs mailserver-*****-****
[INFO] Traefik SSL certificates not used
[INFO] No Let's encrypt live directory found
[INFO] Using /var/mail/ssl/selfsigned/ folder
[INFO] Found DKIM key pair for domain mail.nfsmith.ca - skip creation
[INFO] MariaDB/PostgreSQL hostname not found in /etc/hosts
[ERROR] Container IP not found with embedded DNS server... Abort !
[ERROR] Check your DBHOST environment variable

Configuration (docker-compose.yml, traefik.toml...etc)

Will attach soon...

[hardware]

How DNS resolution works exactly with Kubernetes ? Can you reach other containers with their name from mail server ? What is the default name given by Kubernetes to mariadb container ? Can you reach Kubernetes internal DNS resolver ? Can you provide me an example of resolv.conf generated by Kubernetes ?

Maybe, @fa-at-pulsit will probably have more information on that.

added 'dig search' option for cluster ip resolving (important for k8s) #243

[nsmith5]

Think I found the problem. The dig search option should quote DBHOST:

Currently, on line 302 of run.sh:

 IP=$(dig A ${DBHOST} +short +search)

should be,

IP= $(dig A "$DBHOST" +short +search)

If I open a shell in affected container I get:

$ echo $DBHOST
{:-mariadb}
$ echo "$DBHOST"
mariadb
$ dig A $DBHOST +short +search  # Doesn't work

$ dig A "$DBHOST" +short +search # works..
10.x.x.x

[nsmith5]

How does DNS for services work with docker-compose? Are the service added to /etc/hosts so this DNS lookup is omitted?

[nsmith5]

Nevermind! The quoting was an unrelated human error. DNS lookup issue is related to using hostNetwork: true on Kubernetes. Switching to hostPort should fix this. Sorry for the noise!