Auth not using kube-dns

nabeelio commented 5 years ago

The gist: When using the auth-url annotation to point to a K8 Service, it doesn't seem to go through the kube-dns resolver.

I found a similar report on rancher, here: https://github.com/rancher/rancher/issues/13876 However looking here, I haven't found anything, though maybe my search-foo stinks.

NGINX Ingress controller version:

NGINX Ingress controller
  Release:    0.20.0
  Build:      git-e8d8103
  Repository: https://github.com/kubernetes/ingress-nginx.git
-------------------------------------------------------------------------------

I1116 15:44:26.814562       6 flags.go:180] Watching for Ingress class: nginx
nginx version: nginx/1.15.5
built by gcc 8.2.0 (Debian 8.2.0-7)
built with OpenSSL 1.1.0h  27 Mar 2018

Kubernetes version (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
[debug.txt](https://github.com/kubernetes/ingress-nginx/files/2589955/debug.txt)

Environment:

Cloud provider or hardware configuration: Azure AKS

OS (e.g. from /etc/os-release):

$ cat os-release
PRETTY_NAME="Debian GNU/Linux buster/sid"
NAME="Debian GNU/Linux"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Kernel (e.g. uname -a):

$ uname -a
Linux ingress-nginx-ingress-controller-8f68b94d-x5zcm 4.15.0-1023-azure #24~16.04.1-Ubuntu SMP Wed Aug 29 12:54:36 UTC 2018 x86_64 GNU/Linux

Install tools: I used the helm chart, only setting the load balancer IP. RBAC is enabled.

What happened:

When setting up the auth-url to use a deployed service, the DNS lookup isn't done. A message like this is in the logs:

[error] 50#50: *706 auth-svc could not be resolved (2: Server failure), client: 70.114.197.89, server: _, request: "GET /config/api/v1/apps HTTP/1.1", subrequest: "/_external-auth-L2NvbmZpZw", host: "10.240.1.10"
[error] 50#50: *706 auth request unexpected status: 502 while sending to client, client: 70.114.197.89, server: _, request: "GET /config/api/v1/apps HTTP/1.1", host: "10.240.1.10"

From the debug log:

2018/11/16 15:38:59 [debug] 51#51: *59 http run request: "/_external-auth-L2NvbmZpZw?"
2018/11/16 15:38:59 [debug] 51#51: *59 http upstream check client, write event:1, "/_external-auth-L2NvbmZpZw"
2018/11/16 15:38:59 [debug] 51#51: *59 http upstream resolve: "/_external-auth-L2NvbmZpZw?"
2018/11/16 15:38:59 [error] 51#51: *59 auth-svc could not be resolved (2: Server failure), client: **REMOVED**, server: _, request: "GET /config/api/v1/apps HTTP/1.1", subrequest: "/_external-auth-L2NvbmZpZw", host: "10.240.1.10"
2018/11/16 15:38:59 [debug] 51#51: *59 finalize http upstream request: 502
2018/11/16 15:38:59 [debug] 51#51: *59 finalize http proxy request
2018/11/16 15:38:59 [debug] 51#51: *59 http finalize request: 502, "/_external-auth-L2NvbmZpZw?" a:1, c:2

It does work when it's pointed to the service IP address. I tried adding this to see if it would force it, but it just seems to ignore it and not add it to the config:

nginx.ingress.kubernetes.io/auth-snippet: |
  resolver 10.0.0.10 valid=30s;

How to reproduce it (as minimally and precisely as possible):

Deploy any service, and point the auth-url to it:

nginx.ingress.kubernetes.io/auth-url: http://auth-svc

I've attached a log with nginx in debug in case that yields any clues. I see the nginx config. I also attached the nginx config

debug.txt nginx_config.txt

aledbf commented 5 years ago

@nabeelio this is correct. Right now we use the lua-resty-dns library where the search and ndots options are not available

nabeelio commented 5 years ago

Ah dang... is there a workaround available? Otherwise this renders this ingress unusable for us.

Does the auth-snippet only look for the proxy_* variables? I wonder if forcing the resolver line into the location = /_external-auth-L2NvbmZpZw block would make it work.

I'm trying to find a way to try it manually, but I don't have root access into the container.

aledbf commented 5 years ago

Ah dang... is there a workaround available?

Yes, we are going to replace that library with https://github.com/Kong/lua-resty-dns-client where search and ndots are available

aledbf commented 5 years ago

@nabeelio just in case if you use the full FQDN this works (auth-svc.<namespace>.svc.cluster.local)

nabeelio commented 5 years ago

@aledbf Thanks, seemed like that worked! I hadn't tried it - the other ticket I looked at, it seemed like they were using the full FQDN and it hadn't worked for them. But I see now they're in a different namespace, so maybe that has something to do with it. But that worked and that's good enough for me.

Appreciate the help!

Queuecumber commented 5 years ago

@aledbf was there any progress on replacing that library? I still can't resolve cluster internal addresses without the full FQDN

aledbf commented 5 years ago

@Queuecumber not yet

vmitris commented 3 years ago

This is still reproducing with k8s.gcr.io/ingress-nginx/controller:v0.41.2. For requests coming from within clusters where $host is a FQDN all the requests cannot be resolved. If hardcoded static IP is used it works fine but this is not a solution as there are multiple scenarios when IP is not known at deployment time.

Please reopen it as it seems there is no solution yet. The one proposed by @aledbf with auth-svc..svc.cluster.local does not work :)

jastBytes commented 2 years ago

It looks like I have a similar issue. But after a rollout restart of the nginx ingress controller it works and after a while it stops working. We're using k8s.gcr.io/ingress-nginx/controller:v1.0.5 and it seems that out of a sudden DNS resolving for external auth stops working. Restarting the controller fixes it.

digiserg commented 2 years ago

I can't say whether this is a coincidence or not but I've disabled ipv6 DNS resolution and it has resolved the problem for us.

wackazong commented 1 year ago

FYI I managed to work around this in the following way: add a sidecar container to the ingress-nginx-controller pods which contains another nginx server and listens on port 81. Configure that nginx server to pass all requests to the external auth url using proxy_pass. Then, use http://127.0.0.1:81 as auth-url. Since proxy_pass uses the DNS resolution of its own server we have normal DNS resolution here.

kubernetes / ingress-nginx

Auth not using kube-dns #3426