basic auth broken in 0.21.0

spolspol commented 5 years ago

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version: 0.21.0

Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release): amazon linux (latest)
Kernel (e.g. uname -a):
Install tools:
Others:

What happened:] Not being prompted for login when accessing ingress host with basic-auth configured.

What you expected to happen: To be prompted for login.

How to reproduce it (as minimally and precisely as possible): Below is ingress configuration I use:

annotations: nginx.ingress.kubernetes.io/auth-type: basic nginx.ingress.kubernetes.io/auth-secret: basic-auth nginx.ingress.kubernetes.io/auth-realm: "-UI" nginx.ingress.kubernetes.io/configuration-snippet: |- allow 127.0.0.0/8; allow 10.0.0.0/8; deny all; satisfy any;

Above config requires generic secret named basic-auth in the root of the namespace created from a file named auth.

This setup works with image 0.20.0 - however does not work when running image version 0.21.0

Once basic auth is configured on on an ingress, issue can be replicated simply by switching between images, everything else remaining the same.

Anything else we need to know:

fuhbar commented 5 years ago

We can confirm this behaviour with the following setup:

nginx ingress controller version 0.21.0
nginx verison 1.15.6

We could narrow this down to the combination of basic auth and whitelisting (allow, satisfy any). If your remove your whitelisting, basic auth will work as expected.

Unfortunately we could not find any pending issue with nginx regarding this behaviour. So could anybody confirm this?

fuhbar commented 5 years ago

Downgrading to 0.20.0 fixed the issue for us. So there seems to be some regression between 0.20.0 and 0.21.0 the way the nginx is configured.

Using a bare nginx 1.15.6 and configure basic auth and whitelisting works quite well. It seems to be related how the nginx ingress controller is using the upstream nginx.

aledbf commented 5 years ago

@spolspol @fuhbar please check if you see the real source IP address in the ingress controller pod logs

fuhbar commented 5 years ago

@aledbf yes, the real IP address is shown in the logs.

nerdinand commented 5 years ago

Here's some more information about this issue. Ingress configuration used in both cases:

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: development
  name: httpbin
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: httpbin-basic-auth
    nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"

    nginx.ingress.kubernetes.io/configuration-snippet: |
      # whitelist office IP (everything else uses basic-auth)
      satisfy any;
      allow 1.2.3.4;
      deny all;

    nginx.ingress.kubernetes.io/server-snippet: |
      # for SEO reasons, we return a 301 Moved Permanently for any GET request with trailing slash
      if ($request_method = GET) {
        rewrite ^/(.*)/$ /$1 permanent;
      }

spec:
  rules:
    - host: httpbin.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: httpbin
              servicePort: 80

Broken nginx.conf that leads to basic auth being deactivated, generated by version 0.21.0: broken.nginx.conf.txt

Working nginx.conf that leads to basic auth working, as expected (with a whitelisted IP), generated by version 0.20.0: working.nginx.conf.txt

I'd be grateful if anyone with more knowledge about nginx could have a look. Alternatively, if anyone could give me some pointers how to edit nginx.conf in a pod to try some things, I could also do that...

matthias-k commented 5 years ago

I have a problem that might be the same underlying bug: I have two ingress definitions that should give the same basic behaviour but only one of them results in having basic-auth:

Here are two ingress definitions that both should serve a service at gocd.mydomain.com with basic auth enabled. Both definitions do in fact serve the service at gocd.mydomain.com, however with the first definition basic auth is not enabled. With the second definition, basic auth is enabled.

Doesn't use basic-auth

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/issuer: letsencrypt-production
    nginx.ingress.kubernetes.io/auth-realm: Authentication Required - foo
    nginx.ingress.kubernetes.io/auth-secret: gocd-basic-auth
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  labels:
    app: gocd
    chart: gocd-1.5.8
    component: server
    heritage: Tiller
    release: bunking-toucan
  name: bunking-toucan-gocd-server
  namespace: default
spec:
  backend:
    serviceName: bunking-toucan-gocd-server
    servicePort: 8153
  rules:
  - host: gocd.mydomain.com
  tls:
  - hosts:
    - gocd.mydomain.com
    secretName: gocd-mydomain-com

Does use basic-auth:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/issuer: letsencrypt-production
    nginx.ingress.kubernetes.io/auth-realm: Authentication Required for CICD
    nginx.ingress.kubernetes.io/auth-secret: gocd-basic-auth
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  labels:
    app: gocd
  name: gocd
  namespace: default
spec:
  rules:
  - host: gocd.mydomain.com
    http:
      paths:
      - backend:
          serviceName: bunking-toucan-gocd-server
          servicePort: 8153
  tls:
  - hosts:
    - gocd.mydomain.com
    secretName: gocd-mydomain-com

ajcann commented 5 years ago

I am also experiencing this issue when combining basic auth with a satisfy any and allowing specific ip ranges. Oddly the result is that neither basic auth nor ip whitelisting is functioning, i.e. all connections are allowed. The logs do show the real ip of the client, if that's useful.

As suggested above, I reverted to 0.20.0 and the problem disappeared.

Looking at the release notes from 0.20.0 -> 0.21.0 ( https://github.com/kubernetes/ingress-nginx/blob/master/Changelog.md ) gives me a few leads but nothing certain yet.

aledbf commented 5 years ago

The fix for this issue is in PR #3649. The dev image quay.io/kubernetes-ingress-controller/nginx-ingress-controller:dev contains the fix.

Any of the affected users can confirm the fix, please?

ajcann commented 5 years ago

@aledbf I can confirm that my particular use case of 'satisfy any' is fixed with use of the dev image. Thanks for catching this!

kubernetes / ingress-nginx

basic auth broken in 0.21.0 #3480