search

kubernetes / ingress-nginx

Ingress NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.53k stars 8.26k forks source link

add wso2 oauth2 for prometheus #4850

Closed hari819 closed 4 years ago

hari819 commented 4 years ago

hello ,

i am trying to add wso2 oauth2 for prometheus , nginx-controller :0.9.0

i am using following annotations ,

"nginx.ingress.kubernetes.io/auth-signin": "https://wso2_url/commonauth?rd=/", "nginx.ingress.kubernetes.io/auth-url": "https://wso2_url/oauth2/authorize?response_type=id_token+token&client_id=SECRET&redirect_uri=https://prometheus.*****/&nonce=abc&scope=openid+profile"

if i use the auth-url in browser , it is showing me the login page to auth provider ,it gives 302 error and once i login with credentials it gives 500 error . I can see the access_toke in the browser , but 500 error is thrown.

But when i directly access the prometheus url in brwoser ,i get 500 error with below logs from nginx controller ,and i dont even see the login page for auth provider

10.244.4.1 - [10.244.4.1] - - [19/Dec/2019:15:52:24 +0000] "GET / HTTP/2.0" 302 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" 0 0.099 [external-authentication] 10.0.218.183:443 0 0.098 302 10.244.4.1 - [10.244.4.1] - - [19/Dec/2019:15:52:24 +0000] "GET / HTTP/2.0" 500 595 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" 314 0.099 [external-authentication] - - - - 2019/12/19 15:52:24 [error] 12510#12510: *348950 auth request unexpected status: 302 while sending to client, client: 10.244.4.1, server: prometheus.*, request: "GET / HTTP/2.0", host: "prometheus.**" 10.244.4.1 - [10.244.4.1] - - [19/Dec/2019:15:52:24 +0000] "GET /favicon.ico HTTP/2.0" 302 0 "https://prometheus..**/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" 0 0.048 [external-authentication] 10.0.218.183:443 0 0.048 302 10.244.4.1 - [10.244.4.1] - - [19/Dec/2019:15:52:24 +0000] "GET /favicon.ico HTTP/2.0" 500 595 "https://prometheus..**/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" 106 0.048 [external-authentication] - - - - 2019/12/19 15:52:24 [error] 12510#12510: *348950 auth request unexpected status: 302 while sending to client, client: 10.244.4.1, server: prometheus.., request: "GET /favicon.ico HTTP/2.0", host: "prometheus..**", referrer: "https://prometheus..**/"

any help on what am i missing here ?

aledbf commented 4 years ago

nginx-controller :0.9.0

Please upgrade ASAP. You are using an ancient version of the ingress controller (almost two years old)

If you check the Changelog there were at least 20 fixes for external auth.

hari819 commented 4 years ago

i have updated controller to 0.26.0 and i see the same 302 and 500 error while accessing prometheus URL,

logs from nginx-controller,

10.240.0.5 - - [19/Dec/2019:17:14:59 +0000] "GET / HTTP/2.0" 302 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" 0 0.068 [xxx-prometheus-service-80] [] xx.xx.xxx.xx:443 0 0.052 302 94b671124a23a6f87a1e647a04ade17c 10.244.4.1 - - [19/Dec/2019:17:14:59 +0000] "GET /oauth2/authorize?response_type=id_token+token&client_id=SECRET&redirect_uri=https://prometheus.********/callback&nonce=abc&scope=openid+profile HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" 884 0.046 [xxx-wso2is-service-443] [] 10.244.4.73:443 0 0.048 302 1838c39fc721b298dfcf1cc057bafa75 2019/12/19 17:14:59 [error] 277#277: *4644 auth request unexpected status: 302 while sending to client, client: 10.240.0.5, server: prometheus.****, request: "GET / HTTP/2.0", host: "prometheus.****" 10.240.0.5 - - [19/Dec/2019:17:14:59 +0000] "GET / HTTP/2.0" 500 585 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" 314 0.068 [xxx-prometheus-service-80] [] - - - - 94b671124a23a6f87a1e647a04ade17c 2019/12/19 17:15:00 [error] 277#277: *4644 auth request unexpected status: 302 while sending to client, client: 10.240.0.5, server: prometheus.****, request: "GET /favicon.ico HTTP/2.0", host: "prometheus.****", referrer: "https://prometheus.****/" 10.244.4.1 - - [19/Dec/2019:17:15:00 +0000] "GET /oauth2/authorize?response_type=id_token+token&client_id=SECRET&redirect_uri=https://prometheus.********/callback&nonce=abc&scope=openid+profile HTTP/1.1" 302 0 "https://prometheus.****/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" 831 0.046 [xxx-wso2is-service-443] [] 10.244.4.73:443 0 0.048 302 ef34a92d93df3b036645ddf3455e4623 10.240.0.5 - - [19/Dec/2019:17:15:00 +0000] "GET /favicon.ico HTTP/2.0" 302 0 "https://prometheus.****/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" 0 0.050 [xxx-prometheus-service-80] [] xx.xx.xxx.xx:443 0 0.052 302 652e8d00824715b15e9df26026e1c626 10.240.0.5 - - [19/Dec/2019:17:15:00 +0000] "GET /favicon.ico HTTP/2.0" 500 585 "https://prometheus.****/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" 106 0.050 [xxx-prometheus-service-80] [] - - - - 652e8d00824715b15e9df26026e1c626

hari819 commented 4 years ago

Here is the ingress for wso2 auth service ,

{ "kind": "Ingress",

}

}, "spec": { "tls": [ { "hosts": [ "host.**" ], "secretName": "XXX-secret" } ], "rules": [ { "host": "host.**", "http": { "paths": [ { "path": "/", "backend": { "serviceName": "wso2is-service", "servicePort": 443 } } ] } } ] } }

and here is the ingress for prometheus , { "kind": "Ingress",

  "kubernetes.io/ingress.class": "nginx",
  "nginx.ingress.kubernetes.io/auth-signin": "https://wso2_url/commonauth?rd=/",
  "nginx.ingress.kubernetes.io/auth-url": "https://wso2_url:443/oauth2/authorize?response_type=id_token+token&client_id=SECRET&redirect_uri=https://prometheus.******/callback&nonce=abc&scope=openid+profile",
  "nginx.ingress.kubernetes.io/backend-protocol": "HTTPS"

}

}, "spec": { "tls": [ { "hosts": [ "prometheus.XXXXXXXXX" ], "secretName": "XXX-secret" } ], "rules": [ { "host": "prometheus.XXXXXXXXXXXX", "http": { "paths": [ { "path": "/", "backend": { "serviceName": "prometheus-service", "servicePort": 80 } } ] } } ] } }

i am confused why it is not redirecting to login page of auth_provider when i access prometheus link , am i missing some basic configuration here ?

hari819 commented 4 years ago

@aledbf , can u provide me any clue what am i missing here ,why the login page to auth-proxy is not shown on opening prometheus URL , i can get the same when i enter the complete auth-url directly in the browser .

aledbf commented 4 years ago

"nginx.ingress.kubernetes.io/backend-protocol": "HTTPS" "servicePort": 80

I think you need to check what are you defining in each ingress. It seems the 302 is related to the redirect from prometheus

aledbf commented 4 years ago

Also, please post the ingress with the right format, it doesn't look a valid ingress

hari819 commented 4 years ago

@aledbf , sorry for confusion ,posting the two ingress now,

here is the ingress for prometheus

apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-tls-prometheus annotations:

kubernetes.io/ingress.class: "nginx"
external-dns.alpha.kubernetes.io/target: ingress.XXX.XXX
external-dns.alpha.kubernetes.io/ttl: "30",
nginx.ingress.kubernetes.io/auth-signin: "https://wso2_auth_signin_url",
nginx.ingress.kubernetes.io/auth-url: "https://wso2_auth_url",

spec: tls:

and here is the ingress for wso2-auth service

apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-tls-passthrough-is annotations:

kubernetes.io/ingress.class: "nginx"
ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/secure-backends: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "route"
nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"
external-dns.alpha.kubernetes.io/target: ingress.XXX.XXX
external-dns.alpha.kubernetes.io/ttl: "30"

spec: tls:

hari819 commented 4 years ago

@aledbf , Yes you are correct the 302 is from prometheus where there is a redirection to "/graph", could you please help me on this , as mentioned before if i use the auth-url directly it lands in wso2 sign-in page and once i login with credentials it throws 302 and 500 errros .

but if i access the prometheus url ,it directly goes to 500 error page.

aledbf commented 4 years ago

Yes you are correct the 302

If the prometheus service is returning 302, then use the port 443 directly.

hari819 commented 4 years ago

thanks @aledbf , i will try with 443 port

hari819 commented 4 years ago

@aledbf , it is the same error combination of 302 and 500 even after changing the port to 443,

10.240.0.5 - - [06/Jan/2020:04:55:15 +0000] "GET / HTTP/2.0" 302 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0" 0 0.015 [prometheus-service-443] [] 10.0.218.183:443 0 0.016 302 004b3304656cf03e71221309d62ba317 2020/01/06 04:55:15 [error] 2787#2787: *14895022 auth request unexpected status: 302 while sending to client, client: 10.240.0.5, server: prometheus.XXX.XXX.com, request: "GET / HTTP/2.0", host: "prometheus.XXX.XXX.com" 10.240.0.5 - - [06/Jan/2020:04:55:15 +0000] "GET / HTTP/2.0" 500 183 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0" 212 0.015 [prometheus-service-443] [] - - - - 004b3304656cf03e71221309d62ba317

i have changed container port in prometheus deployment, port in prometheus service and also the port in prometheus ingress to 443 , same result . and in browser also i get 500 error "500 Internal Server Error "

hari819 commented 4 years ago

@aledbf , i am adding the full nginx configuration generated by the ingress ,

`server { server_name prometheus.XXX.XXX.com ;

    listen 80  ;
    listen 443  ssl http2 ;

    set $proxy_upstream_name "-";

    ssl_certificate_by_lua_block {
            certificate.call()
    }

    location = /_external-auth-Lw {
            internal;

            # ngx_auth_request module overrides variables in the parent request,
            # therefore we have to explicitly set this variable again so that when the parent request
            # resumes it has the correct value set for this variable so that Lua can pick backend correctly
            set $proxy_upstream_name "prometheus-service-80";

            proxy_pass_request_body     off;
            proxy_set_header            Content-Length "";
            proxy_set_header            X-Forwarded-Proto "";

            proxy_set_header            Host                    wso2.XXX.com;
            proxy_set_header            X-Original-URL          $scheme://$http_host$request_uri;
            proxy_set_header            X-Original-Method       $request_method;
            proxy_set_header            X-Sent-From             "nginx-ingress-controller";
            proxy_set_header            X-Real-IP               $remote_addr;

            proxy_set_header            X-Forwarded-For        $remote_addr;

            proxy_set_header            X-Auth-Request-Redirect $request_uri;

            proxy_buffering                         off;

            proxy_buffer_size                       4k;
            proxy_buffers                           4 4k;
            proxy_request_buffering                 on;
            proxy_http_version                      1.1;

            proxy_ssl_server_name       on;
            proxy_pass_request_headers  on;

            client_max_body_size        1m;

            # Pass the extracted client certificate to the auth provider

            set $target https://wso2_url/oauth2/authorize?response_type=id_token+token&client_id=SECRET&redirect_uri=https://prometheus.XXX.XXX.com&nonce=abc&scope=openid+profile;
            proxy_pass $target;
    }

    location / {

            set $namespace      "XXX";
            set $ingress_name   "ingress-tls-prometheus";
            set $service_name   "prometheus-service";
            set $service_port   "80";
            set $location_path  "/";

            rewrite_by_lua_block {
                    lua_ingress.rewrite({
                            force_ssl_redirect = false,
                            ssl_redirect = true,
                            force_no_ssl_redirect = false,
                            use_port_in_redirects = false,
                    })
                    balancer.rewrite()
                    plugins.run()
            }

            header_filter_by_lua_block {

                    plugins.run()
            }
            body_filter_by_lua_block {

            }

            log_by_lua_block {

                    balancer.log()

                    monitor.call()

                    plugins.run()
            }

            port_in_redirect off;

            set $balancer_ewma_score -1;
            set $proxy_upstream_name "prometheus-service-80";
            set $proxy_host          $proxy_upstream_name;
            set $pass_access_scheme  $scheme;
            set $pass_server_port    $server_port;
            set $best_http_host      $http_host;
            set $pass_port           $pass_server_port;

            set $proxy_alternative_upstream_name "";

            # this location requires authentication
            auth_request        /_external-auth-Lw;
            auth_request_set    $auth_cookie $upstream_http_set_cookie;
            add_header          Set-Cookie $auth_cookie;

            set_escape_uri $escaped_request_uri $request_uri;
            error_page 401 = https://wso2_url/commonauth?rd=/;

            client_max_body_size                    1m;

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;

            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Request-ID           $req_id;
            proxy_set_header X-Real-IP              $remote_addr;

            proxy_set_header X-Forwarded-For        $remote_addr;

            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   15s;
            proxy_send_timeout                      600s;
            proxy_read_timeout                      600s;

            proxy_buffering                         off;
            proxy_buffer_size                       4k;
            proxy_buffers                           4 4k;

            proxy_max_temp_file_size                1024m;

            proxy_request_buffering                 on;
            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout;
            proxy_next_upstream_timeout             0;
            proxy_next_upstream_tries               3;

            proxy_pass http://upstream_balancer;

            proxy_redirect                          off;

    }

}`,

could you please help me on this , am i missing any annotations required , many examples i checked seems to work fine without much trouble , https://stackoverflow.com/questions/53425501/access-log-off-has-not-effect-in-nginx

But i am confused what am i missing here

hari819 commented 4 years ago

@aledbf , i tried to add extra annotations using "configuration-snippet" , and nginx controller throws exception as "unknown directive "|""

"nginx.ingress.kubernetes.io/configuration-snippet": "| auth_request_set $token $upstream_http_authorization proxy_set_header Authorization $token",

Error: exit status 1 2020/01/06 16:23:28 [emerg] 31138#31138: unknown directive "|" in /tmp/nginx-cfg917364355:1923 nginx: [emerg] unknown directive "|" in /tmp/nginx-cfg917364355:1923 nginx: configuration file /tmp/nginx-cfg917364355 test failed

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 4 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

fejta-bot commented 4 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 4 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/4850#issuecomment-639058218): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.