search

kubernetes / ingress-nginx

Ingress-NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.3k stars 8.22k forks source link

HTTPS ExternalName services 502 proxy error #4494

Closed dwatrous closed 5 years ago

dwatrous commented 5 years ago

This issue is very similar to https://github.com/kubernetes/ingress-nginx/issues/1600, but the solution on that issue hasn't worked for me.

I have an Ingress with multiple paths and a mix of traditional Service resources for container workloads and ExternalName Service. I have confirmed the ExternalName Service is working, but the nginx Ingress is not working.

I have the following Service resources

[centos@k8s-master-0 ~]$ kubectl get svc -n sonar-cube-front-end-cicd
NAME                                  TYPE           CLUSTER-IP      EXTERNAL-IP                   PORT(S)    AGE
forgerock-proxy                       ExternalName   <none>          gateways01.hrpt.com   <none>     20h
loginproxy-service-master             ClusterIP      10.233.52.154   <none>                        5000/TCP   85d
msweb-service-master                  ClusterIP      10.233.9.6      <none>                        80/TCP     85d
sonar-cube-front-end-service-master   ClusterIP      10.233.38.176   <none>                        81/TCP     85d

And the ExternalName Service is defined as follows

[centos@k8s-master-0 ~]$ kubectl get svc forgerock-proxy -o yaml -n sonar-cube-front-end-cicd
apiVersion: v1
kind: Service
metadata:
  annotations:
  creationTimestamp: "2019-08-26T14:50:07Z"
  name: forgerock-proxy
  namespace: sonar-cube-front-end-cicd
  resourceVersion: "37144748"
  selfLink: /api/v1/namespaces/sonar-cube-front-end-cicd/services/forgerock-proxy
  uid: cbe34c82-c810-11e9-b56e-fa163e6c897e
spec:
  externalName: gateways01.hrpt.com
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}

I have confirmed this works by creating an Alpine container in the same namespace and executing a request, as follows

[centos@k8s-master-0 ~]$ kubectl attach alpine-76c4b9b99c-bxvlv -c alpine -i -t
/ # curl -k -v -sb -X POST -H "Content-type: application/json" "https://forgerock-proxy.sonar-cube-front-end-cicd:443/myauth/services/v1.0/authentication/signon?realm=sw_hrp" --
data "{\"emplid\":\"00001239829\",\"userpassword\":\"1234\"}" -H "Accept: application/json"
* Could not resolve host: POST
*   Trying 192.168.12.115:443...
* TCP_NODELAY set
* Connected to forgerock-proxy.sonar-cube-front-end-cicd (192.168.12.115) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / DHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Texas; L=Austin; O=TriNet Group Inc; CN=*.hrpt.com
*  start date: Jan 14 13:09:01 2019 GMT
*  expire date: Apr 13 13:39:01 2021 GMT
*  issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2012 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1K
*  SSL certificate verify ok.
> POST /myauth/services/v1.0/authentication/signon?realm=sw_hrp HTTP/1.1
> Host: forgerock-proxy.sonar-cube-front-end-cicd
> User-Agent: curl/7.65.1
> Content-type: application/json
> Accept: application/json
> Content-Length: 46
>
* upload completely sent off: 46 out of 46 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 27 Aug 2019 11:08:12 GMT
< Server: Oracle-HTTP-Server-11g
< Content-Length: 106
< X-Powered-By: Servlet/3.0 JSP/2.2
< Content-Type: application/json
< Content-Language: en
<
* Connection #1 to host forgerock-proxy.sonar-cube-front-end-cicd left intact
VALID_RESPONSE

I then updated my Ingress as follows

[centos@k8s-master-0 ~]$ kubectl get ingress sonar-cube-front-end-ingress-master -o yaml -n sonar-cube-front-end-cicd
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/secure-backends: "true"
    kubernetes.io/ingress.class: nginx
  creationTimestamp: "2019-06-02T14:23:17Z"
  generation: 6
  name: sonar-cube-front-end-ingress-master
  namespace: sonar-cube-front-end-cicd
  resourceVersion: "37430890"
  selfLink: /apis/extensions/v1beta1/namespaces/sonar-cube-front-end-cicd/ingresses/sonar-cube-front-end-ingress-master
  uid: f720041c-8541-11e9-a17d-fa163ed6baf9
spec:
  rules:
  - host: master.sonar-cube-front-end-cicd.dc.k8s.com
    http:
      paths:
      - backend:
          serviceName: sonar-cube-front-end-service-master
          servicePort: 81
        path: /sonar-cube-front-end
      - backend:
          serviceName: msweb-service-master
          servicePort: 80
        path: /
      - backend:
          serviceName: loginproxy-service-master
          servicePort: 5000
        path: /loginproxy
      - backend:
          serviceName: forgerock-proxy
          servicePort: 443
        path: /myauth
  tls:
  - hosts:
    - master.sonar-cube-front-end-cicd.dc.k8s.com
    secretName: sonar-cube-front-end-cicd-tls-secret-autogen
status:
  loadBalancer: {}

But when I try to make a request, it fails with a 502 Bad Gateway error

/ # curl -k -v -sb -X POST -H "Content-type: application/json" "https://master.sonar-cube-front-end-cicd.dc.k8s.com/myauth/services/v1.0/authentication/signon?re
alm=sw_hrp" --data "{\"emplid\":\"00001239829\",\"userpassword\":\"1234\"}" -H "Accept: application/json"
* Could not resolve host: POST
*   Trying 10.50.20.49:443...
* TCP_NODELAY set
* Connected to master.sonar-cube-front-end-cicd.dc.k8s.com (10.50.20.49) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.sonar-cube-front-end-cicd.dc.k8s.com
*  start date: Aug  7 23:35:01 2019 GMT
*  expire date: Nov  5 23:35:01 2019 GMT
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5570f7269540)
> POST /myauth/services/v1.0/authentication/signon?realm=sw_hrp HTTP/2
> Host: master.sonar-cube-front-end-cicd.dc.k8s.com
> User-Agent: curl/7.65.1
> Content-type: application/json
> Accept: application/json
> Content-Length: 46
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* We are completely uploaded and fine
< HTTP/2 502
< server: nginx/1.15.10
< date: Tue, 27 Aug 2019 11:08:41 GMT
< content-type: text/html
< content-length: 158
< strict-transport-security: max-age=15724800; includeSubDomains
<
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.15.10</center>
</body>
</html>
* Connection #1 to host master.sonar-cube-front-end-cicd.dc.k8s.com left intact

I started a shell in the nginx container and see that it has this configuration

[centos@k8s-master-0 ~]$ kubectl exec -it nginx-ingress-controller-7f8f8cc5fc-jq6vq -n ingress-nginx -- /bin/bash
www-data@nginx-ingress-controller-7f8f8cc5fc-jq6vq:/etc/nginx$ cat nginx.conf
...
    ## start server master.sonar-cube-front-end-cicd.dc.k8s.com
    server {
        server_name master.sonar-cube-front-end-cicd.dc.k8s.com ;

        listen 80;

        set $proxy_upstream_name "-";
        set $pass_access_scheme $scheme;
        set $pass_server_port $server_port;
        set $best_http_host $http_host;
        set $pass_port $pass_server_port;

        listen 443  ssl http2;

        # PEM sha: b7bfd48e2c0f0758bfcca547f08399ad48552829
        ssl_certificate                         /etc/ingress-controller/ssl/default-fake-certificate.pem;
        ssl_certificate_key                     /etc/ingress-controller/ssl/default-fake-certificate.pem;

        ssl_certificate_by_lua_block {
            certificate.call()
        }

        location /sonar-cube-front-end {

            set $namespace      "sonar-cube-front-end-cicd";
            set $ingress_name   "sonar-cube-front-end-ingress-master";
            set $service_name   "sonar-cube-front-end-service-master";
            set $service_port   "81";
            set $location_path  "/sonar-cube-front-end";

            rewrite_by_lua_block {
                lua_ingress.rewrite({
                    force_ssl_redirect = true,
                    use_port_in_redirects = false,
                })
                balancer.rewrite()
                plugins.run()
            }

            header_filter_by_lua_block {

                plugins.run()
            }
            body_filter_by_lua_block {

            }

            log_by_lua_block {

                balancer.log()

                monitor.call()

                plugins.run()
            }

            if ($scheme = https) {
                more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains";
            }

            port_in_redirect off;

            set $proxy_upstream_name    "sonar-cube-front-end-cicd-sonar-cube-front-end-service-master-81";
            set $proxy_host             $proxy_upstream_name;

            client_max_body_size                    1m;

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;

            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Request-ID           $req_id;
            proxy_set_header X-Real-IP              $the_real_ip;

            proxy_set_header X-Forwarded-For        $the_real_ip;

            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            proxy_set_header X-Original-URI         $request_uri;

            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_buffering                         off;
            proxy_buffer_size                       4k;
            proxy_buffers                           4 4k;
            proxy_request_buffering                 on;

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout;
            proxy_next_upstream_tries               3;

            proxy_pass http://upstream_balancer;

            proxy_redirect                          off;

        }

        location /myauth {

            set $namespace      "sonar-cube-front-end-cicd";
            set $ingress_name   "sonar-cube-front-end-ingress-master";
            set $service_name   "forgerock-proxy";
            set $service_port   "443";
            set $location_path  "/myauth";

            rewrite_by_lua_block {
                lua_ingress.rewrite({
                    force_ssl_redirect = true,
                    use_port_in_redirects = false,
                })
                balancer.rewrite()
                plugins.run()
            }

            header_filter_by_lua_block {

                plugins.run()
            }
            body_filter_by_lua_block {

            }

            log_by_lua_block {

                balancer.log()

                monitor.call()

                plugins.run()
            }

            if ($scheme = https) {
                more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains";
            }

            port_in_redirect off;

            set $proxy_upstream_name    "sonar-cube-front-end-cicd-forgerock-proxy-443";
            set $proxy_host             $proxy_upstream_name;

            client_max_body_size                    1m;

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;

            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Request-ID           $req_id;
            proxy_set_header X-Real-IP              $the_real_ip;

            proxy_set_header X-Forwarded-For        $the_real_ip;

            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            proxy_set_header X-Original-URI         $request_uri;

            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_buffering                         off;
            proxy_buffer_size                       4k;
            proxy_buffers                           4 4k;
            proxy_request_buffering                 on;

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout;
            proxy_next_upstream_tries               3;

            proxy_pass http://upstream_balancer;

            proxy_redirect                          off;

        }

        location /loginproxy {

            set $namespace      "sonar-cube-front-end-cicd";
            set $ingress_name   "sonar-cube-front-end-ingress-master";
            set $service_name   "loginproxy-service-master";
            set $service_port   "5000";
            set $location_path  "/loginproxy";

            rewrite_by_lua_block {
                lua_ingress.rewrite({
                    force_ssl_redirect = true,
                    use_port_in_redirects = false,
                })
                balancer.rewrite()
                plugins.run()
            }

            header_filter_by_lua_block {

                plugins.run()
            }
            body_filter_by_lua_block {

            }

            log_by_lua_block {

                balancer.log()

                monitor.call()

                plugins.run()
            }

            if ($scheme = https) {
                more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains";
            }

            port_in_redirect off;

            set $proxy_upstream_name    "sonar-cube-front-end-cicd-loginproxy-service-master-5000";
            set $proxy_host             $proxy_upstream_name;

            client_max_body_size                    1m;

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;

            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Request-ID           $req_id;
            proxy_set_header X-Real-IP              $the_real_ip;

            proxy_set_header X-Forwarded-For        $the_real_ip;

            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            proxy_set_header X-Original-URI         $request_uri;

            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_buffering                         off;
            proxy_buffer_size                       4k;
            proxy_buffers                           4 4k;
            proxy_request_buffering                 on;

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout;
            proxy_next_upstream_tries               3;

            proxy_pass http://upstream_balancer;

            proxy_redirect                          off;

        }

        location / {

            set $namespace      "sonar-cube-front-end-cicd";
            set $ingress_name   "sonar-cube-front-end-ingress-master";
            set $service_name   "msweb-service-master";
            set $service_port   "80";
            set $location_path  "/";

            rewrite_by_lua_block {
                lua_ingress.rewrite({
                    force_ssl_redirect = true,
                    use_port_in_redirects = false,
                })
                balancer.rewrite()
                plugins.run()
            }

            header_filter_by_lua_block {

                plugins.run()
            }
            body_filter_by_lua_block {

            }

            log_by_lua_block {

                balancer.log()

                monitor.call()

                plugins.run()
            }

            if ($scheme = https) {
                more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains";
            }

            port_in_redirect off;

            set $proxy_upstream_name    "sonar-cube-front-end-cicd-msweb-service-master-80";
            set $proxy_host             $proxy_upstream_name;

            client_max_body_size                    1m;

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;

            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Request-ID           $req_id;
            proxy_set_header X-Real-IP              $the_real_ip;

            proxy_set_header X-Forwarded-For        $the_real_ip;

            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            proxy_set_header X-Original-URI         $request_uri;

            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_buffering                         off;
            proxy_buffer_size                       4k;
            proxy_buffers                           4 4k;
            proxy_request_buffering                 on;

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout;
            proxy_next_upstream_tries               3;

            proxy_pass http://upstream_balancer;

            proxy_redirect                          off;

        }

    }
    ## end server master.sonar-cube-front-end-cicd.dc.k8s.com

NGINX Ingress controller version: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1

Kubernetes version (use kubectl version): [centos@k8s-master-0 ~]$ kubectl version Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

Environment:

What you expected to happen: I expect the Ingress to allow traffic to flow through to the External service.

How to reproduce it (as minimally and precisely as possible): see above

aledbf commented 5 years ago

From the example

curl -k -v -sb -X POST -H "Content-type: application/json" "https://forgerock-proxy.sonar-cube-front-end-cicd:443/myauth/services/v1.0/authentication/signon?realm=sw_hrp"

  • backend: serviceName: forgerock-proxy servicePort: 443 path: /myauth

that service requires an https connection. You need to create a different ingress rule for that path, for the same host and add the annotation https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#backend-protocol (without this annotation, nginx uses http to reach the service)

dwatrous commented 5 years ago

I just added the annotation you mention, but I'm still getting a 502. This is what I have.

[centos@k8s-master-0 ~]$ kubectl get ing sonar-cube-front-end-ingress-master -o yaml -n sonar-cube-front-end-cicd
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/secure-backends: "true"
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
  creationTimestamp: "2019-06-02T14:23:17Z"
  generation: 6
  name: sonar-cube-front-end-ingress-master
  namespace: sonar-cube-front-end-cicd
  resourceVersion: "37463793"
  selfLink: /apis/extensions/v1beta1/namespaces/sonar-cube-front-end-cicd/ingresses/sonar-cube-front-end-ingress-master
  uid: f720041c-8541-11e9-a17d-fa163ed6baf9
spec:
  rules:
  - host: master.sonar-cube-front-end-cicd.dc.k8s.com
    http:
      paths:
      - backend:
          serviceName: sonar-cube-front-end-service-master
          servicePort: 81
        path: /sonar-cube-front-end
      - backend:
          serviceName: msweb-service-master
          servicePort: 80
        path: /
      - backend:
          serviceName: loginproxy-service-master
          servicePort: 5000
        path: /loginproxy
      - backend:
          serviceName: forgerock-proxy
          servicePort: 443
        path: /myauth
  tls:
  - hosts:
    - master.sonar-cube-front-end-cicd.dc.k8s.com
    secretName: sonar-cube-front-end-cicd-tls-secret-autogen
status:
  loadBalancer: {}
dwatrous commented 5 years ago

I also see that when I add that annotation, all of the paths return 502, not just the /myauthpath.

I see that this annotation changes the nginx.conf by making these all https proxy_pass https://upstream_balancer;

aledbf commented 5 years ago

I also see that when I add that annotation, all of the paths return 502, not just the /myauthpath.

Well, you added the backend-protocol protocol to the ingress where you have all the paths. that means nginx will try to use https to reach any of those paths.

aledbf commented 5 years ago

I see that this annotation changes the nginx.conf by making these all https proxy_pass https://upstream_balancer;

This is a common question. For that reason, we added a comment https://github.com/kubernetes/ingress-nginx/blob/master/rootfs/etc/nginx/template/nginx.tmpl#L411

dwatrous commented 5 years ago

Is there some way to annotate only one backend for HTTPS?

Also, is there some way to tell it not to validate SSL? I see this annotation nginx.ingress.kubernetes.io/proxy-ssl-verify, but the default is 'off', so would that help to set it?

aledbf commented 5 years ago

Like I posted in my first comment, you need to create a new ingress for that path

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
  name: forgerock-proxy-sonar-cube-front-end-ingress-master
  namespace: sonar-cube-front-end-cicd
spec:
  rules:
  - host: master.sonar-cube-front-end-cicd.dc.k8s.com
    http:
      paths:
      - backend:
          serviceName: forgerock-proxy
          servicePort: 443
        path: /myauth
aledbf commented 5 years ago

Also, is there some way to tell it not to validate SSL? I see this annotation nginx.ingress.kubernetes.io/proxy-ssl-verify, but the default is 'off', so would that help to set it?

That feature is not released yet https://github.com/kubernetes/ingress-nginx/pull/4327

In the meantime, you can use the configuration-snippet annotation to set proxy_ssl_verify off;

dwatrous commented 5 years ago

The requests are now being proxied. Thank you.

However, the request is being modified in a way that is causing it to fail. From the requests above, I know the service is working with this curl request when operating directly against the service and not going through the nginx proxy.

curl -k -v \
-X POST \
-H "Content-type: application/json" \
-H "Accept: application/json" \
"https://master.sonar-cube-front-end-cicd.dc.k8s.com/myauth/services/v1.0/authentication/signon?realm=sw_hrp" \
--data "{\"emplid\":\"00001239829\",\"userpassword\":\"1234\"}"

Does nginx pass through the headers or modifies them?

I see header_filter_by_lua_block, but I'm not sure what it does.

Is it possible to disable all the proxy_set_header directives for that path?

aledbf commented 5 years ago

Is it possible to disable all the proxy_set_header directives for that path?

Not really.

How are you testing this and what's the output in the log? The error is in your service?

dwatrous commented 5 years ago

I'm testing it with the exact curl commands I show above. When I access the service directly, I get a 200 valid response back. When I go through the nginx proxy, I get a 302 redirect, which means that the auth endpoint didn't like the request. That can happen if I don't set the content-type and accept headers. It may happen if other headers are set, but I'm not sure. Is there some way to know exactly what is included in the request from nginx to my service?

afsilvasantos commented 4 years ago

Hi @dwatrous Did you manage to solve your problem? I am facing the same issue now

matteosilv commented 10 months ago

facing the same issue too now