Closed dottodot closed 4 years ago
However once I've applied these changes I get lots of errors such as the following 6���Zك7�g̮["\�/�+�0�,��'g�(k��̨̩̪������������������$j�#@�
It seems HTTPS traffic is being sent to the HTTP port, please check the port mapping in the ingres-nginx service and your DO console
@aledbf my service ports are
ports:
- name: http
nodePort: 32342
port: 80
protocol: TCP
targetPort: http
- name: https
nodePort: 31346
port: 443
protocol: TCP
targetPort: https
and the port forwarding on my load balancer is set to TCP on port 80 > TCP on port 32342 TCP on port 443 > TCP on port 31346
Can confirm this issue, same configuration here across the board.
Looking for a solution to this as well. All public requests work but internal traffic to a host of the ingress fail.
2019/05/23 19:59:51 [error] 411#411: *870261 broken header: "��*EpS�M;I��K��WT}�:^ͼ�0���0�,�(�$��
����kjih9876�����2�.�*�&���=5" while reading PROXY protocol, client: 10.244.2.1, server: 0.0.0.0:443
It works for me using the helm chart with the following values:
## nginx configuration
## https://github.com/helm/charts/blob/master/stable/nginx-ingress/values.yaml
## https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-proxy-protocol
## https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-loadbalancer
controller:
config:
use-proxy-protocol: "true"
service:
externalTrafficPolicy: "Local"
annotations:
# https://www.digitalocean.com/docs/kubernetes/how-to/configure-load-balancers/
# https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/annotations.md
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
I've got a similar setup. Which is working if I access the host publicly. However, accessing the host from within the cluster seems to fail (i.e. a server side request from one pod to another pod using the host)
Ingress
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: ingress
annotations:
certmanager.k8s.io/cluster-issuer: letsencrypt
kubernetes.io/ingress.class: nginx
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
spec:
tls:
- hosts:
- 'example.com'
- '*.example.com'
secretName: example-tls
rules:
- host: example.com
http:
paths:
- backend:
serviceName: example-app
servicePort: 80
- host: api.example.com
http:
paths:
- backend:
serviceName: example-backend
servicePort: 80
...
Config
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
data:
compute-full-forwarded-for: "true"
use-forwarded-headers: "true"
use-proxy-protocol: "true"
Curl Example Error
$ curl -v https://api.example.com
* Rebuilt URL to: https://api.example.com/
* Trying (123.456.789.000...
* TCP_NODELAY set
* Connected to api.example.com (123.456.789.000) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to api.example.com:443
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to api.example.com:443
Log from failed request
[error] 661#661: *981009 broken header: "�/�9��ނ���R�6ަ�@%Qe�lG�3.���0�,�(�$��
����kjih9876�����2�.�*�&���=5" while reading PROXY protocol, client: 10.244.3.1, server: 0.0.0.0:443
I don't think service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
should be set in the Ingress annotation. It has probably no effect here.
My understanding is it must be set on the nginx-ingress service. I think it simply tells DO to activate the "Use Proxy Protocol" setting when the load balancer is created.
However, accessing the host from within the cluster seems to fail (i.e. a server side request from one pod to another pod using the host)
@tlaverdure that's because you are no specifying the flag --haproxy-protocol
in the curl command. If you enable proxy protocol in the ingress controller you need to decode it.
I don't think service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true" should be set in the Ingress annotation. It has probably no effect here.
That is correct.
@dperetti I have service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
on the nginx-ingress service and get the same issue as @tlaverdure
@aledbf I'm not using curl command but get the same errors.
I also have: externalTrafficPolicy: Local
on the nginx-ingress-controller service and of course use-proxy-protocol: "true"
in the configmap.
My Kubernetes version is 1.14.1-do.2 .
@dperetti thanks for the tip. I think I added it initially when testing and forgot to remove after I added that annotation to the ingress-nginx
service.
@aledbf I'm experiencing this with any type of server side http request. Curl was used to verify the issue, but any server side scripting language that makes a http request to the host (i.e. node) is failing.
I also have externalTrafficPolicy: Local
so don't think that's related.
Stupid check, but if you go to the load balancer's settings in the DO admin panel, it's enabled, right? Because if it's not while use-proxy-protocol = true you end up with the same kind of encoding mismatch.
Yes, Proxy Protocol is enabled.
Just tested setting use-proxy-protocol: "false"
in the ConfigMap. This kills my ability to reach the host externally but allows me to access the host within the cluster.
OK I've had some advise back from Digital Ocean
Current options for workaround are to have pods access other DOKS services through their resolvable service names or by using the desired services clusterIP.
Using the service hostname as described below or the service cluster IP could be used for traffic originating inside the cluster. kubectl get svc
Will return a list of your services and their clusterIP. You can use this IP to reach your service within the cluster. Note this IP will only work from within the cluster.
You can find documentation for accessing services properly in kubernetes here: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#services
The usage for accessing a service is: my-svc.my-namespace.svc.cluster.local:443
Using either of these methods will mean that that traffic no longer needs to go outside the traffic and interact with the proxy and can just get direct access to the service.
Only problem is I'm not entirely sure how to find which pods have the issue and need updating.
Also when i turn on proxy protocol the logs suggest that not all requests have a broken header so how do I identify what's causing the broken header and fix it.
I ran into this and at one point, I deleted the ingress service and recreated it and it worked. I got the broken headers issue when I had the nginx configmap set, but not the annotations on the ingress service that created the DO LB. Manually configuring "Proxy Protocol" on the LB w/ the Web UI didn't work for me.
Anyways, here is a config that worked for me:
mandatory.yaml
# Please note this file has been customized from the original at:
# https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
#... stuff in the mandatory.yaml file that doesn't need to be customized.
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
# https://www.digitalocean.com/community/questions/how-to-set-up-nginx-ingress-for-load-balancers-with-proxy-protocol-support?answer=50244
data:
use-forwarded-headers: "true"
compute-full-forwarded-for: "true"
use-proxy-protocol: "true"
# ... more stuff in the mandatory.yaml file that doesn't need to be customized.
cloud-generic.yaml
# Please note this file has been customized from the original at:
# https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
# https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/annotations.md
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"
spec:
externalTrafficPolicy: Local
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: https
port: 443
targetPort: https
ingress.yaml (I wanted a whitelist for only CloudFlare)
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: allup
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: "2400:cb00::/32, 2606:4700::/32, 2803:f800::/32, 2405:b500::/32, 2405:8100::/32, 2a06:98c0::/29, 2c0f:f248::/32, 173.245.48.0/20, 103.21.244.0/22, 103.22.200.0/22, 103.31.4.0/22, 141.101.64.0/18, 108.162.192.0/18, 190.93.240.0/20, 188.114.96.0/20, 197.234.240.0/22, 198.41.128.0/17, 162.158.0.0/15, 104.16.0.0/12, 172.64.0.0/13, 131.0.72.0/22"
spec:
tls:
- secretName: cloudflare-tls-cert
rules:
- host: example.com
http:
paths:
- path: /
backend:
serviceName: app-www-service
servicePort: http
Is it possible to allow nginx to listen on the same http/https port with and without proxy protocol like this setup?
@w1ndy no. You need a custom template for that https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/custom-template/
The same issue happens in Kubernetes cluster on top of OpenStack cloud (using openstack-cloud-controller-manager).
The Ingress service could be accessed from outside the cluster, but not from the cluster node or in the pod.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
/remove-lifecycle stale
I'm a bit unclear whether this issue is the same one we just encountered, but perhaps the following is helpful:
We have a microservice architecture, where services talk to each other via the axios library, all inside the same cluster. What we'd misconfigured was the URL by which the services talk to each other. We had one service talk to the other via the external dns record by which the target service was known e.g foo.domain.com, causing traffic for it to go all the way out and back into the cluster again. When nginx tried to handle the request, the header looked broken because the request wasn't preceded by the instruction PROXY TCP4 10.8.0.18 [TARGET_IP] 55966 443
(which is what happens when you curl --haproxy-protocol
, and is what happens to all inbound traffic handled by the ingress controller when you enable the "Proxy Protocol" setting)
By changing the URL of the target service to the internal dns record e.g http://[container].[namespace].svc.cluster.local
by which it was known, traffic was sent directly to the target, not back through the ingress controller.
Temporary solved it by using Cloudflare proxy mode for subdomains. In this case, all traffic goes via Cloudflare proxy and works the same way for all traffic (Internal inside the cluster and external traffic).
Looking forward to the resolution of this issue.
For users of DigitalOcean's managed Kubernetes offering (DOKS) and possibly others: you are most likely running into a specific bypassing behavior of kube-proxy that causes requests to never leave the cluster but go straight to the internal pod. For receiving pods that expect a certain protocol to be implemented by the load balancer (like proxy protocol or TLS termination), this breaks work flows.
Pointers to the related upstream issue and workarounds are described in our CCM README. The tracking issue to address the problem long-term for DOKS is at digitalocean/DOKS#8.
Hi,
A workaround is to add the service.beta.kubernetes.io/do-loadbalancer-hostname
annotation to the service with a hostname pointing to the loadbalancer IP.
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
service.beta.kubernetes.io/do-loadbalancer-hostname: "do-k8s.example.com"
spec:
externalTrafficPolicy: Local
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https
Thank you service.beta.kubernetes.io/do-loadbalancer-hostname
seems to work. No more broken headers using proxy protocol
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten
.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten
Hello, I have the same issue, but my cluster is self hosted, and my Load Balancer is MetalLB. So the 2 headers added on the service have no effets. Any idea on how to solve it ?
Hi @Oznup, Does your LoadBalancer have an External-IP ? On bare-metal, my only workaround is to have my LoadBalancers with the "pending" message for working ingresses. Maybe help : https://github.com/kubernetes/kubernetes/issues/66607#issuecomment-601100890
For users of DigitalOcean's managed Kubernetes offering (DOKS) and possibly others: you are most likely running into a specific bypassing behavior of kube-proxy that causes requests to never leave the cluster but go straight to the internal pod. For receiving pods that expect a certain protocol to be implemented by the load balancer (like proxy protocol or TLS termination), this breaks work flows.
Pointers to the related upstream issue and workarounds are described in our CCM README. The tracking issue to address the problem long-term for DOKS is at digitalocean/DOKS#8.
This is precisely the problem we are facing and the workaround was to add the hostname as described. I can confirm that this also works with wildcard DNS matches.
Workaround does not work for me.
adding service.beta.kubernetes.io/do-loadbalancer-hostname
works for DO. But it's not clear what to do at the first time when reading DO docs (from here and here)
Basically it contains the following steps:
kubectl annotate service ingress-nginx-controller service.beta.kubernetes.io/do-loadbalancer-hostname=mydomain.com -n ingress-nginx --overwrite
After this you can create your Ingress resource and use Cert Manager to issue SSL certificate as usual, and when you get
your ingress it should show ADDRESS
which is your domain. Hope this help
Stupid check, but if you go to the load balancer's settings in the DO admin panel, it's enabled, right? Because if it's not while use-proxy-protocol = true you end up with the same kind of encoding mismatch.
This solved my problem
I face the same problem,how to solve?
I have the same issue with the option while integrating it with AKS services. Same issue, same error: AKS v1.21.9 Ingress-nginx v1.3.1 (v1.1.1 the hook) but somehow I get it working by using proxy-protocol: 'true' and it just works.
Hello,
I am running EKS with kubernetes nginx ingress on NLB with SSL termination but the proxy protocol annotations was not working.
controller:
electionID: external-ingress-controller-leader
ingressClassResource:
name: external-nginx
enabled: true
default: false
controllerValue: "k8s.io/external-ingress-nginx"
config:
use-proxy-protocol": "true"
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "environment=dev,owner=platform-engineering"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-west-2:<REDACTED>:certificate/<REDACTED>
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-<REDACTED>, subnet-<REDACTED>
# below annotations were not working
# service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
# service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: "preserve_client_ip.enabled=false,proxy_protocol_v2.enabled=true"
But once the NLB and ingress objects are created, the services wont be reachable because the proxy protocol on the NLB target groups was not turned on automatically even with the provided annotations. I got the below error in the controller pod logs,
2022/11/03 20:50:23 [error] 32#32: *26209 broken header: "GET /index.html HTTP/1.1
Host: nginx.<REDACTED>
Connection: keep-alive
Cache-Control: max-age=0
sec-ch-ua: "Google Chrome";v="107", "Chromium";v="107", "Not=A?Brand";v="24"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
DNT: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: cross-site
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,ta;q=0.8
" while reading PROXY protocol, client: <REDACTED>, server: 0.0.0.0:80
Once I turned them on manually through console (NLB -> Listeners -> Target Group attributes), it works (I get 200 response in the same controller logs).
But both the annotations in the helm values does not seem to work.
Can anyone help on this ?
However, accessing the host from within the cluster seems to fail (i.e. a server side request from one pod to another pod using the host)
@tlaverdure that's because you are no specifying the flag
--haproxy-protocol
in the curl command. If you enable proxy protocol in the ingress controller you need to decode it.
Appending '--haproxy-protocol' to curl commands works
I still have the same issue but on bare-metal, and I am not able to resolve it. Though, I have a question. Why does everybody use the externalpolicy local?
This policy preserves the source IP anyway. What is the benifit of Proxy Protocol if I am going to use the local policy?
For anyone stumbling across this using Hetzner LoadBalancers:
You propably need to add "load-balancer.hetzner.cloud/uses-proxyprotocol": 'true'
as an annotation to the ingress service (at the same place you are likely already setting "load-balancer.hetzner.cloud/location"
and/or "load-balancer.hetzner.cloud/type"
)
In our case the problem was that we tried to deploy two environments on nginx (backend) port 80 - one basic and one going through proxy:
server
{
listen 80;
server_name test.com;
.
.
.
}
server
{
listen 80 proxy_protocol;
server_name production.com;
.
.
.
}
We thought that traffic from first LB configured to use proxy just reach block "listen 80 proxy_protocol" while traffic from second LB just reach block "listen 80;", but it won't happen due to error "broken header: ... while reading PROXY protocol, client: ...". Nginx does not allow configuring one port in these two ways and crashes on searching right server block, even if the correct one exists. One of solutions is to deploy one of the environments on another port - this works fine.
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT
NGINX Ingress controller version: 0.24.0
Kubernetes version (use
kubectl version
): Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-04T04:48:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:31:33Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}Environment:
What happened: Digital Ocean now allows for use of Proxy Protocol https://www.digitalocean.com/docs/kubernetes/how-to/configure-load-balancers/#proxy-protocol So I've added the annotation to my service
and updated my config as follows
However once I've applied these changes I get lots of errors such as the following
Digital Ocean's response to these error was
I can't see what I'm missing
What you expected to happen: No errors