Closed wdullaer closed 2 years ago
@theunrealgeek, any comments on this one
/assign @strongjz
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
/remove-lifecycle rotten
This issue is still relevant I believe
@wdullaer, do you have a PR in mind.
Thanks, ; Long
On Thu, 5 May, 2022, 12:30 AM wdullaer, @.***> wrote:
/remove-lifecycle rotten
This issue is still relevant I believe
— Reply to this email directly, view it on GitHub https://github.com/kubernetes/ingress-nginx/issues/8086#issuecomment-1117697256, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZVWWZZO6TI34AEVSHYZTVILCOVANCNFSM5K5VMXJQ . You are receiving this because you commented.Message ID: @.***>
Wrapping the code here with
{{ if not (eq $proxySetHeader "grpc_set_header") }}
# Allow websocket connections
{{ $proxySetHeader }} Upgrade $http_upgrade;
{{ if $location.Connection.Enabled}}
{{ $proxySetHeader }} Connection {{ $location.Connection.Header }};
{{ else }}
{{ $proxySetHeader }} Connection $connection_upgrade;
{{ end }}
{{ end }}
will fix the issue for grpc backends.
If you really want to be spec compliant this is probably not sufficient: the HTTP/2 spec prohibits the use of the Connection header, and this will only prevent its use for gRPC based backends. In the future other web servers may start rejecting requests.
The project has to be spec compliant.
/triage accepted /help
; Long Wu Yuan
On 06-May-2022, at 1:49 PM, wdullaer @.***> wrote:
Wrapping the code here https://github.com/kubernetes/ingress-nginx/blob/main/rootfs/etc/nginx/template/nginx.tmpl#L1357-L1362 with
{{ if not (eq $proxySetHeader "grpc_set_header") }}
# Allow websocket connections {{ $proxySetHeader }} Upgrade $http_upgrade; {{ if $location.Connection.Enabled}} {{ $proxySetHeader }} Connection {{ $location.Connection.Header }}; {{ else }} {{ $proxySetHeader }} Connection $connection_upgrade; {{ end }}
{{ end }} will fix the issue for grpc backends.
If you really want to be spec compliant this is probably not sufficient: the HTTP/2 spec prohibits the use of the Connection header, and this will only prevent its use for gRPC based backends. In the future other web servers may start rejecting requests.
— Reply to this email directly, view it on GitHub https://github.com/kubernetes/ingress-nginx/issues/8086#issuecomment-1119370987, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZVWXJZU4EVKJDYWHPSCTVITIXVANCNFSM5K5VMXJQ. You are receiving this because you commented.
@longwuyuan: This request has been marked as needing help from a contributor.
Please ensure that the issue body includes answers to the following questions:
For more details on the requirements of such an issue, please see here and ensure that they are met.
If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help
command.
@wdullaer nit help. Please fix the md formatting in the original post. Only some parts after "what happened" are malformed.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/reopen
/remove-lifecycle rotten
Please send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
@k8s-triage-robot: Closing this issue, marking it as "Not Planned".
NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
Issue is applicable to master based on the code I checked
Kubernetes version (use
kubectl version
):Environment:
How was the ingress-nginx-controller installed: Using kustomize. The only relevant parameter from the configmap is
upstream-keepalive-connections
Current State of the controller: Not Applicable
Others:
What happened: As of 3 months ago, the go-grpc http/2 server performs stricter enforcement of HTTP/2 requests. Specifically, any request that includes a
Connection
header is to be considered malformed. See here for relevant code: https://github.com/grpc/grpc-go/blame/master/internal/transport/http2_server.go#L410-L416 This MDN page also highlights the risk of usingConnection
headers in HTTP/2 requests and responses: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ConnectionUnfortunately the nginx.conf from the ingress controller will always insert a
Connection
to 'enable websockets' when the config variableUpstreamKeepaliveConnections
is set to0
. See https://github.com/kubernetes/ingress-nginx/blob/main/rootfs/etc/nginx/template/nginx.tmpl#L384-L392 and https://github.com/kubernetes/ingress-nginx/blob/main/rootfs/etc/nginx/template/nginx.tmpl#L1261-L1267If the upstream is configured as HTTP/2 or GRPC, those stanza's should not be included in the config at all. (Enabling websockets makes little sense for gRPC services anyway).
Nginx will log errors of the form
when trying to proxy to a go-grpc based service.
When setting the
GRPC_GO_LOG_SEVERITY_LEVEL=info
GRPC_GO_LOG_VERBOSITY_LEVEL=2
headers, the following logs will be printed in the gRPC service:What you expected to happen: Request should be proxied
How to reproduce it:
Install the ingress controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml
Install a gRPC application built with the latest go-grpc (you can follow the instructions here for the k8s part: https://github.com/kubernetes/ingress-nginx/tree/main/docs/examples/grpc)
You may want to set the aforementioned headers to get some actionable debug output from the webserver.
kubectl apply -f your-svc.yaml
Create an ingress
echo " apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: foo-bar annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/backend-protocol: "GRPC" spec: ingressClassName: nginx # omit this if you're on controller version below 1.0.0 rules:
make a request
POD_NAME=$(k get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx -o NAME) kubectl exec -it -n ingress-nginx $POD_NAME -- grpcurl -H 'Host: foo.bar' localhost list
Anything else we need to know: You can workaround the issue by using a server snippet annotation that 'blanks' out the internal nginx.conf variables for this server: