Move web socket upgrade config to route

pitabwire commented 8 months ago

Description:

Secure websocket is not working on envoy gateway, If I connect to the server without tls, just plain ws://, everything works fine however once I connect with wss:// of which the connection is terminated at the lb as shown in the deployment file below, the connection fails. My assumption was initially a simple mis configuration I however haven't managed to get it working.

Repro steps:


apiVersion: apps/v1
kind: Deployment
metadata:
name: websocket-test
spec:
selector:
matchLabels:
app.kubernetes.io/name: websocket-test
template:
metadata:
labels:
app.kubernetes.io/name: websocket-test
spec:
containers:
- name: websocket-test
image: jmalloc/echo-server           
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 300m
memory: 500Mi
requests:
cpu: 10m
memory: 80Mi
ports:
- name: default
containerPort: 80
env:
- name: PORT
value: '80'

apiVersion: v1 kind: Service metadata: name: websocket-test labels: app.kubernetes.io/name: websocket-test spec: ports:

name: default port: 80 targetPort: 80 selector: app.kubernetes.io/name: websocket-test

apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: websocket-test spec: parentRefs:

kind: Gateway name: default namespace: envoy-gateway-system hostnames:
"test.domain.com" rules:
matches:
- path: value: "/" backendRefs:
- group: "" kind: Service name: websocket-test port: 80 weight: 1

apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: default namespace: envoy-gateway-system spec: gatewayClassName: envoy-gateway listeners:

name: http protocol: HTTP port: 80 allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-enabled: "true"
name: https protocol: HTTPS port: 443 allowedRoutes: namespaces: from: Selector selector: matchLabels: gateway-enabled: "true" tls: mode: Terminate certificateRefs:
- name: cloudflare-origin-cert


*Environment*:
> - Gateway version is : docker.io/envoyproxy/gateway:v0.6.0 
   - Envoy version is : envoyproxy/envoy:distroless-v1.28-latest

*Logs*:
```console
envoy-envoy-gateway-system-default-b435bbfd-58c8b4d8d-j4r6h {"start_time":"2024-02-16T13:25:57.423Z","method":"GET","x-envoy-origin-path":"/","protocol":"HTTP/1.1","response_code":"403","response_flags":"-","response_code_details":"upgrade_failed","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"0","x-envoy-upstream-service-time":"-","x-forwarded-for":"197.239.12.79,10.244.1.69","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36","x-request-id":"e041de6c-b9d9-43d7-8395-ed023cb92771",":authority":"ws.stawi.io","upstream_host":"-","upstream_cluster":"httproute/realtime/websocket-test/rule/0","upstream_local_address":"-","downstream_local_address":"10.244.5.222:10443","downstream_remote_address":"10.244.1.69:38732","requested_server_name":"-","route_name":"httproute/realtime/websocket-test/rule/0/match/0/ws_stawi_io"}                                                                                                                                                                                                                                                                           
envoy-envoy-gateway-system-default-b435bbfd-58c8b4d8d-cxkrc {"start_time":"2024-02-16T13:26:03.477Z","method":"GET","x-envoy-origin-path":"/","protocol":"HTTP/1.1","response_code":"403","response_flags":"-","response_code_details":"upgrade_failed","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"0","x-envoy-upstream-service-time":"-","x-forwarded-for":"197.239.12.79,10.244.1.69","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36","x-request-id":"d0ca06e8-0872-4f94-9c41-04a2c3ee1fc9",":authority":"ws.stawi.io","upstream_host":"-","upstream_cluster":"httproute/realtime/websocket-test/rule/0","upstream_local_address":"-","downstream_local_address":"10.244.4.70:10443","downstream_remote_address":"10.244.1.69:46876","requested_server_name":"-","route_name":"httproute/realtime/websocket-test/rule/0/match/0/ws_stawi_io"}                                                                                                                                                                                                                                                                            
envoy-envoy-gateway-system-default-b435bbfd-58c8b4d8d-j4r6h {"start_time":"2024-02-16T13:26:09.396Z","method":"GET","x-envoy-origin-path":"/","protocol":"HTTP/1.1","response_code":"403","response_flags":"-","response_code_details":"upgrade_failed","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"0","x-envoy-upstream-service-time":"-","x-forwarded-for":"197.239.12.79,10.244.1.69","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36","x-request-id":"cf65f33f-216e-410d-b62e-0017731e34db",":authority":"ws.stawi.io","upstream_host":"-","upstream_cluster":"httproute/realtime/websocket-test/rule/0","upstream_local_address":"-","downstream_local_address":"10.244.5.222:10443","downstream_remote_address":"10.244.1.69:37792","requested_server_name":"-","route_name":"httproute/realtime/websocket-test/rule/0/match/0/ws_stawi_io"}

zetaab commented 8 months ago

@arkodg ideas how this can be configured? I am thinking that can be this enabled by default? Is it wrong to configure it everywhere by default https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/upgrades

if cannot be enabled by default, which is the correct place to configure that? BackendTrafficPolicy?