search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
25.07k stars 4.82k forks source link

Envoy TLS config causes SSL WRONG_VERSION_NUMBER error #37254

Open ziler-orca opened 2 days ago

ziler-orca commented 2 days ago

Title: Envoy TLS config causes SSL WRONG_VERSION_NUMBER

Version: envoyproxy/envoy:v1.32.1

Description: Trying setup an envoy HTTP proxy in Kubernetes cluster. The internal pods forward 

curl -k http://envoy-egress:8080 -I http://www.google.com         # This is successful
curl -k http://envoy-egress:8080 -I https://www.google.com       # This is failing with OpenSSL wrong number

What issue is being seen? Describe what should be happening instead of When the tester pod issues the curl command, envoy throws an curl: (35) OpenSSL/3.0.13: error:0A00010B:SSL routines::wrong version number error.

root@tester-pod:/# curl -v -k http://envoy-egress:8080 -I https://www.gmail.com
* Host envoy-egress:8080 was resolved.
* IPv6: (none)
* IPv4: 10.102.135.111
*   Trying 10.102.135.111:8080...
* Connected to envoy-egress (10.102.135.111) port 8080
> HEAD / HTTP/1.1
> Host: envoy-egress:8080
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 503 Service Unavailable
HTTP/1.1 503 Service Unavailable
< content-length: 211
content-length: 211
< content-type: text/plain
content-type: text/plain
< date: Tue, 19 Nov 2024 22:53:49 GMT
date: Tue, 19 Nov 2024 22:53:49 GMT
< server: envoy
server: envoy

< 
* Connection #0 to host envoy-egress left intact
* Uses proxy env variable HTTPS_PROXY == 'http://envoy-egress:8080'
* Hostname envoy-egress was found in DNS cache
*   Trying 10.102.135.111:8080...
* Connected to envoy-egress (10.102.135.111) port 8080
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Establish HTTP proxy tunnel to www.gmail.com:443
> CONNECT www.gmail.com:443 HTTP/1.1
> Host: www.gmail.com:443
> User-Agent: curl/8.5.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< date: Tue, 19 Nov 2024 22:53:49 GMT
date: Tue, 19 Nov 2024 22:53:49 GMT
< server: envoy
server: envoy
< 

* CONNECT phase completed
* CONNECT tunnel established, response 200
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL/3.0.13: error:0A00010B:SSL routines::wrong version number
* Closing connection
curl: (35) OpenSSL/3.0.13: error:0A00010B:SSL routines::wrong version number

Envoy Proxy Logs

[2024-11-19 22:59:42.224][20][trace][connection] [source/common/network/connection_impl.cc:614] [Tags: "ConnectionId":"93"] socket event: 2
[2024-11-19 22:59:42.224][20][trace][connection] [source/common/network/connection_impl.cc:737] [Tags: "ConnectionId":"93"] write ready
[2024-11-19 22:59:42.224][20][trace][connection] [source/common/tls/ssl_socket.cc:282] [Tags: "ConnectionId":"93"] ssl write returns: 517
[2024-11-19 22:59:42.252][1][debug][main] [source/server/server.cc:237] flushing stats
[2024-11-19 22:59:42.284][20][trace][connection] [source/common/network/connection_impl.cc:614] [Tags: "ConnectionId":"93"] socket event: 3
[2024-11-19 22:59:42.284][20][trace][connection] [source/common/network/connection_impl.cc:737] [Tags: "ConnectionId":"93"] write ready
[2024-11-19 22:59:42.284][20][trace][connection] [source/common/network/connection_impl.cc:654] [Tags: "ConnectionId":"93"] read ready. dispatch_buffered_data=0
[2024-11-19 22:59:42.284][20][trace][connection] [source/common/tls/ssl_socket.cc:92] [Tags: "ConnectionId":"93"] ssl read returns: 40
[2024-11-19 22:59:42.284][20][trace][connection] [source/common/tls/ssl_socket.cc:92] [Tags: "ConnectionId":"93"] ssl read returns: -1
[2024-11-19 22:59:42.284][20][trace][connection] [source/common/tls/ssl_socket.cc:132] [Tags: "ConnectionId":"93"] ssl error occurred while read: WANT_READ
[2024-11-19 22:59:42.284][20][trace][connection] [source/common/tls/ssl_socket.cc:168] [Tags: "ConnectionId":"93"] ssl read 40 bytes
[2024-11-19 22:59:42.284][20][trace][http] [source/common/http/filter_manager.cc:904] [Tags: "ConnectionId":"92","StreamId":"14141408323196708715"] commonEncodePrefix end_stream: false, isHalfCloseEnabled: false
[2024-11-19 22:59:42.284][20][trace][misc] [source/common/event/scaled_range_timer_manager_impl.cc:60] enableTimer called on 0x41cff4ea580 for 300000ms, min is 300000ms
[2024-11-19 22:59:42.284][20][trace][http] [source/common/http/filter_manager.cc:904] [Tags: "ConnectionId":"92","StreamId":"14141408323196708715"] commonEncodePrefix end_stream: false, isHalfCloseEnabled: false

Repro steps:

Include sample requests, environment, etc. All data and inputs

Configuration for the Envoy Proxy Pod

static_resources:
  listeners:
  - name: forward_proxy_listener
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 8080
    listener_filters:
    - name: "envoy.filters.listener.tls_inspector"
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: AUTO
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: dynamic_forward_proxy
              domains: "*"
              #domains: ["*"]
              routes:
              - match: { prefix: "/" }
                route:
                  cluster: dynamic_forward_proxy
              - match:
                  connect_matcher: {}
                route:
                  cluster: dynamic_forward_proxy
                  upgrade_configs:
                    - upgrade_type: CONNECT
                      connect_config: {}
          http_filters:
          - name: dynamic_forward_proxy
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
              dns_cache_config:
                name: dynamic_forward_proxy_cache
                dns_lookup_family: V4_ONLY
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: dynamic_forward_proxy
    connect_timeout: 5s
    # lb_policy: ROUND_ROBIN
    lb_policy: CLUSTER_PROVIDED
    cluster_type:
        name: envoy.clusters.dynamic_forward_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
          dns_cache_config:
            name: dynamic_forward_proxy_cache
            dns_lookup_family: V4_ONLY
    transport_socket:
      name: envoy.transport_sockets.tls    
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        common_tls_context:
          alpn_protocols: ["h2", "http/1.1"] # support both HTTP/2 and HTTP/1.1
          validation_context:
            trusted_ca:
              filename: /etc/ssl/certs/ca-certificates.crt
admin:
  access_log:
  - name: envoy.access_loggers.file
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
      path: "/dev/stdout"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901

Envoy Deployment

# envoy-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: envoy-egress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: envoy-egress
  template:
    metadata:
      labels:
        app: envoy-egress
    spec:
      containers:
      - name: envoy
        image: envoyproxy/envoy:v1.32.1
        volumeMounts:
        - name: envoy-config
          mountPath: /etc/envoy
          readOnly: true
        - name: envoy-cert
          mountPath: /etc/ssl/certs
          readOnly: true
        ports:
        - containerPort: 8080
        - containerPort: 9901
        args:
        - "--config-path"
        - "/etc/envoy/envoy.yaml"
      volumes:
      - name: envoy-config
        configMap:
          name: envoy-config
      - name: envoy-cert
        configMap:
          name: public-ca-cert

envoy-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: envoy-egress
  labels:
    app: envoy
spec:
  selector:
    app: envoy-egress # Ensure this matches the labels on the Envoy pod
  ports:
  - name: http
    protocol: TCP
    port: 8080
    targetPort: 8080  # Port exposed by the Envoy container
  - name: admin
    protocol: TCP
    port: 9901
    targetPort: 9901  # Optional: for the Envoy admin interface
  type: ClusterIP

tester-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: tester-pod
spec:
  containers:
  - name: tester
    image: ubuntu:latest
    #image: curlimages/curl:latest
    command: ["sleep", "3600"]
    env:
    - name: HTTP_PROXY
      value: "http://envoy-egress:8080"
    - name: HTTPS_PROXY
      value: "http://envoy-egress:8080"
soulxu commented 2 days ago

cc @ggreenway

ggreenway commented 1 day ago

Sometimes an error like this is due to trying to talk TLS to a plaintext service.

cc @alyssawilk because this deals with tunnels/CONNECT.

ziler-orca commented 1 day ago

@soulxu This is the output with the -v option

$ curl -v -k http://envoy-egress:8080 -I https://www.google.com
* Host envoy-egress:8080 was resolved.
* IPv6: (none)
* IPv4: 10.102.135.111
*   Trying 10.102.135.111:8080...
* Connected to envoy-egress (10.102.135.111) port 8080
* using HTTP/1.x
> HEAD / HTTP/1.1
> Host: envoy-egress:8080
> User-Agent: curl/8.11.0
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 503 Service Unavailable
HTTP/1.1 503 Service Unavailable
< content-length: 211
content-length: 211
< content-type: text/plain
content-type: text/plain
< date: Thu, 21 Nov 2024 00:29:58 GMT
date: Thu, 21 Nov 2024 00:29:58 GMT
< server: envoy
server: envoy
< 

* Connection #0 to host envoy-egress left intact
* Uses proxy env variable HTTPS_PROXY == 'http://envoy-egress:8080'
* Hostname envoy-egress was found in DNS cache
*   Trying 10.102.135.111:8080...
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Establish HTTP proxy tunnel to www.google.com:443
> CONNECT www.google.com:443 HTTP/1.1
> Host: www.google.com:443
> User-Agent: curl/8.11.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< date: Thu, 21 Nov 2024 00:29:58 GMT
date: Thu, 21 Nov 2024 00:29:58 GMT
< server: envoy
server: envoy
< 

* CONNECT phase completed
* CONNECT tunnel established, response 200
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLS connect error: error:0A00010B:SSL routines::wrong version number
* closing connection #1
curl: (35) TLS connect error: error:0A00010B:SSL routines::wrong version number
~ $ command terminated with exit code 137