search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
25.05k stars 4.82k forks source link

Original Dst apparently not working for me #36804

Open AdrianSchlegel opened 3 weeks ago

AdrianSchlegel commented 3 weeks ago

Hello Dear Envoy Team,

I am trying to create a proxy which takes any traffic (on loopback addresses and http1.1) and converts it to http 2. So basically 127.0.0.10:7777 gets transferred to 127.0.0.10:7777 after a http2 conversion and so on.

I have solved the part of http2 converson with envoy proxy however I am having problems with the sending to the appropriate destination. I was using the original_dst cluster policy / listening filter with none working.

At some point I tried out the example configuration of this code from the envoy github repository found here: https://github.com/envoyproxy/envoy/blob/main/configs/original-dst-cluster/proxy_config.yaml

However this code doesnt work either. This code doesnt have the http2 conversion yet and is only supposed to dynamically find the destination. To test this I hosted a python http server with: python3 -m http.server 80.

Then I tried reaching it (first without proxy):

adrian@adrian-virtual-machine:~/Documents/5g-pentesting-project$ curl localhost 

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
</ul>
<hr>
</body>
</html>

As you see its working fine.

Now I use the code from the before mentioned link as config for my proxy (https://github.com/envoyproxy/envoy/blob/main/configs/original-dst-cluster/proxy_config.yaml) (please note i changed the listener address to 127.0.0.1:8082):

adrian@adrian-virtual-machine:~/Documents/5g-pentesting-project$ curl localhost -x 127.0.0.1:8082 -v

*   Trying 127.0.0.1:8082...
* Connected to (nil) (127.0.0.1) port 8082 (#0)
> GET http://localhost/ HTTP/1.1
> Host: localhost
> User-Agent: curl/7.81.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< content-length: 148
< content-type: text/plain
< date: Thu, 24 Oct 2024 11:19:28 GMT
< server: envoy
< x-envoy-upstream-service-time: 538
< 
* Connection #0 to host (nil) left intact
upstream connect error or disconnect/reset before headers. reset reason: local connection failure, transport failure reason: socket creation failure

With the envoy proxy showing this in stdout:

[2024-10-24 13:19:24.945][66651][info][main] [source/server/server.cc:990] starting main dispatch loop
[2024-10-24 13:19:29.328][66659][error][envoy_bug] [source/common/network/socket_interface_impl.cc:98] envoy bug failure: false. Details: socket(2) failed, got error: Too many open files
[2024-10-24 13:19:29.329][66659][error][envoy_bug] [./source/common/common/assert.h:38] stacktrace for envoy bug
[symbolize_elf.inc : 1072] RAW: /proc/self/task/66651/maps: errno=24
[2024-10-24 13:19:29.331][66659][error][envoy_bug] [./source/common/common/assert.h:45] #0 UNKNOWN [0x64f153577083]
[2024-10-24 13:19:29.331][66659][error][envoy_bug] [./source/common/common/assert.h:45] #1 UNKNOWN [0x64f1534577b5]
[2024-10-24 13:19:29.332][66659][error][envoy_bug] [./source/common/common/assert.h:45] #2 UNKNOWN [0x64f153452535]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #3 UNKNOWN [0x64f153446c8f]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #4 UNKNOWN [0x64f15343caa6]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #5 UNKNOWN [0x64f1530378ca]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #6 UNKNOWN [0x64f153037260]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #7 UNKNOWN [0x64f15301388b]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #8 UNKNOWN [0x64f1530136f4]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #9 UNKNOWN [0x64f1530149aa]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #10 UNKNOWN [0x64f153014d63]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #11 UNKNOWN [0x64f15302b412]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #12 UNKNOWN [0x64f15302e9f9]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #13 UNKNOWN [0x64f15301bd22]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #14 UNKNOWN [0x64f153273125]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #15 UNKNOWN [0x64f153295ea4]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [source/common/network/connection_impl.cc:89] envoy bug failure: false. Details: Client socket failure
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:38] stacktrace for envoy bug
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #0 UNKNOWN [0x64f1534526a5]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #1 UNKNOWN [0x64f15345255c]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #2 UNKNOWN [0x64f153446c8f]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #3 UNKNOWN [0x64f15343caa6]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #4 UNKNOWN [0x64f1530378ca]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #5 UNKNOWN [0x64f153037260]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #6 UNKNOWN [0x64f15301388b]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #7 UNKNOWN [0x64f1530136f4]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #8 UNKNOWN [0x64f1530149aa]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #9 UNKNOWN [0x64f153014d63]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #10 UNKNOWN [0x64f15302b412]
[2024-10-24 13:19:29.333][66659][error][envoy_bug] [./source/common/common/assert.h:45] #11 UNKNOWN [0x64f15302e9f9]
[2024-10-24 13:19:29.334][66659][error][envoy_bug] [./source/common/common/assert.h:45] #12 UNKNOWN [0x64f15301bd22]
[2024-10-24 13:19:29.334][66659][error][envoy_bug] [./source/common/common/assert.h:45] #13 UNKNOWN [0x64f153273125]
[2024-10-24 13:19:29.334][66659][error][envoy_bug] [./source/common/common/assert.h:45] #14 UNKNOWN [0x64f153295ea4]
[2024-10-24 13:19:29.334][66659][error][envoy_bug] [./source/common/common/assert.h:45] #15 UNKNOWN [0x64f15327c7bc]

I also know that there is no sort of issue with my envoy because when i am running my other envoy configuration for the http2 conversion (config code following). I get these results:

###endpoint that only can get http2 packets. With proof using --http2-prior-knowledge and withotu using it###
adrian@adrian-virtual-machine:~/Documents/5g-pentesting-project$ curl 'http://127.0.0.10:7777/nnrf-disc/v1/nf-instances?target-nf-type=AMF&requester-nf-type=SMF' --http2-prior-knowledge
{"validityPeriod":30,"nfInstances":[{"nfInstanceId":"452b4c46-9076-41ef-a30b-053a47f5ba84","nfType":"AMF","nfStatus":"REGISTERED","heartBeatTimer":10,"plmnList":[{"mcc":"999","mnc":"70"}],"ipv4Addresses":["127.0.0.5"],"allowedNfTypes":["SCP","SMF","AMF"],"priority":0,"capacity":100,"load":0,"amfInfo":{"amfSetId":"001","amfRegionId":"02","guamiList":[{"plmnId":{"mcc":"999","mnc":"70"},"amfId":"020040"}],"taiList":[{"plmnId":{"mcc":"999","mnc":"70"},"tac":"000001"}]},"nfServices":[{"serviceInstanceId":"452d0450-9076-41ef-a30b-053a47f5ba84","serviceName":"namf-comm","versions":[{"apiVersionInUri":"v1","apiFullVersion":"1.0.0"}],"scheme":"http","nfServiceStatus":"REGISTERED","ipEndPoints":[{"ipv4Address":"127.0.0.5","port":7777}],"allowedNfTypes":["SMF","AMF"],"priority":0,"capacity":100,"load":0}],"nfProfileChangesSupportInd":true}]}

adrian@adrian-virtual-machine:~/Documents/5g-pentesting-project$ curl 'http://127.0.0.10:7777/nnrf-disc/v1/nf-instances?target-nf-type=AMF&requester-nf-type=SMF'
curl: (1) Received HTTP/0.9 when not allowed

###Request works using my proxy config###
adrian@adrian-virtual-machine:~/Documents/5g-pentesting-project$ curl 'http://127.0.0.10:7777/nnrf-disc/v1/nf-instances?target-nf-type=AMF&requester-nf-type=SMF' -x http://127.0.0.1:8082
{"validityPeriod":30,"nfInstances":[{"nfInstanceId":"452b4c46-9076-41ef-a30b-053a47f5ba84","nfType":"AMF","nfStatus":"REGISTERED","heartBeatTimer":10,"plmnList":[{"mcc":"999","mnc":"70"}],"ipv4Addresses":["127.0.0.5"],"allowedNfTypes":["SCP","SMF","AMF"],"priority":0,"capacity":100,"load":0,"amfInfo":{"amfSetId":"001","amfRegionId":"02","guamiList":[{"plmnId":{"mcc":"999","mnc":"70"},"amfId":"020040"}],"taiList":[{"plmnId":{"mcc":"999","mnc":"70"},"tac":"000001"}]},"nfServices":[{"serviceInstanceId":"452d0450-9076-41ef-a30b-053a47f5ba84","serviceName":"namf-comm","versions":[{"apiVersionInUri":"v1","apiFullVersion":"1.0.0"}],"scheme":"http","nfServiceStatus":"REGISTERED","ipEndPoints":[{"ipv4Address":"127.0.0.5","port":7777}],"allowedNfTypes":["SMF","AMF"],"priority":0,"capacity":100,"load":0}],"nfProfileChangesSupportInd":true}]}adrian@adrian-virtual-machine:~/Documents/5g-pentesting-project$

MY PROXY CONFIG:

admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 15000

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 127.0.0.1
        port_value: 8082
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: service_http2

  clusters:
  - name: service_http2
    connect_timeout: 0.25s
    type: strict_dns
    load_assignment:
      cluster_name: service_http2
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.10  # Upstream service IP
                port_value: 7777        # Upstream service port
    http2_protocol_options: {}  # Enable HTTP/2 for upstream

I would very much appreciate it if someone could tell me what I am doing wrong in the first part of the issue. I am literally using the example in the github repo and it is not working for me with the error codes which I have shown. I wish to have a proxy which just forwards the requests further to their destination.

I am also open to other solutions to this without the original_dst if they work too.

Thank you very much for the help :)

wbpcode commented 3 weeks ago

original dst cluster will try to get SO_ORIGINAL_DST to get the original destination. Or you can also let the cluster to get original destination from the header. See https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/original_dst