THREESCALE-10582 fix integration of upstream connection policy with camel policy

[eguzki]

What

Fixes: https://issues.redhat.com/browse/THREESCALE-10582

Upstream timeouts don't work with Camel Service. Actually upstream connection policy is not working when https_proxy is being used (either with env vars or policy).

This PR fixes the integration of upstream connection with any use case for https_proxy.

It was considered adding connection options to the proxy policy and camel policy. Mainly because "Upstream connection" policy is referring to "upstream", which can be confusing when a proxy is being used, as the connection to upstream backend is no longer made by APIcast. Instead, APIcast creates a connection to the proxy. So the Upstream connection opts should, ideally, only apply to connections to backend "upstream".

We decided to apply upstream connection policy to any "upstream" connection the APIcast make initiate, either a proxy or a actual upstream backend. That way, implementation wise it is easier. No need to add extended connection parameters to the proxy policies. Furthermore, if users are using Upstream connection policy together with http_proxy, the configuration still applies. With a new connection parameters in the proxy policies, this last use case would be broken. Additionally, if connection opts are added to the policies as optional params, we would need to add new env vars as well for the use case where proxies are configured via env vars. Too complex just to stick "upstream" concept to the actual backend (api_backend in the service configuration). Instead, upstram connection policy applies to any "upstream" connection APIcast does, regardless of being it a proxy or backend upstream.

Verification Steps

Upstream connection integration with https_proxy camel proxy

• Build docker image from this git branch

make runtime-image IMAGE_NAME=apicast-test

• Camel proxy dev environment setup

cd dev-environments/camel-proxy
make certs

• Testing https_proxy use case: APIcast --> camel proxy --> upstream (TLS)

This env uses as api_backend the real env https://echo-api.3scale.net:443. My roundtrip latency is ~400ms.

❯ curl -i -w "tcp:%{time_total}\n" https://echo-api.3scale.net:443 2>/dev/null
HTTP/1.1 200 OK
content-type: application/json
x-3scale-echo-api: echo-api/1.0.3
vary: Origin
x-content-type-options: nosniff
content-length: 524
x-envoy-upstream-service-time: 0
date: Mon, 05 Feb 2024 14:50:56 GMT
server: envoy

{
  "method": "GET",
  "path": "/",
  "args": "",
  "body": "",
  "headers": {
    "HTTP_VERSION": "HTTP/1.1",
    "HTTP_HOST": "echo-api.3scale.net",
    "HTTP_USER_AGENT": "curl/7.81.0",
    "HTTP_ACCEPT": "*/*",
    "HTTP_X_FORWARDED_FOR": "81.61.128.254",
    "HTTP_X_FORWARDED_PROTO": "https",
    "HTTP_X_ENVOY_EXTERNAL_ADDRESS": "81.61.128.254",
    "HTTP_X_REQUEST_ID": "a78c1edb-b0bf-42a2-8c44-e1ae9d4dba49",
    "HTTP_X_ENVOY_EXPECTED_RQ_TIMEOUT_MS": "15000"
  },
  "uuid": "b48cdf4d-1419-4aaf-871e-10cd9c67f68e"
}tcp:0.398028

Let's start with timeouts set to 1 sec and the request should be accepted.

patch <<EOF
diff --git a/dev-environments/camel-proxy/apicast-config.json b/dev-environments/camel-proxy/apicast-config.json
index 91201afa..8f92f029 100644
--- a/dev-environments/camel-proxy/apicast-config.json
+++ b/dev-environments/camel-proxy/apicast-config.json
@@ -44,6 +44,14 @@
           "host": "backend"
         },
         "policy_chain": [
+          {
+            "name": "apicast.policy.upstream_connection",
+            "configuration": {
+              "connect_timeout": 1,
+              "send_timeout": 1,
+              "read_timeout": 1
+            }
+          },
           {
             "name": "apicast.policy.camel",
             "configuration": {
EOF

Run environment

make gateway IMAGE_NAME=apicast-test

The request should be accepted (200 OK), as the connection timeouts should not be exceeded.

curl --resolve https-proxy.example.com:8080:127.0.0.1 -v "http://https-proxy.example.com:8080/?user_key=123"

Now, let's lower the timeouts threshold to something like 100ms that should be exceeded because the upstream is far away.

Stop the gateway

CTRL-C

Restore apicast-config.json file

git checkout apicast-config.json

Apply 100ms timeouts

patch <<EOF
diff --git a/dev-environments/camel-proxy/apicast-config.json b/dev-environments/camel-proxy/apicast-config.json
index 91201afa..8f92f029 100644
--- a/dev-environments/camel-proxy/apicast-config.json
+++ b/dev-environments/camel-proxy/apicast-config.json
@@ -44,6 +44,14 @@
           "host": "backend"
         },
         "policy_chain": [
+          {
+            "name": "apicast.policy.upstream_connection",
+            "configuration": {
+              "connect_timeout": 0.1,
+              "send_timeout": 0.1,
+              "read_timeout": 0.1
+            }
+          },
           {
             "name": "apicast.policy.camel",
             "configuration": {
EOF

Run environment

make gateway IMAGE_NAME=apicast-test

The request should fail (502 Bad Gateway), as the connection timeouts should be exceeded.

curl --resolve https-proxy.example.com:8080:127.0.0.1 -v "http://https-proxy.example.com:8080/?user_key=123"

The logs should show the following line

[error] 19#19: *2 lua tcp socket read timed out

Clean the env before starting next step

make clean

Upstream connection integration with https_proxy with proxy policy (tinyproxy)

• Build docker image from this git branch

cd ${APICAST_PROJECT} && git checkout THREESCALE-10582-upstream-policy-with-camel-policy

make runtime-image IMAGE_NAME=apicast-test

• https_proxy dev environment setup

cd dev-environments/https-proxy-upstream-tlsv1.3
make certs

• Testing https_proxy use case: APIcast --> tiny proxy --> upstream (TLS)

Timeouts set to 0.1 sec

patch <<EOF
diff --git a/dev-environments/https-proxy-upstream-tlsv1.3/apicast-config.json b/dev-environments/https-proxy-upstream-tlsv1.3/apicast-config.json
index 5227c5aa..34a2ed23 100644
--- a/dev-environments/https-proxy-upstream-tlsv1.3/apicast-config.json
+++ b/dev-environments/https-proxy-upstream-tlsv1.3/apicast-config.json
@@ -11,6 +11,14 @@
           "host": "backend"
         },
         "policy_chain": [
+          {
+            "name": "apicast.policy.upstream_connection",
+            "configuration": {
+              "connect_timeout": 0.1,
+              "send_timeout": 0.1,
+              "read_timeout": 0.1
+            }
+          },
           {
             "name": "apicast.policy.http_proxy",
             "configuration": {
EOF

Run environment

make gateway IMAGE_NAME=apicast-test

The request should be accepted (200 OK), as the connection timeouts should not be exceeded.

curl --resolve get.example.com:8080:127.0.0.1 -i "http://get.example.com:8080/?user_key=123" 

Now, let's simulate some network latency using docker containers with Traffic control. We will add 200ms of latency to the container running socat between the proxy and the backend upstream. It is called example.com service.

Stop the gateway

CTRL-C

We are going to modify network-related stuff, so the NET_ADMIN capability is needed.

patch <<EOF
diff --git a/dev-environments/https-proxy-upstream-tlsv1.3/docker-compose.yml b/dev-environments/https-proxy-upstream-tlsv1.3/docker-compose.yml
index 9fa735f7..0f802e8b 100644
--- a/dev-environments/https-proxy-upstream-tlsv1.3/docker-compose.yml
+++ b/dev-environments/https-proxy-upstream-tlsv1.3/docker-compose.yml
@@ -39,6 +39,8 @@ services:
     restart: unless-stopped
     volumes:
       - ./cert/example.com.pem:/etc/pki/example.com.pem
+    cap_add:
+      - NET_ADMIN
   two.upstream:
     image: kennethreitz/httpbin
     expose:
EOF

Run environment with the new config

make gateway IMAGE_NAME=apicast-test

install the tc (traffic control) command

docker compose exec example.com apk add iproute2-tc

Add 200ms latency to the outbound traffic of example.com service.

docker compose exec example.com tc qdisc add dev eth0 root netem delay 200ms

The request should be rejected (503 Service Temporarily Unavailable), as the connection timeouts should not be exceeded.

curl --resolve get.example.com:8080:127.0.0.1 -i "http://get.example.com:8080/?user_key=123" 

The logs should show the following line

[error] 19#19: *2 lua tcp socket read timed out

[eguzki]

all comments addressed

[tkan145]

LGTM!