SecurityPolicy is not limiting access to the IPs in clientCIDRs

[bleal-kitmanlabs]

Description: Using SecurityPolicy crd, envoy gateway should limit access only to the IPs in the clientCIDRs as suggested by the docs: https://gateway.envoyproxy.io/docs/tasks/security/restrict-ip-access/

My IP is not listed in the clientCIDRs. So the expected behavior is to receive 403 as suggested by the docs, however I'm receiving 302 (Redirecring me to the service).

Context: I want to allow only some people to be able to access a K8s Service (for testing propose I'm using prometheus endpoint).

Repro steps: I'm deploying it in the EKS and using envoyproxy to create the gateway. Under the hood, it creates an ALB in AWS.

Environment: Kubernetes v1.28 Envoy Gateway v1.0.1 (Check Update 1 - I found it was the issue, but now its not running the deployment)

Testing access:

The endpoint is reachable, however my IP is not in the list. So, I should be receiving 403 instead 302.

╰─ curl http://<ALB Address>/
<a href="/graph">Found</a>.

The Security Policy:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: prometheus-server-security-policy
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: prometheus-server
  authorization:
    defaultAction: Deny
    rules:
      - action: Allow
        principal:
          clientCIDRs:
            <People IPS>

I also configured the Client Traffic Policy:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: enable-client-ip-detection
spec:
  clientIPDetection:
    xForwardedFor:
      numTrustedHops: 1
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: gw

The HTTP Route:

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: prometheus-server
  labels:
    app.kubernetes.io/name: prometheus-server
spec:
  parentRefs:
    - name: gw
      sectionName: http-listener
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: prometheus-server
          port: 80
          weight: 1

      matches:
        - path:
            type: PathPrefix
            value: /

The Gateway:

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gw
spec:
  gatewayClassName: eg
  listeners:
    - name: http-listener
      port: 80
      protocol: HTTP

The Gateway Class:

---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
  namesspace: default
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller

What should I do? Am I missing some configuration?

---

Update 1

Based on the CRD of securitypolicies.gateway.envoyproxy.io, it does not support Authorization in the version 1.0: https://gateway.envoyproxy.io/v1.0/api/extension_types/#securitypolicyspec

Bumping to version v1.1.0 seems to be the solution, but the deploy fails:

╰─ k logs deploy/envoy-gateway
2024-07-29T13:52:02.202Z    INFO    admin   admin/server.go:37  starting admin server   {"address": "127.0.0.1:19000", "enablePprof": false}
2024-07-29T13:52:02.202Z    INFO    metrics metrics/register.go:165 initialized metrics pull endpoint   {"address": "0.0.0.0:19001", "endpoint": "/metrics"}
2024-07-29T13:52:02.203Z    INFO    metrics metrics/register.go:54  starting metrics server {"address": "0.0.0.0:19001"}
2024-07-29T13:52:02.205Z    INFO    provider    runner/runner.go:41 Using provider  {"runner": "provider", "type": "Kubernetes"}
2024-07-29T13:52:02.208Z    INFO    provider    kubernetes/controller.go:111    created gatewayapi controller   {"runner": "provider"}
Error: failed to create provider Kubernetes: failted to create gatewayapi controller: no matches for kind "GRPCRoute" in version "gateway.networking.k8s.io/v1"
Usage:
  envoy-gateway server [flags]

Aliases:
  server, serve

Flags:
  -c, --config-path string   The path to the configuration file.
  -h, --help                 help for server

failed to create provider Kubernetes: failted to create gatewayapi controller: no matches for kind "GRPCRoute" in version "gateway.networking.k8s.io/v1"

I'm not using GRPC, is it possible to disable it?

[arkodg]

@bleal-kitmanlabs you'll need to reinstall the v1.1.0 helm chart which has the updated CRDs, as this feature is not available in v1.0.x

helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.1.0 -n envoy-gateway-system --create-namespace

[bleal-kitmanlabs]

@arkodg Thank you for your answer!

I did it and it seems its working, however its denying everything. I added a few IPs and did the test just opening chrome on these instances and hitting the endpoint. Its always throwing 403. I'm not using the TrafficPolicy yet, Just the SecurityPolicy.

I'm sure the IPs are correct. What else could be the issue?

[zhaohuabing]

@arkodg Thank you for your answer!

I did it and it seems its working, however its denying everything. I added a few IPs and did the test just opening chrome on these instances and hitting the endpoint. Its always throwing 403. I'm not using the TrafficPolicy yet, Just the SecurityPolicy.

I'm sure the IPs are correct. What else could be the issue?

@bleal-kitmanlabs You may need to configure ClientTrafficPolicy and use x_forwarded_for header for testing.

Normally the Envoy won't be able to see the original client ip address. It can only see the LB address.

[bleal-kitmanlabs]

@zhaohuabing Thank you for answering. I added it and with the header in a curl command it works, but it does not work when accessing the service with the browser (chrome).

Imaging I want to protect the endpoint and let it be available only for some people who will access the service through the browser (chrome, firefox etc), what changes do I need to make? Its not clear for me yet.

[zhaohuabing]

@zhaohuabing Thank you for answering. I added it and with the header in a curl command it works, but it does not work when accessing the service with the browser (chrome).

Imaging I want to protect the endpoint and let it be available only for some people who will access the service through the browser (chrome, firefox etc), what changes do I need to make? Its not clear for me yet.

In a real world deployment, normally a network middleware will add x_forwarded_for or proxy protocol header to the request.

[bleal-kitmanlabs]

Hi Folks - Thanks for all the support you gave me.

At the end, I updated Envoy to use AWS Controller to spin up an NLB instead of the classic LB. Using that, I'm able to put in place security groups and reject connects from different IPs. Perhaps I misunderstood the SecurityPolicy and TrafficPolicy, and thought I could use them for blocking the connections in a network layer, however it seems it runs at the application level (that's why I see the docs suggesting using the header).

I'm closing the issue.