EnvoyExtensionPolicy with Wasm on Gateway causes issues when routing inbound requests

[rsaelim]

Description: I use HTTPRoute and SecurityPolicies to expose api routes. I applied an EnvoyExtensionPolicy with the goal to count requests base on request headers. After I applied it, all requests with matching HTTPRoutes with an open security policy resulted in 503s, and all requests with matching HTTPRoutes with an authz security policy (using extAuthz gRPC) resulted in 403s.

Repro steps:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: envoy-gateway-system
spec:
  telemetry:
    accessLog:
      settings:
        - format:
            type: JSON
            json:
              start_time: "%START_TIME(%Y-%m-%dT%H:%M:%S%z)%"
              authority: "%REQ(:AUTHORITY)%"
              bytes_received: "%BYTES_RECEIVED%"
              bytes_sent: "%BYTES_SENT%"
              x_client_id: "%REQ(X-Client-Id)%"
              duration: "%DURATION%"
              method: "%REQ(:METHOD)%"
              path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
              status: "%RESPONSE_CODE%"
              upstream_host: "%UPSTREAM_HOST%"
              user_agent: "%REQ(USER-AGENT)%"
              x_forwarded_for: "%REQ(X-FORWARDED-FOR)%"
              x_request_id: "%REQ(X-REQUEST-ID)%"
              cluster: "%UPSTREAM_CLUSTER%"
          sinks:
            - type: File
              file:
                path: /dev/stdout
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend-open
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /openRoute
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: open-policy
  namespace: envoy-gateway-system
spec:
  cors:
    allowHeaders:
    - x-api-key
    - authorization
    - content-type
    - user-agent
    allowMethods:
    - GET
    - POST
    - PUT
    - PATCH
    - DELETE
    - OPTIONS
    allowOrigins:
    - '*'
    maxAge: 5m0s
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend-open
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: wasm-extension
  namespace: envoy-gateway-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  wasm:
  - name: wasm-filter
    code:
      type: HTTP
      http:
        url: http://hostname:8080/apiCount.wasm

Environment: Envoy gateway v1.1

Logs: envoy-envoy-gateway-system-eg-5391c79d-657b89f787-6ppmz envoy {"path":"/openRoute","bytes_received":0,"x_user_id":null,"bytes_sent":0,"authority":"www.example.com","x_client_id":null,"upstream_host":null,"cluster":null,"duration":0,"method":"GET","x_forwarded_for":"xx.xxx.xxx.xxx","start_time":"2024-09-05T15:22:26+0000","user_agent":"axios/1.5.1","status":503,"x_request_id":"d8c5ccb9-742c-426b-a6fb-32a23bdef6a0"}

^Something odd with the log is that a lot of fields that come from request headers are unexpectedly null, so its possible this is not an EG issue but rather the wasm is messing up the request, though my wasm binary is compiled from something very similar if not nearly identical to this: https://github.com/tetratelabs/proxy-wasm-go-sdk/blob/main/examples/metrics/main.go.

[arkodg]

• do you see any error (Accepted: False) in the status of the policies ?
• do requests work w/o the WASM module ?

[rsaelim]

• do you see any error (Accepted: False) in the status of the policies ?
• do requests work w/o the WASM module ?

No, Accepted: True in security policies and envoyextension policy. Yes, the requests are routed properly without the wasm module added via the envoyextension policy.

[arkodg]

closing this issue for now, @rsaelim feel free to reopen if the issue seems to be related to Envoy Gateway and not the WASM logic

[rsaelim]

@arkodg Can you reopen the issue (closed by author so I don't have an option to reopen). I don't see why EG's routing decisions / HTTPRoute resource would be affected by the WASM logic (all I'm doing is reading request header value, counting, and the host continues processing).

[arkodg]

@rsaelim does this WASM example work for you https://gateway.envoyproxy.io/docs/tasks/extensibility/wasm/ ?

[rsaelim]

@rsaelim does this WASM example work for you https://gateway.envoyproxy.io/docs/tasks/extensibility/wasm/ ?

It does work...Thanks! Is the source code of this test wasm publically available?

[arkodg]

cc @zhaohuabing

[zhaohuabing]

@rsaelim Your can find source code here: https://github.com/envoyproxy/examples/blob/main/wasm-cc/envoy_filter_http_wasm_example.cc

[rsaelim]

Thanks @zhaohuabing @arkodg. Turns out this might be a documentation issue. The sha256 field in HTTPWasmCodeSource | EnvoyExtensionPolicy is required. If omitted, will result in the bug in this issue.

This field is required in envoyproxy doc.

[zhaohuabing]

@rsaelim this should be fixed in https://github.com/envoyproxy/gateway/pull/4547