docs: choosing and configuring envoy filters

[dettanym]

Possible options for PII detection, in order of priority are:

• configure a TAP filter
• a Go plugin (configured via HTTP Golang filter), or
• roll our own Envoy filter.

New experimental filter: TAP filter: Intended to intercept HTTP traffic (requests or responses) that match certain criteria. Matched traffic will be output to a "sink" that can be an API endpoint or a file.

HTTP Golang filter: Can write a custom plugin to intervene during an HTTP request / response lifecycle in Go. (The plugin can actively modify HTTP request and responses afaict) Sample sandbox

Rolling our own Envoy filter: Plugging in a new Envoy filter that hooks into the HTTP request / response lifecycle, in C++. (The http-filter example hooks in after a 'decode filter' and adds in a new header.)

Other sandboxes:

• Jaeger distributed tracing with Envoy's Zipkin client or "native" tracing with Jaeger client CPP (outdated repo)
• Ext Authz filter with OPA
• OpenTelemetry Collector

[dettanym]

Configuring envoy filters.

Istio provides an EnvoyFilter CRD to customize the Envoy sidecar image. The CRD is defined in terms of what listeners / filters you want to insert where in the lifecycle of inbound / outbound requests to the sidecar. The specification for the Envoy filter itself is placed under the patch.value key.

The [OPA Envoy plugin] includes two examples:

• a native example with a sample app, a single Envoy sidecar container talking to an OPA container.
• an Istio example

In the latter example config file, they configure an EnvoyFilter resource. Under the patch.value key, we can find Envoy keys for the ext_authz filter. (Additional examples of this ext_authz filter are shown in the Envoy docs.)

Istio's support for Distributed Tracing --- including Jaeger, OpenTelmetry Collector.

[dettanym]

Go plugin notes:

The Go Plugin follows this StreamFilter interface, which is included via an import in the sandbox filter.go file. The plugin implements sets of encode and decode methods. It decodes requests, operates on them, whereas it modifies responses, before encoding them.

It appears that this StreamFilter interface doesn't distinguish between requests made by the service or made to the service. The sandbox example attaches the Envoy proxy to a helloworld service, which plausibly isn't making requests to any other services. So all requests that the service proxy gets, would be made by a client, and would fall into the second category.

I thought that the notion of 'upstream' and 'downstream' (respectively) in Envoy captures this distinction, looking at the docs. The StreamFilter API includes 'Downstream' and 'Upstream' addresses in the 'StreamInfo' struct, but it is unclear whether these addresses can be used to make this distinction. Relatedly, the sandbox example seems to consider the helloworld service as a gateway service and uses a path suffix to indicate requests that are to be routed to (?) an upstream service, whose responses should be modified (or encoded) by the Go plugin.

I noticed that Istio's EnvoyFilter API allows attaching plugins to 'outbound' or 'inbound' sidecars, which matches its model of egress and ingress listeners for its sidecars.

So we may need to separate out logic for requests made by the service to be applied by Istio's EnvoyFilter to outbound sidecars (listeners) and logic for requests made to the service should be attached to inbound sidecars. Common logic can be applied to "all" sidecars, but we should keep the ordering in mind.

[dettanym]

Objects created with the EnvoyFilter CRD don't seem to be having any impact. I tried debugging the sidecars and the Istio control plane Istiod service; no configurations regarding the EnvoyFilters were seen anywhere. E.g. under /etc/istio/proxy/envoy-rev.json in the sidecar or in the sidecar logs or using any debugging commands for the proxy. Some EnvoyFilter sample objects.

Edit: Made OPA EnvoyFilter work with Istio. (Ran into namespacing issues.)

[dettanym]

Progress so far on Lua filter:

• Made a Lua filter that calls the Presidio service for PII detection.
• Made two different Lua filters, one for requests to the pod ("inbound sidecar"), and another one for requests made by the pod ("outbound sidecar"). Tested the inbound filter on the reviews service of Bookinfo, and the outbound filter on its productpage service, as it functions as an API gateway service, directing requests to the reviews, details pages etc.

Go Filter: Ran the Envoy Golang plugin. Github link

Next steps:

• Confirm that the EnvoyFilter code is run before OPA.

• For the 'outbound sidecar' filter, test against a pod that makes external requests.

• Change inputs from being static to passing the request body. Add PII identified in Presidio's response as an HTTP header for OPA.

• Tweak the Go filter example: Create a filter for each direction of the request; do the steps in the bullet point above. Test it against the reviews and productpage services of the bookinfo example (can run the Bookinfo example via Docker).

• Problem with the Lua filter: First, it doesn't have a lot of tooling support as compared to Golang. Second, the Lua filter would not be packaged along with our application; it would just sit inlined within the Istio EnvoyFilter configuration. Instead, to load it as a separate file, we will need to load it into each sidecar.

If we were to integrate the Go filter plugin into Prose, then a Github workflow, which should be similar to the build script in the Golang plugin example, would generate the lib.so file and plug it in at the right location in the prose-k8s-home-ops repo. Integrating this plugin into Istio requires each sidecar, to run the generated lib.so file, and so we need to load it first into the sidecars.

To address the problem of loading either the Lua filter text file or the Golang binary, we could use K8s volumes and configmaps. We can use K8s volumes, which are similar to docker volumes, to load data from various sources (NFS, host's local FS, configmaps). In the deployment setting, the DevOps engineer can just clone our prose-k8s-home-ops repo onto each K8s node, include GitRepos for their source code, and run the task cluster:install command, which will create necessary K8s objects from the configuration stored in the repo. So we could have a K8s volume of the type that loads from the host's local FS, i.e. the K8s node's local FS, which will contain this repo and the Go binary / Lua filter. We could use configmaps too.

Another option is to use Istio resource annotations. Specifically, the sidecar.istio.io/userVolume annotation can be used to specify a particular user volume to add to the sidecar. However, the use-case satisfied by Istio resource annotations is for certain selected pods to have this annotation. So we'd need to individually annotate each pod, in the namespace with the istio enabled tag. An example blogpost doing pod annotations to load the user volume. Can use kube-annotate to automatically annotate pods.

[dettanym]

EnvoyFilter functions tasks:

• [x] Confirm that the EnvoyFilter code is run before OPA.
• [x] Change inputs from being static to passing the request body.
• [x] Add PII identified in Presidio's response as an HTTP header for OPA.

[dettanym]

X-Envoy-Peer-Metadata header may not be available on outgoing requests to mesh external domains: see https://github.com/istio/istio/issues/17635