skyerus
commented
1 year ago POC
I have put together a POC using kubernetes gateway API (KGA) as means to ingest external traffic (intended as successor to an ingress).
The deployment pattern this POC is catered to:
- Deployment of an application exposed by a Service
- Gateway exposes the app's Service via a HTTPRoute and listener (with hostname)
Assuming that the above is true (application is readily deployed and ready to receive external traffic), one can create a ClientSideConfiguration CR (naming TBD).
The creation of a ClientSideConfiguration CR triggers its reconciliation loop which does the following:
- Creates a deployment of flagd
- Creates a service exposing the flagd deployment
- Appends a listener to the Gateway resource (name, port defined by the CR)
- Creates a HTTPRoute exposing the flagd deployment service to the gateway (hostname defined by the CR)
The end result is two exposed applications: the client application and flagd. Both are exposed by the Gateway, which routes the request to the appropriate service based on the matching of the Host header (this is checked against the hostname defined in the Gateway's listeners). The following diagram depicts this.
Note: To get this to work locally I had to add hostnames to my hosts file to route them to localhost. Without this the browser doesn't send the appropriate Host header to the Gateway, preventing it from being able to route the request.
How to run the POC
Prerequisites:
- Kind
- Checkout OFO at the branch in the linked PR
- istioctl (this can be substituted for any other KGA implementor but make sure to replace references to istio with the substitute)
# build OFO
IMG=ghcr.io/open-feature/open-feature-operator:local ARCH=arm64 make docker-build
# create kind cluster
kind create cluster --config ./test/e2e/kind-cluster.yml
# load OFO into cluster
kind load docker-image ghcr.io/open-feature/open-feature-operator:local
# pull ofo-client-app
docker pull ghcr.io/skyerus/ofo-client-app:latest
# load ofo-client-app into cluster
kind load docker-image ghcr.io/skyerus/ofo-client-app:latest
# deploy OFO
IMG=ghcr.io/open-feature/open-feature-operator:local make deploy-operator
# install KGA CRDs
kubectl get crd gateways.gateway.networking.k8s.io &> /dev/null || \
{ kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v0.6.1" | kubectl apply -f -; }
# install istio gateway
istioctl install --set profile=minimal -y
# deploy resources
kubectl apply -f config/samples/dev.yaml
# add hostnames to hosts file (location of hosts file is macOS specific in this example)
sudo sh -c 'echo "127.0.0.1 ofoclientapp.com" >> /private/etc/hosts'
sudo sh -c 'echo "127.0.0.1 flagd.ofoclientapp.com" >> /private/etc/hosts'
# port forward 8080 to the gateway pod
kubectl port-forward $(kubectl get pods --selector=istio.io/gateway-name=clientsidegateway -o \
jsonpath='{.items[*].metadata.name}') 8080:80Go to http://ofoclientapp.com:8080/ in your browser and click the Update color button to verify that the traffic is routed to flagd correctly.
Questions/Concerns
- KGA and its implementors are currently in beta.
- Can KGA and Ingress coexist or should we enforce anyone who wishes to use OFO to deploy flagd to use KGA?
KGA and ingress are different means of achieving the same result (routing inbound traffic to the correct service). Using both would mean one sits behind the other with duplicated rules, this seems wrong. IMO, if we want to support ingress we could introduce a
networkTypefield to the CRD and split code paths for KGA/ingress. The ingress path would mutate the existing ingress resource in similar fashion to what has been done for KGA - Do we want to support creating a Gateway if it doesn't already exist or just return an error? Initially, for simplicity, return an error if the resource doesn't exist.
- Can local development be improved to not require adding hostnames to hosts file? This drawback is only impactful to end-to-end development, which this configuration is not suitable nor intended for. There exists other means of making requests which allow headers (Host header) to be set directly (e.g. curl). These means are more appropriate for assessing the validity of the configured networking (no hosts file meddling required).
- Does KGA support bi-directional HTTP2 connections? KGA supports GRPC routes (experimentally for now). This implies support of bi-directional HTTP2 connections.
- Can scalability be demonstrated? Kubernetes typical notions of scaling (e.g. Horizontal Pod Autoscaling) are applicable.
Further Steps
- [x] Address the above questions/concerns
- [ ] Decide on naming of the
ClientSideConfigurationCR - [ ] Decide whether ingress support is required
- [ ] Move shared code into their own packages and import to existing invokers (a lot of duplicated code for POC)
- [ ] Flesh out POC to include other sync provider deployments (currently only kubernetes provider is available)
AlexsJones
toddbaert
beeme1mr
With client spec changes and SDKs on the way, we need to decide what sort of support we want for the client/web in OFO (if any). We already have the required bulk-resolution functionality in flagd, as well as CORs options to support local development
@james-milligan already did some research here with a PoC deployment, I believe. That may feed into this discussion.
If we were to support this, we'd want enhancements to the operator that could facilitate the following:
Non functionally, we should consider: