External Authorization Server with Istio

Tutorial to setup an external authorization server for istio. In this setup, the ingresss-gateway will first send the inbound request headers to another istio service which check the header values submitted by the remote user/client. If the header values passes some criteria, the external authorization server will instruct the authorization server to proceed with the request upstream.

The check criteria can be anything (kerberos ticket, custom JWT) but in this example, it is the simple presence of the header value match as defined in configuration.

In this setup, it is important to ensure the authorization server is always (and exclusively) called by the ingress gateway and that the upstream services must accept the custom JWT token issued by the authorization server.

To that end, this configuration sets up mTLS, RBAC and ORIGIN authentication. RBAC ensures service->service traffic flows between the gateway, authorization server and the upstream systems. Each upstream service will only allow ORIGIN JWT tokens issued by the authorization server.

images/istio-extauthz.svg

This tutorial is a continuation of the istio helloworld application.

12/11/24: Use minikube 11/25/21: Updated for example to NOT use an actual service account. Instead, use the istio built gen-jwtpy in JWT issuers

3/20/21: Updated for istio 1.9: Integrate external authorization system (e.g. OPA, oauth2-proxy, etc.) with Istio using AuthorizationPolicy. Part of the upgrade is to use the v3 API (go-control-plane/envoy/config/core/v3, go-control-plane/envoy/service/auth/v3)

References

Setup

The following setup uses a minikube and a convenient JWK endpoint provided by an Istio sample JWT authentication tutorial.

First install istio

minikube start --driver=kvm2  --cpus=4 --kubernetes-version=v1.28 --host-only-cidr 192.168.39.1/24
minikube addons enable metallb

## in a new window
minikube dashboard

## get the IP, for me it was the following
$ minikube ip
192.168.39.1

## setup a loadbalancer metallb, enter the ip range shown below
minikube addons configure metallb
# -- Enter Load Balancer Start IP: 192.168.39.104
# -- Enter Load Balancer End IP: 192.168.39.110

## download and install istio
export ISTIO_VERSION=1.24.0 
export ISTIO_VERSION_MINOR=1.24

wget -P /tmp/ https://github.com/istio/istio/releases/download/$ISTIO_VERSION/istio-$ISTIO_VERSION-linux-amd64.tar.gz
tar xvf /tmp/istio-$ISTIO_VERSION-linux-amd64.tar.gz -C /tmp/
rm /tmp/istio-$ISTIO_VERSION-linux-amd64.tar.gz

export PATH=/tmp/istio-$ISTIO_VERSION/bin:$PATH

istioctl install --set profile=demo \
 --set meshConfig.enableAutoMtls=true  \
 --set values.gateways.istio-ingressgateway.runAsRoot=true \
 --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY \
 --set meshConfig.defaultConfig.gatewayTopology.forwardClientCertDetails=SANITIZE_SET

Build and push images

You can use the following prebuilt containers for this tutorial if you want to.

If you would rather build and stage your own, the Dockerfile for each container is provided in this repo.

The images we will use here has the following endpoints enabled:

salrashid123/svc: Frontend service
- /version: Displays a static "version" number for the image. If using salrashid123/svc:1 then the version is 1. If using salrashid123/svc:2 the version is 2
- /backend: Makes an HTTP Ret call to the backend service's /backend and /headerz endpoints.
salrashid123/besvc: Backend Service
- /headerz: Displays the http headers
- /backend: Displays the pod name
salrashid123/ext-authz-server: External Authorization gRPC Server
- gRPC Authorization server running in namespace authz-ns as service authz
- Authorization server reads an environment variable that lists the set of authorized (eg authzallowedusers: "alice,bob") This server will read the "Authorization: Bearer " header value from the incoming request to determine the username

To build your own, create a public dockerhub images with the names specified below:

Build External Authorization Server (you can ofcourse use your own dockerhub repo!)

cd authz_server/
docker build -t salrashid123/ext-authz-server .
docker push salrashid123/ext-authz-server

Build Frontend

cd frontend 
docker build  --build-arg VER=1 -t salrashid123/svc:1 .
docker build  --build-arg VER=2 -t salrashid123/svc:2 .
docker push salrashid123/svc:1
docker push salrashid123/svc:2

Build Backend


cd backend
docker build  --build-arg VER=1 -t salrashid123/besvc:1 .
docker build  --build-arg VER=1 -t salrashid123/besvc:2 .

docker push salrashid123/besvc:1 docker push salrashid123/besvc:2


### Verify istio is installed

```bash
kubectl label namespace default istio-injection=enabled
kubectl get no,po,rc,svc,ing,deployment -n istio-system

Deploy Istio Gateway and services

kubectl apply -f istio-lb-certs.yaml
sleep 10
## create the ingress gateway
kubectl apply -f istio-ingress-gateway.yaml

## kill and restart the ingress pod since the LB cert's may not have been loaded
INGRESS_POD_NAME=$(kubectl get po -n istio-system | grep ingressgateway\- | awk '{print$1}'); echo ${INGRESS_POD_NAME};
kubectl delete po/$INGRESS_POD_NAME -n istio-system

kubectl apply -f istio-app-config.yaml

export GATEWAY_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $GATEWAY_IP

Debugging ingress-gateway

INGRESS_POD_NAME=$(kubectl get po -n istio-system | grep ingressgateway\- | awk '{print$1}'); echo ${INGRESS_POD_NAME};
kubectl exec --namespace=istio-system $INGRESS_POD_NAME -c istio-proxy -- curl -X POST  http://localhost:15000/logging\?level\=debug
kubectl logs $INGRESS_POD_NAME -n istio-system

Deploy application

Deploy the baseline application without the external authorization server

$ kubectl apply -f app-deployment.yaml

$ kubectl get po,svc
NAME                         READY   STATUS    RESTARTS   AGE
pod/be-v1-8589f84d6-ll82f    2/2     Running   0          74s
pod/be-v2-6ff75fccd8-chj92   2/2     Running   0          74s
pod/svc1-bdb4d7c59-fgfk5     2/2     Running   0          74s
pod/svc2-7f65cc98f-hxcw9     2/2     Running   0          74s

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/be           ClusterIP   10.116.6.105   <none>        8080/TCP   74s
service/kubernetes   ClusterIP   10.116.0.1     <none>        443/TCP    5m26s
service/svc1         ClusterIP   10.116.9.247   <none>        8080/TCP   75s
service/svc2         ClusterIP   10.116.12.54   <none>        8080/TCP   74s

Send Traffic

Verify traffic for the frontend and backend services. (we're using jq to help parse the response)

# Access the frontend for svc1,svc2
curl -s --cacert certs/CA_crt.pem -w " %{http_code}\n" --resolve svc1.example.com:443:$GATEWAY_IP  https://svc1.example.com/version
curl -s --cacert certs/CA_crt.pem -w " %{http_code}\n" --resolve svc2.example.com:443:$GATEWAY_IP  https://svc2.example.com/version

# Access the backend through svc1,svc2
curl -s --cacert certs/CA_crt.pem -w " %{http_code}\n" --resolve svc1.example.com:443:$GATEWAY_IP  https://svc1.example.com/backend | jq '.'
curl -s --cacert certs/CA_crt.pem -w " %{http_code}\n" --resolve svc2.example.com:443:$GATEWAY_IP  https://svc2.example.com/backend | jq '.'

If you would rather run this in a loop:

 for i in {1..1000}; do curl -s -w " %{http_code}\n" --cacert certs/CA_crt.pem --resolve svc1.example.com:443:$GATEWAY_IP  https://svc1.example.com/version; sleep 1; done

Kiali Dashboard

If you want, launch the kiali dashboard (default password is admin/admin). In a new window, run:

echo $ISTIO_VERSION 
echo $ISTIO_VERSION_MINOR
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/samples/addons/prometheus.yaml
sleep 20
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/samples/addons/kiali.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/samples/addons/grafana.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/samples/addons/jaeger.yaml

### in a new window, install prometheus, kaili, jager and grafana
## open a tunnel and access the kiali dashboard at  http://localhost:20001/kiali (admin/admi)
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app.kubernetes.io/name=kiali -o jsonpath='{.items[0].metadata.name}') 20001:20001

images/default-traffic.png

Generate Authz config

First we need to setup the auth* configs to use a convenient JWT/JWK issuer istio provides (you can use any jWT issuer, ofcourse; this is just a demo...do not use this in production!!!)

Use Istio's sample JWT issuer script

Istio provides a convenient JWT issuer, JWK and script the gateway will for authentication. You are certainly supposed to use your own JWK/JWT issuer; we're just using this one since it has a convenient JWK endpoint to verify the tokens with

We will use following script to issue a JWT and verify the JWK. This will be the same key that the external authorization server uses.

wget --no-verbose https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/security/tools/jwt/samples/gen-jwt.py
wget --no-verbose https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/security/tools/jwt/samples/key.pem

# may need pip3 install jwcrypto
python3 gen-jwt.py -aud some.audience -expire 3600 key.pem

{
  "alg": "RS256",
  "kid": "DHFbpoIUqrY8t2zpA2qXfCmr5VO5ZEr4RzHU_-envvQ",
  "typ": "JWT"
}

{
  "aud": "some.audience",
  "exp": 1635174518,
  "iat": 1635170918,
  "iss": "testing@secure.istio.io",
  "sub": "testing@secure.istio.io"
}

You can also see that its kid key-id is visible too "DHFbpoIUqrY8t2zpA2qXfCmr5VO5ZEr4RzHU_-envvQ"

The JWK endpoint istio will use to validate a JWT issued by the authorization server is:

$ curl -s https://raw.githubusercontent.com/istio/istio/release-1.10/security/tools/jwt/samples/jwks.json | jq '.'
{
  "keys": [
    {
      "e": "AQAB",
      "kid": "DHFbpoIUqrY8t2zpA2qXfCmr5VO5ZEr4RzHU_-envvQ",
      "kty": "RSA",
      "n": "xAE7eB6qugXyCAG3yhh7pkDkT65pHymX-P7KfIupjf59vsdo91bSP9C8H07pSAGQO1MV_xFj9VswgsCg4R6otmg5PV2He95lZdHtOcU5DXIg_pbhLdKXbi66GlVeK6ABZOUW3WYtnNHD-91gVuoeJT_DwtGGcp4ignkgXfkiEm4sw-4sfb4qdt5oLbyVpmW6x9cfa7vs2WTfURiCrBoUqgBo_-4WTiULmmHSGZHOjzwa8WtrtOQGsAFjIbno85jp6MnGGGZPYZbDAa_b3y5u-YpW7ypZrvD8BgtKVjgtQgZhLAGezMt0ua3DRrWnKqTZ0BJ_EyxOGuHJrLsn00fnMQ"
    }
  ]
}

NOTE: we will not be issuing these JWTs. The external authorization server will use the private key to reissue a JWT intended for a given service.

Apply the preset environment variables to ext_authz_filter.yaml:

export SERVICE_ACCOUNT_EMAIL="testing@secure.istio.io"
wget --no-verbose https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/security/tools/jwt/samples/key.pem
export KEY_ID="DHFbpoIUqrY8t2zpA2qXfCmr5VO5ZEr4RzHU_-envvQ"
export SVC_ACCOUNT_KEY=`base64 -w 0 key.pem && echo`

echo $SERVICE_ACCOUNT_EMAIL
echo $KEY_ID
echo $SVC_ACCOUNT_KEY

Apply Authz rules

envsubst < "ext_authz_rules.yaml.tmpl" > "ext_authz_rules.yaml"
kubectl apply -f ext_authz_rules.yaml

This will cause a 'deny' for everyone since we specified some headers that cannot be met (since we didnt' even deploy the authzserver in the first place that'd issue the JWT we just declared above!)

Deploy ExtAuthz server

Edit mesh-config

kubectl edit configmap istio -n istio-system

append the section for extensionProviders to the top of the mesh definition as such (remember to delete the definition of extensionProviders already set with envoyOtelAls)

apiVersion: v1 
data: 
  mesh: |-  
    extensionProviders:
    - name: "my-ext-authz-grpc"
      envoyExtAuthzGrpc:
        service: "authz.authz-ns.svc.cluster.local"
        port: "50051"
    - name: otel 
      envoyOtelAls: 
        port: 4317 
        service: opentelemetry-collector.istio-system.svc.cluster.local

images/config_image.png

please note the name for the provider: "my-ext-authz-grpc". This is defined in the ext_authz.yaml provider filter

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ext-authz
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: CUSTOM
  provider:
    name: "my-ext-authz-grpc"
  rules:
  - to:
    - operation:
        paths: ["/*"]

Reload the gateway:

INGRESS_POD_NAME=$(kubectl get po -n istio-system | grep ingressgateway\- | awk '{print$1}'); echo ${INGRESS_POD_NAME};
kubectl delete po/$INGRESS_POD_NAME -n istio-system

Apply the authz config

envsubst < "ext_authz.yaml.tmpl" > "ext_authz.yaml"
kubectl apply -f ext_authz.yaml

$ kubectl get PeerAuthentication,RequestAuthentication,AuthorizationPolicy -n authz-ns
NAME                                                                     MODE     AGE
peerauthentication.security.istio.io/ing-authzserver-peer-authn-policy   STRICT   13s

NAME                                                                 ACTION   AGE
authorizationpolicy.security.istio.io/deny-all-authz-ns                       13s
authorizationpolicy.security.istio.io/ing-authzserver-authz-policy   ALLOW    13s

$ kubectl get PeerAuthentication,RequestAuthentication,AuthorizationPolicy -n default
NAME                                                                     AGE
requestauthentication.security.istio.io/ing-svc1-request-authn-policy    109s
requestauthentication.security.istio.io/ing-svc2-request-authn-policy    109s
requestauthentication.security.istio.io/svc-be-v1-request-authn-policy   109s
requestauthentication.security.istio.io/svc-be-v2-request-authn-policy   109s

NAME                                                            ACTION   AGE
authorizationpolicy.security.istio.io/deny-all-default                   109s
authorizationpolicy.security.istio.io/ing-svc1-authz-policy     ALLOW    109s
authorizationpolicy.security.istio.io/ing-svc2-authz-policy     ALLOW    109s
authorizationpolicy.security.istio.io/svc1-be-v1-authz-policy   ALLOW    109s
authorizationpolicy.security.istio.io/svc1-be-v2-authz-policy   ALLOW    109s

Access Frontend

The static/demo configuration here uses two users (alice, bob), two frontend services (svc1,svc2) one backend service with two labled versions (be, version=v1,version=v2).

The following conditions are coded into the authorization server:

If the authorization server sees alice, it issues a JWT token with svc1 and be as the targets (multiple audiences)
If the authorization server sees bob, it issues a JWT token with svc2 as the target
If the authorization server sees carol, it issues a JWT token with svc1 as the target only.

var aud []string
if token == "alice" {
    aud = []string{"http://svc1.default.svc.cluster.local:8080/", "http://be.default.svc.cluster.local:8080/"}
} else if token == "bob" {
    aud = []string{"http://svc2.default.svc.cluster.local:8080/"}
} else if token == "carol" {
    aud = []string{"http://svc1.default.svc.cluster.local:8080/"}
} else {
    aud = []string{}
}

The net effect of that is alice can view svc1, bob can view svc2 using ORIGIN authentication.

As Alice:

export USER=alice

curl -s \
  --cacert certs/CA_crt.pem  --resolve svc1.example.com:443:$GATEWAY_IP \
  -H "Authorization: Bearer $USER" \
  -w " %{http_code}\n"  \
   https://svc1.example.com/version

curl -s \
  --cacert certs/CA_crt.pem  --resolve svc2.example.com:443:$GATEWAY_IP \
  -H "Authorization: Bearer $USER" \
  -w " %{http_code}\n"  \
   https://svc2.example.com/version

>>> 1 200
>>> Audiences in Jwt are not allowed 403

If you want to view the authz logs

AUTHZ_POD_NAME=$(kubectl get po -n authz-ns | grep authz\- | awk '{print$1}'); echo ${AUTHZ_POD_NAME};

kubectl logs -n authz-ns $AUTHZ_POD_NAME -c authz-container

You should see some debug logs as well as the actual reissued JWT header

2024/11/13 12:33:41 Starting gRPC Server at :50051
2024/11/13 12:34:01 >>> Authorization called check()
2024/11/13 12:34:01 Authorization Header Bearer alice
2024/11/13 12:34:01 Using Claim {alice [http://svc1.default.svc.cluster.local:8080/ http://be.default.svc.cluster.local:8080/] {testing@secure.istio.io testing@secure.istio.io [] 2024-11-13 12:35:01.446845225 +0000 UTC m=+79.858574856 <nil> 2024-11-13 12:34:01.446845119 +0000 UTC m=+19.858574750 }}
2024/11/13 12:34:01 Issuing outbound Header eyJhbGciOiJSUzI1NiIsImtpZCI6IkRIRmJwb0lVcXJZOHQyenBBMnFYZkNtcjVWTzVaRXI0UnpIVV8tZW52dlEiLCJ0eXAiOiJKV1QifQ.eyJ1aWQiOiJhbGljZSIsImF1ZCI6WyJodHRwOi8vc3ZjMS5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsOjgwODAvIiwiaHR0cDovL2JlLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWw6ODA4MC8iXSwiaXNzIjoidGVzdGluZ0BzZWN1cmUuaXN0aW8uaW8iLCJzdWIiOiJ0ZXN0aW5nQHNlY3VyZS5pc3Rpby5pbyIsImV4cCI6MTczMTUwMTMwMSwiaWF0IjoxNzMxNTAxMjQxfQ.Q-itZwt9AkSjpR3JjHxAyMMdhUOMwythPdB3IH-kZspwP4PH87BGyKNs71WzRqTCbOkd9U9EoiGEO16blV_EFpBQHKkHCIp-T070D2eyJu262MXr3tCwsrp0YHl-tx7qyICoLtMe77LNduI8bWj_2Q61fwALnAOslyH0Cj47u7Gq1FQ0-dFXssR8oMXM8eNSaF30oU2SHf_FGrd56TgJ8gCcl0Qhik6qC11ihjKl8S3_ccw1D48iCX8JlA8cWR5JMTqHhwQEEdTZtMJAR7HB0DuSAMKxWu2ENuyE6_lLDQmbLPbwTW6dy1nJa4JQGA9Eo6JtfWf3FHlAc7QuFfvz3Q
2024/11/13 12:34:01 >>> Authorization called check()
2024/11/13 12:34:01 Authorization Header Bearer alice
2024/11/13 12:34:01 Using Claim {alice [http://svc1.default.svc.cluster.local:8080/ http://be.default.svc.cluster.local:8080/] {testing@secure.istio.io testing@secure.istio.io [] 2024-11-13 12:35:01.504861628 +0000 UTC m=+79.916591235 <nil> 2024-11-13 12:34:01.50486156 +0000 UTC m=+19.916591168 }}
2024/11/13 12:34:01 Issuing outbound Header eyJhbGciOiJSUzI1NiIsImtpZCI6IkRIRmJwb0lVcXJZOHQyenBBMnFYZkNtcjVWTzVaRXI0UnpIVV8tZW52dlEiLCJ0eXAiOiJKV1QifQ.eyJ1aWQiOiJhbGljZSIsImF1ZCI6WyJodHRwOi8vc3ZjMS5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsOjgwODAvIiwiaHR0cDovL2JlLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWw6ODA4MC8iXSwiaXNzIjoidGVzdGluZ0BzZWN1cmUuaXN0aW8uaW8iLCJzdWIiOiJ0ZXN0aW5nQHNlY3VyZS5pc3Rpby5pbyIsImV4cCI6MTczMTUwMTMwMSwiaWF0IjoxNzMxNTAxMjQxfQ.Q-itZwt9AkSjpR3JjHxAyMMdhUOMwythPdB3IH-kZspwP4PH87BGyKNs71WzRqTCbOkd9U9EoiGEO16blV_EFpBQHKkHCIp-T070D2eyJu262MXr3tCwsrp0YHl-tx7qyICoLtMe77LNduI8bWj_2Q61fwALnAOslyH0Cj47u7Gq1FQ0-dFXssR8oMXM8eNSaF30oU2SHf_FGrd56TgJ8gCcl0Qhik6qC11ihjKl8S3_ccw1D48iCX8JlA8cWR5JMTqHhwQEEdTZtMJAR7HB0DuSAMKxWu2ENuyE6_lLDQmbLPbwTW6dy1nJa4JQGA9Eo6JtfWf3FHlAc7QuFfvz3Q

note JWT headers include cliams and audiences

{
  "alg": "RS256",
  "kid": "DHFbpoIUqrY8t2zpA2qXfCmr5VO5ZEr4RzHU_-envvQ",
  "typ": "JWT"
}

{
  "uid": "alice",
  "aud": [
    "http://svc1.default.svc.cluster.local:8080/",
    "http://be.default.svc.cluster.local:8080/"
  ],
  "exp": 1635173568,
  "iat": 1635173508,
  "iss": "testing@secure.istio.io",
  "sub": "testing@secure.istio.io"
}

As Bob:

export USER=bob

curl -s \
  --cacert certs/CA_crt.pem  --resolve svc1.example.com:443:$GATEWAY_IP \
  -H "Authorization: Bearer $USER" \
  -w " %{http_code}\n"  \
   https://svc1.example.com/version

curl -s \
  --cacert certs/CA_crt.pem  --resolve svc2.example.com:443:$GATEWAY_IP \
  -H "Authorization: Bearer $USER" \
  -w " %{http_code}\n"  \
   https://svc2.example.com/version

>>> Audiences in Jwt are not allowed 403
>>> 2 200

As Carol

export USER=carol

curl -s \
  --cacert certs/CA_crt.pem  --resolve svc1.example.com:443:$GATEWAY_IP \
  -H "Authorization: Bearer $USER" \
  -w " %{http_code}\n"  \
   https://svc1.example.com/version

curl -s \
  --cacert certs/CA_crt.pem  --resolve svc2.example.com:443:$GATEWAY_IP \
  -H "Authorization: Bearer $USER" \
  -w " %{http_code}\n"  \
   https://svc2.example.com/version

>>> 1 200
>>> Audiences in Jwt are not allowed 403

images/authz_ns_flow_fe.png

note, it seems the traffic from the gateway to the authorization server isn't correctly detected to be associated with the ingress-gateway (maybe a bug or some label is missing)

Access Backend

The configuration also defines Authorization policies on the svc1-> be traffic using BOTH PEER and ORIGIN.

PEER:

This is done using normal RBAC service identities:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
 name: svc1-be-v1-authz-policy
 namespace: default
spec:
 action: ALLOW
 selector:
   matchLabels:
     app: be
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/svc1-sa"]
   to:
   - operation:
       methods: ["GET"]

Backend PEER and ORIGIN

Note the from->source->principals denotes the service account svc1 runs as.

ORIGIN

THis step is pretty unusual and requires some changes to application code to forward its inbound authentication token.

Recall the inbound JWT token to svc1 for alice includes two audiences:

    aud = []string{"http://svc1.default.svc.cluster.local:8080/", "http://be.default.svc.cluster.local:8080/"}

This means we can use the same JWT token on the backend service if we setup an authentication and authz rule:

## svc --> be-v1
apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
 name: svc-be-v1-request-authn-policy
 namespace: default
spec:
  selector:
    matchLabels:
      app: be
      version: v1
  jwtRules:
  - issuer: "$SERVICE_ACCOUNT_EMAIL"
    audiences:
    - "http://be.default.svc.cluster.local:8080/"   
    jwksUri: "https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/security/tools/jwt/samples/jwks.json" 
    outputPayloadToHeader: x-jwt-payload  
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
 name: svc1-be-v1-authz-policy
 namespace: default
spec:
 action: ALLOW
 selector:
   matchLabels:
     app: be
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/svc1-sa"]
   to:
   - operation:
       methods: ["GET"]
   when:
   - key: request.auth.claims[iss]
     values: ["$SERVICE_ACCOUNT_EMAIL"]
   - key: request.auth.claims[aud]
     values: ["http://be.default.svc.cluster.local:8080/"]

The RequestAuthentication accepts a JWT token signed by the external authz server and must also include the audience of the backend (which alice's token has). The second authorization (redundantly) rule further parses out the token and looks for the same.

Istio does not automatically forward the inbound token (though it maybe possible with SIDECAR_INBOUND->SIDECAR_OUTBOUND forwarding somehow...)...to achieve this requres some application code changes. The folloing snippet is the code within frontend/app.js which take the token and uses it on the backend api call.

4/27/20: update on the comment "(though it maybe possble with SIDECAR_INBOUND->SIDECAR_OUTBOUND forwarding somehow...)" Its not; envoy doens't carry state from the filters forward like this. You need to either accept and forward the header in code as shown below:

var resp_promises = []
var urls = [
            'http://' + host + ':' + port + '/backend',
            'http://' + host + ':' + port + '/headerz',
]

out_headers = {};
if (FORWARD_AUTH_HEADER == 'true') {
    var auth_header = request.headers['authorization']; 
    logger.info("Got Authorization Header: [" + auth_header + "]");
      out_headers = {
          'authorization':  auth_header,
      };
    }

urls.forEach(element => {
     resp_promises.push( getURL(element,out_headers) )
});

Or configure istio to make an OUTBOUND ext_authz filter call. The external authz filter will return a new Authorization server token intended for ust svcb.

You will also need to set allowed_client_headers so that the auth token returned by ext-authz server is sent to the upstream (in this case, upstream is svcb)

I think the config would be something like this:

apiVersion: networking.istio.io/v1
kind: EnvoyFilter
metadata:
  name: ext-authz-service
  namespace: default
spec:
  workloadLabels:
    app: svc1
  filters:
  - listenerMatch:
      listenerType: OUTBOUND    #  <<<<  OUTBOUND svc1->*  
      listenerProtocol: HTTP 
    insertPosition:
      index: FIRST           
    filterName: envoy.ext_authz
    filterType: HTTP
    filterConfig:
      grpc_service:
        envoy_grpc:
          cluster_name: patched.authz.authz-ns.svc.cluster.local      
          authorization_response:
            allowed_client_headers:
              patterns:
                - exact: "Authorization"

(ofcourse changes are needed to ext-authz server as provided in this repo..)

Note: i added both ORIGIN and PEER just to demonstrate this...Until its easier forward the token by envoy/istio, i woudn't recommend doing this bit..

Anwyay, to test all this out

export USER=alice

curl -s \
  --cacert certs/CA_crt.pem  --resolve svc1.example.com:443:$GATEWAY_IP \
  -H "Authorization: Bearer $USER" \
  -w " %{http_code}\n"  \
   https://svc1.example.com/backend | jq '.'

export USER=bob

curl -s \
  --cacert certs/CA_crt.pem  --resolve svc2.example.com:443:$GATEWAY_IP \
  -H "Authorization: Bearer $USER" \
  -w " %{http_code}\n"  \
   https://svc2.example.com/backend | jq '.'

export USER=carol

curl -s \
  --cacert certs/CA_crt.pem  --resolve svc1.example.com:443:$GATEWAY_IP \
  -H "Authorization: Bearer $USER" \
  -w " %{http_code}\n"  \
   https://svc1.example.com/backend | jq '.'

Sample output

-Alice

Alice's TOKEN issued by the authorization server includes two audiences:

aud = []string{"http://svc1.default.svc.cluster.local:8080/", "http://be.default.svc.cluster.local:8080/"}

Which is allowed by backend services RequestAuthentication policy.

export USER=alice
curl -s \
  --cacert certs/CA_crt.pem  --resolve svc1.example.com:443:$GATEWAY_IP \
  -H "Authorization: Bearer $USER" \
  -w " %{http_code}\n"  \
   https://svc1.example.com/backend | jq '.'

[
  {
    "url": "http://be.default.svc.cluster.local:8080/backend",
    "body": "pod: [be-v2-64d9cf5fb4-mpsq5]    node: [gke-istio-1-default-pool-b516bc56-xz2c]",
    "statusCode": 200
  },
  {
    "url": "http://be.default.svc.cluster.local:8080/headerz",
    "body": "{\"host\":\"be.default.svc.cluster.local:8080\",\"x-forwarded-proto\":\"http\",\"x-request-id\":\"bb31942c-f04e-9b12-ba69-d68603a520af\",\"content-length\":\"0\",\"x-forwarded-client-cert\":\"By=spiffe://cluster.local/ns/default/sa/be-sa;Hash=2e0f9ca7bea6ac081f4c256de79ffdb4db2e55968b0ded2526e95cb89f4c36ac;Subject=\\\"\\\";URI=spiffe://cluster.local/ns/default/sa/svc1-sa\",\"x-b3-traceid\":\"cda6d87c8d342998ee1f797471592dff\",\"x-b3-spanid\":\"6dc54e848db21050\",\"x-b3-parentspanid\":\"ee1f797471592dff\",\"x-b3-sampled\":\"1\"}",
    "statusCode": 200
  }
]

Bob's token does not include the backend service

aud = []string{"http://svc2.default.svc.cluster.local:8080/"}

Which means the RequestAuthentication will fail. Bob is only allowed to invoke svc2 anyway

export USER=bob
curl -s \
  --cacert certs/CA_crt.pem  --resolve svc2.example.com:443:$GATEWAY_IP \
  -H "Authorization: Bearer $USER" \
  -w " %{http_code}\n"  \
   https://svc2.example.com/backend  | jq '.'

[
  {
    "url": "http://be.default.svc.cluster.local:8080/backend",
    "body": "Audiences in Jwt are not allowed",
    "statusCode": 403
  },
  {
    "url": "http://be.default.svc.cluster.local:8080/headerz",
    "body": "Audiences in Jwt are not allowed",
    "statusCode": 403
  }
]

Carol

Carol's token is allowed to invoke svc1 but does not include the issuer to pass the RequestAuthentication policy

aud = []string{"http://svc1.default.svc.cluster.local:8080/"}

export USER=carol
curl -s \
  --cacert certs/CA_crt.pem  --resolve svc1.example.com:443:$GATEWAY_IP \
  -H "Authorization: Bearer $USER" \
  -w " %{http_code}\n"  \
   https://svc1.example.com/backend | jq '.'

[
  {
    "url": "http://be.default.svc.cluster.local:8080/backend",
    "body": "Audiences in Jwt are not allowed",
    "statusCode": 403
  },
  {
    "url": "http://be.default.svc.cluster.local:8080/headerz",
    "body": "Audiences in Jwt are not allowed",
    "statusCode": 403
  }
]

images/authz_ns_flow_fe.png

If you would rather run these tests in a loop

 for i in {1..1000}; do curl -s \
  --cacert certs/CA_crt.pem  --resolve svc1.example.com:443:$GATEWAY_IP \
  -H "Authorization: Bearer $USER" \
  -w " %{http_code}\n"  \
   https://svc1.example.com/version; sleep 1; done

At this point, the system is setup to to always use mTLS, ORIGIN and PEER authentication plus RBAC. If you want to verify any component of PEER, change the policy and change the service account that is the target service authorization policy accepts and reapply the config.

Change either the settings RequestAuthentication or AuthorizationPolicy depending on which layer you are testing

(remember to replace the value for $ISTIO_VERSION_MINOR )

## svc --> be-v1
apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
 name: svc-be-v1-request-authn-policy
 namespace: default
spec:
  selector:
    matchLabels:
      app: be
      version: v1
  jwtRules:
  - issuer: "$SERVICE_ACCOUNT_EMAIL"
    audiences:
    - "http://be.default.svc.cluster.local:8080/"      ##  or CHANGE ORIGIN  <<<<  "Audiences in Jwt are not allowed"
    jwksUri: "https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/security/tools/jwt/samples/jwks.json"  
    # forwardOriginalToken: true
    outputPayloadToHeader: x-jwt-payload   
---
## svc --> be-v2
apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
 name: svc-be-v2-request-authn-policy
 namespace: default
spec:
  selector:
    matchLabels:
      app: be
      version: v2
  jwtRules:
  - issuer: "$SERVICE_ACCOUNT_EMAIL"
    audiences:
    - "http://be.default.svc.cluster.local:8080/"   ##  or CHANGE ORIGIN  <<<<  "Audiences in Jwt are not allowed"
    jwksUri: "https://raw.githubusercontent.com/istio/istio/release-$ISTIO_VERSION_MINOR/security/tools/jwt/samples/jwks.json"
    # forwardOriginalToken: true
    outputPayloadToHeader: x-jwt-payload
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: svc1-be-v1-authz-policy
 namespace: default
spec:
 action: ALLOW
 selector:
   matchLabels:
     app: be
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/svc1-sa"]    #  CHANGE  PEER  <<<<  "RBAC: access denied"
   to:
   - operation:
       methods: ["GET"]
   when:
   - key: request.auth.claims[iss]
     values: ["$SERVICE_ACCOUNT_EMAIL"]        ##  or CHANGE ORIGIN at Authz <<<<  "RBAC: access denied"
   - key: request.auth.claims[aud]
     values: ["http://be.default.svc.cluster.local:8080/"]          
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
 name: svc1-be-v2-authz-policy
 namespace: default
spec:
 action: ALLOW
 selector:
   matchLabels:
     app: be
     version: v2
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/svc1-sa"]   # CHANGE PEER <<<<  "RBAC: access denied"
   to:
   - operation:
       methods: ["GET"]
   when:
   - key: request.auth.claims[iss]
     values: ["$SERVICE_ACCOUNT_EMAIL"]        ##  or CHANGE ORIGIN at Authz <<<<  "RBAC: access denied"
   - key: request.auth.claims[aud]
     values: ["http://be.default.svc.cluster.local:8080/"]

then reapply the config and access the backend as alice

export USER=alice
curl -s \
  --cacert certs/CA_crt.pem  --resolve svc1.example.com:443:$GATEWAY_IP \
  -H "Authorization: Bearer $USER" \
  -w " %{http_code}\n"  \
   https://svc1.example.com/backend | jq '.'

[
  {
    "url": "http://be.default.svc.cluster.local:8080/backend",
    "body": "RBAC: access denied",
    "statusCode": 403
  },
  {
    "url": "http://be.default.svc.cluster.local:8080/headerz",
    "body": "RBAC: access denied",
    "statusCode": 403
  }
]

Finally, the external server is attached to the ingress gateway but you could also attach it to a sidecar for an endpoint. In this mode, the authorization decision is done not at the ingress gateway but locally on a service's sidecar. To use that mode, define the EnvoyFilter workloadLabel and listenerType. eg:

apiVersion: networking.istio.io/v1
kind: EnvoyFilter
metadata:
  name: svc1-authz-filter
  namespace: default
spec:
  workloadSelector:
    labels:
      app: svc1
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.router"
      patch:
        operation: INSERT_FIRST
        value:
         name: "envoy.filters.http.ext_authz"
         config:
           grpc_service:
             envoy_grpc:
               cluster_name: patched.authz.authz-ns.svc.cluster.local

If you do this, you will have to setup PEER policies that allow the service to connect and use the authorization server.

Debugging

You can debug issues using these resources

To set the log level higher and inspect a pod's logs:

istioctl manifest apply --set values.global.proxy.accessLogFile="/dev/stdout"

Ingress pod

INGRESS_POD_NAME=$(kubectl get po -n istio-system | grep ingressgateway\- | awk '{print$1}'); echo ${INGRESS_POD_NAME};

kubectl exec -ti $INGRESS_POD_NAME -n istio-syste -- /bin/bash
istioctl proxy-config log  $INGRESS_POD_NAME --level debug
kubectl logs -f --tail=0 $INGRESS_POD_NAME -n istio-system
istioctl dashboard envoy $INGRESS_POD_NAME.istio-system
istioctl experimental  authz check  $INGRESS_POD_NAME.istio-system

$ istioctl experimental  authz check  $INGRESS_POD_NAME.istio-system
Checked 2/2 listeners with node IP 10.48.2.5.
LISTENER[FilterChain]     CERTIFICATE                                 mTLS (MODE)     JWT (ISSUERS)     AuthZ (RULES)
0.0.0.0_80                none                                        no (none)       no (none)         no (none)
0.0.0.0_443               /etc/istio/ingressgateway-certs/tls.crt     no (none)       no (none)         no (none)

$ istioctl authn tls-check  $INGRESS_POD_NAME.istio-system authz.authz-ns.svc.cluster.local
HOST:PORT                                  STATUS     SERVER     CLIENT     AUTHN POLICY     DESTINATION RULE
authz.authz-ns.svc.cluster.local:50051     AUTO       STRICT     -          /default         -

Authz pod

AUTHZ_POD_NAME=$(kubectl get po -n authz-ns | grep authz\- | awk '{print$1}'); echo ${AUTHZ_POD_NAME};
istioctl proxy-config log  $AUTHZ_POD_NAME -n authz-ns  --level debug
kubectl logs -f --tail=0 $AUTHZ_POD_NAME -c authz-container -n  authz-ns
istioctl dashboard envoy $AUTHZ_POD_NAME.authz-ns
istioctl experimental  authz check $AUTHZ_POD_NAME.authz-ns

SVC1 pod

SVC1_POD_NAME=$(kubectl get po -n default | grep svc1\- | awk '{print$1}'); echo ${SVC1_POD_NAME};

$ istioctl authn tls-check  $SVC1_POD_NAME.default be.default.svc.cluster.local
HOST:PORT                             STATUS     SERVER     CLIENT           AUTHN POLICY     DESTINATION RULE
be.default.svc.cluster.local:8080     OK         STRICT     ISTIO_MUTUAL     /default         default/be-destination

SVC2 pod

SVC2_POD_NAME=$(kubectl get po -n default | grep svc2\- | awk '{print$1}'); echo ${SVC2_POD_NAME};

$ istioctl authn tls-check  $SVC2_POD_NAME.default be.default.svc.cluster.local
HOST:PORT                             STATUS     SERVER     CLIENT           AUTHN POLICY     DESTINATION RULE
be.default.svc.cluster.local:8080     OK         STRICT     ISTIO_MUTUAL     /default         default/be-destination

Using Google OIDC ORIGIN authentication at Ingress

If you want to use OIDC JWT authentication at the ingress gateway and then have that token forwarded to the external authz service, apply the RequestAuthentication policies on the ingress gateway as shown in the equivalent Envoy configuration here. You can generate an id-token using the script found under jwt_client/ folder.

Debugging

kubectl get pods -n istio-system -o name -l istio=ingressgateway | sed 's|pod/||' | while read -r pod; do istioctl proxy-config log "$pod" -n istio-system --level rbac:debug; done

salrashid123 / istio_external_authorization_server

readme

External Authorization Server with Istio

References

Setup

Build and push images

Deploy Istio Gateway and services

Debugging ingress-gateway

Deploy application

Send Traffic

Kiali Dashboard

Generate Authz config

Use Istio's sample JWT issuer script

Apply Authz rules

Deploy ExtAuthz server

Access Frontend

Access Backend

Backend PEER and ORIGIN

Debugging

Using Google OIDC ORIGIN authentication at Ingress

Debugging