search

open-policy-agent / opa

Open Policy Agent (OPA) is an open source, general-purpose policy engine.
https://www.openpolicyagent.org
Apache License 2.0
9.71k stars 1.35k forks source link

OPA [ Open Policy Agent ] is not working with istio in minikube #2454

Closed ajoysinha closed 4 years ago

ajoysinha commented 4 years ago

I am trying to implement sample provided by https://github.com/open-policy-agent/opa-istio-plugin

Implement step by step from https://github.com/open-policy-agent/opa-istio-plugin#quick-start

It deployed successfully and service also exposed

Also I can see that opa-istio is injected as sidecar with all the deployed applications

but opa policy is not working

As documentation says "Check that alice can access /productpage BUT NOT /api/v1/products"

product page is displaying with invocation of curl --user alice:password -i http://$GATEWAY_URL/productpage

when i am accessing curl --user alice:password -i http://$GATEWAY_URL/api/v1/products

It is displaying following result 

ajoy@workspace:~$ curl --user alice:password -i http://$GATEWAY_URL/api/v1/products
HTTP/1.1 200 OK
content-type: application/json
content-length: 395
server: istio-envoy
date: Wed, 03 Jun 2020 09:50:13 GMT
x-envoy-upstream-service-time: 13

[{"id": 0, "title": "The Comedy of Errors", "descriptionHtml": "<a href=\"https://en.wikipedia.org/wiki/The_Comedy_of_Errors\">Wikipedia Summary</a>: The Comedy of Errors is one of <b>William Shakespeare's</b> early plays. It is his shortest and one of his most farcical comedies, with a major part of the humour coming from slapstick and mistaken identity, in addition to puns and word play."}]

But this is not expected.. It seems that sample is not working properly

patrick-east commented 4 years ago

Can you share some more information about the environment? Kubernetes version, type of deployment, network config, istio version. Enough info that someone can reproduce the issue.

It would also be useful if you can share the logs from the various containers involved.

ashutosh-narkar commented 4 years ago

Istio version: 1.5.4

Envoy version:

$ kubectl exec -it prometheus-567f84f4d8-zrnwt -c istio-proxy -n istio-system pilot-agent request GET server_info
{
 "version": "9b4239dee83dd8894bfc579d412ccd894cff2597/1.13.1-dev/Clean/RELEASE/BoringSSL"
}

Kubernetes version: v1.14.0

OPA-Istio version: opa:0.20.5-istio

With the above setup, the policy is enforced as expected:

$ curl -s -o /dev/null -w "%{http_code}" --user alice:password http://$GATEWAY_URL/productpage
200

$ curl -s -o /dev/null -w "%{http_code}" --user alice:password http://$GATEWAY_URL/api/v1/products
403

Also the ext_authz filter seems to be properly set in the Envoy config:

$ istioctl proxy-config listeners productpage-v1-88646568d-wlqfq --port 15006 -o json

>>>>
 {
                                    "name": "envoy.ext_authz",
                                    "config": {
                                        "grpc_service": {
                                            "google_grpc": {
                                                "stat_prefix": "ext_authz",
                                                "target_uri": "127.0.0.1:9191"
                                            }
                                        },
                                        "with_request_body": {
                                            "allow_partial_message": true,
                                            "max_request_bytes": 8192
                                        }
                                    }
                                },

>>>>

Istio version: 1.6.0

Envoy version:

$ kubectl exec -it prometheus-57b7b99577-wmztl -c istio-proxy -n istio-system pilot-agent request GET server_info
{
"version": "12cfbda324320f99e0e39d7c393109fcd824591f/1.14.1/Clean/RELEASE/BoringSSL"
}

Kubernetes version: v1.14.0

OPA-Istio version: opa:0.20.5-istio

With the above setup, the policy is NOT enforced as expected:

$ curl -s -o /dev/null -w "%{http_code}" --user alice:password http://$GATEWAY_URL/productpage
200

$ curl -s -o /dev/null -w "%{http_code}" --user alice:password http://$GATEWAY_URL/api/v1/products
200

The issue seems to be that Envoy is not calling out to OPA since the ext_authz filter is not set in the Envoy config for the productpage app.

We may have to update the EnvoyFilter CRD to remove any deprecated fields to make this work.

ajoysinha commented 4 years ago

Here are the version list i am currently using in my local environment, [ using minikube ]

istio Version : 1.6.0 Minikube Version : 1.9.2 OPA-Istio version: opa:0.20.5-istio

ajoysinha commented 4 years ago

I have tested it with istio Version : 1.5.4 [ changed from 1.6.0 ] Minikube Version : 1.9.2 OPA-Istio version: opa:latest-istio

and application is working as expected with OPA policy 

curl -s -o /dev/null -w "%{http_code}" --user alice:password http://$GATEWAY_URL/productpage
200

curl -s -o /dev/null -w "%{http_code}" --user alice:password http://$GATEWAY_URL/api/v1/products

403

It seems that there is some issue with istio version 1.6.0

ashutosh-narkar commented 4 years ago

Thanks for verifying @ajoysinha . If we update the EnvoyFilter CRD to use the newer fields it should probably work. 1.6.0 may not be supporting the older format anymore. I'll try that out as well.

ajoysinha commented 4 years ago

I am upgraded my istio version to 1.6.3 ..but still facing the issue. ### we are planning to use istio plugin with this plugin version in production .. can you please tell me ..if this is possible or we should revert back to some old version. what version should be appropriate .. 

ajoy@workspace:~$ curl -s -o /dev/null -w "%{http_code}" --user alice:password http://$GATEWAY_URL/productpage
200
ajoy@workspace:~$ curl -s -o /dev/null -w "%{http_code}" --user alice:password http://$GATEWAY_URL/api/v1/products
200

my versions are

istio Version : 1.6.3 minikube version: v1.11.0

istioctl version client version: 1.6.3 control plane version: 1.6.3 data plane version: 1.6.3 (9 proxies)

ashutosh-narkar commented 4 years ago

@ajoysinha anything before 1.6.0 should work. I can spend some time on this next week to update the Envoy filter crd with the new format. I believe if we do that, 1.6.x should work as well.

blastdan commented 4 years ago

Hey we are running into the same issue. I'm unsure what from the filter needs to be removed to get this to work. Any way you could point me in the right direction?

ashutosh-narkar commented 4 years ago

Hello @blastdan, the Envoy Filter needs to be updated to follow this format.

blastdan commented 4 years ago

I tried to modify the filter to crd to the value below. It's still not picking anything up at this point.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ext-authz
  namespace: istio-system
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: "envoy.ext_authz"              
    patch:
      operation: INSERT_FIRST
      value:
        name: "envoy.ext_authz_config"
        config: 
          with_request_body:
            max_request_bytes: 8192
            allow_partial_message: true
          grpc_service:
            # NOTE(tsandall): when this was tested with the envoy_grpc client the gRPC
            # server was receiving check requests over HTTP 1.1. The gRPC server in
            # OPA-Istio would immediately close the connection and log that a bogus
            # preamble was sent by the client (it expected HTTP 2). Switching to the
            # google_grpc client resolved this issue.
            google_grpc:
              target_uri: 127.0.0.1:9191
              stat_prefix: "ext_authz"
  - applyTo: NETWORK_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: "envoy.ext_authz"
    patch:
      operation: INSERT_FIRST
      value:
        name: "ext_authz_config"
        config:
          stat_prefix: "ext_authz"
          grpc_service:
            google_grpc:
              target_uri: 127.0.0.1:9191
              stat_prefix: "ext_authz"

is there anything obvious I'm doing wrong? The only thing I can't seem to specify with the new format is the Listener Protocol. I'm going to keep working on it to see how it goes.

blastdan commented 4 years ago

Ok, I've done a lot more work on this.

I have the HTTP_FILTER being applied properly and it's successfully blocking the requests

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ext-authz
  namespace: istio-system
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: INSERT_BEFORE
      value:
        name: "envoy.ext_authz"
        typed_config: 
          "@type": type.googleapis.com/envoy.config.filter.http.ext_authz.v2.ExtAuthz
          with_request_body:
            max_request_bytes: 8192
            allow_partial_message: true
          grpc_service:
            # NOTE(tsandall): when this was tested with the envoy_grpc client the gRPC
            # server was receiving check requests over HTTP 1.1. The gRPC server in
            # OPA-Istio would immediately close the connection and log that a bogus
            # preamble was sent by the client (it expected HTTP 2). Switching to the
            # google_grpc client resolved this issue.
            google_grpc:
              target_uri: 127.0.0.1:9191
              stat_prefix: "ext_authz"

I used INSERT_BEFORE so that it will be applied to sub httpFilters collection instead of INSERT_FIRST that tries to apply it to the top filters. The documentation shows that if you don't apply a subfilter query it will apply the results to the top of the list

Insert operation on an array of named objects. This operation is typically useful only in the context of filters, where the order of filters matter. For clusters and virtual hosts, order of the element in the array does not matter. Insert before the selected filter or sub filter. If no filter is selected, the specified filter will be inserted at the front of the list.

this produces the expected results.

$ curl -s -o /dev/null -w "%{http_code}" --user alice:password http://$GATEWAY_URL/productpage
200

$ curl -s -o /dev/null -w "%{http_code}" --user alice:password http://$GATEWAY_URL/api/v1/products
403

However when I add the NETWORK_FILTER to try and replicate the TCP filter defined in the original quickstart.yaml

############################################################
# Envoy External Authorization filter that will query OPA.
############################################################
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ext-authz
  namespace: istio-system
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: INSERT_BEFORE
      value:
        name: "envoy.ext_authz"
        typed_config: 
          "@type": type.googleapis.com/envoy.config.filter.http.ext_authz.v2.ExtAuthz
          with_request_body:
            max_request_bytes: 8192
            allow_partial_message: true
          grpc_service:
            # NOTE(tsandall): when this was tested with the envoy_grpc client the gRPC
            # server was receiving check requests over HTTP 1.1. The gRPC server in
            # OPA-Istio would immediately close the connection and log that a bogus
            # preamble was sent by the client (it expected HTTP 2). Switching to the
            # google_grpc client resolved this issue.
            google_grpc:
              target_uri: 127.0.0.1:9191
              stat_prefix: "ext_authz"
  - applyTo: NETWORK_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: "envoy.tcp_proxy"
    patch:
      operation: INSERT_FIRST
      value:
        name: "ext_authz"
        typed_config: 
          "@type": type.googleapis.com/envoy.config.filter.network.ext_authz.v2.ExtAuthz
          stat_prefix: "ext_authz"
          grpc_service:
            google_grpc:
              target_uri: 127.0.0.1:9191
              stat_prefix: "ext_authz"

The opa-istio sidecars fail to start because the healthchecks are being blocked by the filter. The go into a crashloop.

Is the NETWORK_FILTER needed, or is the HTTP filter sufficient?

ashutosh-narkar commented 4 years ago

@blastdan this is great progress ! We will need the network filter as well. I'm curious why aren't you applying the ext_authz filter before the tcp_proxy filter ?

blastdan commented 4 years ago

Basically the match looks for all instances for "envoy.tcp_proxy" and places the ext_auth filter first in that filters list. I was trying to replicate the 

- insertPosition:
      index: FIRST

that was in the original quickstart.

The way I have the network filter written it places the filter above "istio.metadata_exchange", "istio.stats". I'll try the filter with INSERT_BEFORE to see if that works,

blastdan commented 4 years ago

I tried with INSERT_BEFORE

pod details returns

Events:
  Type     Reason     Age                From                     Message
  ----     ------     ----               ----                     -------
  Normal   Scheduled  51s                default-scheduler        Successfully assigned default/productpage-v1-6987489c74-gnbs5 to docker-desktop
  Normal   Pulling    50s                kubelet, docker-desktop  Pulling image "docker.io/istio/proxyv2:1.6.4"
  Normal   Pulled     46s                kubelet, docker-desktop  Successfully pulled image "docker.io/istio/proxyv2:1.6.4"
  Normal   Created    46s                kubelet, docker-desktop  Created container istio-init
  Normal   Started    46s                kubelet, docker-desktop  Started container istio-init
  Normal   Pulled     45s                kubelet, docker-desktop  Container image "docker.io/istio/examples-bookinfo-productpage-v1:1.16.2" already present on machine
  Normal   Created    45s                kubelet, docker-desktop  Created container productpage
  Normal   Started    44s                kubelet, docker-desktop  Started container productpage
  Normal   Pulling    44s                kubelet, docker-desktop  Pulling image "docker.io/istio/proxyv2:1.6.4"
  Normal   Pulled     43s                kubelet, docker-desktop  Successfully pulled image "docker.io/istio/proxyv2:1.6.4"
  Normal   Created    42s                kubelet, docker-desktop  Created container istio-proxy
  Normal   Started    42s                kubelet, docker-desktop  Started container istio-proxy
  Normal   Pulled     17s (x2 over 42s)  kubelet, docker-desktop  Container image "openpolicyagent/opa:0.21.0-istio" already present on machine
  Normal   Created    17s (x2 over 42s)  kubelet, docker-desktop  Created container opa-istio
  Normal   Started    17s (x2 over 42s)  kubelet, docker-desktop  Started container opa-istio
  Warning  Unhealthy  17s (x3 over 37s)  kubelet, docker-desktop  Liveness probe failed: Get "http://10.1.0.251:8282/health?plugins": EOF
  Normal   Killing    17s                kubelet, docker-desktop  Container opa-istio failed liveness probe, will be restarted
  Warning  Unhealthy  7s (x4 over 37s)   kubelet, docker-desktop  Readiness probe failed: Get "http://10.1.0.251:8282/health?plugins": EOF

All filter chains end up with the ext_authz filter applied at the network level as seen below. I'm wondering if we need to only have it applied to some filter chains?

"filters": [
                    {
                        "name": "istio.metadata_exchange",
                        "typedConfig": {
                            "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                            "typeUrl": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
                            "value": {
                                "protocol": "istio-peer-exchange"
                            }
                        }
                    },
                    {
                        "name": "istio.stats",
                        "typedConfig": {
                            "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                            "typeUrl": "type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm",
                            "value": {
                                "config": {
                                    "configuration": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\"\n}\n",
                                    "root_id": "stats_inbound",
                                    "vm_config": {
                                        "code": {
                                            "local": {
                                                "inline_string": "envoy.wasm.stats"
                                            }
                                        },
                                        "runtime": "envoy.wasm.runtime.null",
                                        "vm_id": "tcp_stats_inbound"
                                    }
                                }
                            }
                        }
                    },
                    {
                        "name": "ext_authz",
                        "typedConfig": {
                            "@type": "type.googleapis.com/envoy.config.filter.network.ext_authz.v2.ExtAuthz",
                            "statPrefix": "ext_authz",
                            "grpcService": {
                                "googleGrpc": {
                                    "targetUri": "127.0.0.1:9191",
                                    "statPrefix": "ext_authz"
                                }
                            }
                        }
                    },
                    {
                        "name": "envoy.tcp_proxy",
                        "typedConfig": {
                            "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                            "statPrefix": "InboundPassthroughClusterIpv4",
                            "cluster": "InboundPassthroughClusterIpv4",
                            "accessLog": [
                                {
                                    "name": "envoy.file_access_log",
                                    "typedConfig": {
                                        "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
                                        "path": "/dev/stdout",
                                        "format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
                                    }
                                }
                            ]
                        }
                    }
                ]
ashutosh-narkar commented 4 years ago

So the opa-istio container comes up but fails liveness and readiness checks as those calls are expected to be authorized ? Is this accurate @blastdan or am I missing something here ?

blastdan commented 4 years ago

That seems to be the issue for me @ashutosh-narkar. I'm not 100% sure how to move forward, as I don't fully understand what level of blocking is required in each filter section to satisfy the security requirements.

ashutosh-narkar commented 4 years ago

I'll take your example and experiment with the network filter. Thanks !

ashutosh-narkar commented 4 years ago

I've created this PR with the updated Envoy Filter configuration using @blastdan's provided example. I've tested this with Istio v1.5.x and v1.6.x and it seems to work fine. @ajoysinha feel free it try it out. Thanks @blastdan for your contribution !

bbdimitriu commented 4 years ago

@ashutosh-narkar the latest PR does not contain the NETWORK_FILTER. Was that intentional? I've managed to get it to work with the following configuration (verified in Istio 1.5 and 1.6):

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: authx
  namespace: istio-system
spec:
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.tcp_proxy"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.ext_authz
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.network.ext_authz.v3.ExtAuthz
            failure_mode_allow: false
            stat_prefix: ext_authz
            grpc_service:
              google_grpc:
                target_uri: 127.0.0.1:9191
                stat_prefix: ext_authz
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.http_connection_manager"
              subFilter:
                name: "envoy.router"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.ext_authz
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
            failure_mode_allow: false
            grpc_service:
              google_grpc:
                target_uri: 127.0.0.1:9191
                stat_prefix: ext_authz
ajoysinha commented 4 years ago

@bbdimitriu , When i am trying to implement NETWORK_FILTER it is throwing error. but only HTTP_FILTER is working fine. is NETWORK_FILTER is must for EnvoyFiltter .. I am running on 1.6.11 istio version on EKS 1.16

bbdimitriu commented 4 years ago

@ajoysinha the exact above configuration works fine on my setup (Istio 1.6.8 + Kubernetes 1.15). Can you provide details of the thrown error?

ajoysinha commented 4 years ago

@bbdimitriu , i am trying to execute this on following setup ..

Istio : 1.6.11 k8s : 1.16 [ EKS ] OAP : 0.24

And I am getting following error during deployment.

Liveness probe failed: Get http://172.31.43.142:8282/health?plugins: EOF Readiness probe failed: Get http://172.31.43.142:8282/health?plugins: EOFut

But, if I remove NETWORK_FILTER patching, [ keeping HTTP_FILTER only ] it is working fine.

@ashutosh-narkar do we need NETWORK_FILTER for sidecar communication.

bbdimitriu commented 4 years ago

@ajoysinha have you named your service as described here: https://istio.io/v1.6/docs/ops/configuration/traffic-management/protocol-selection/#manual-protocol-selection?

ajoysinha commented 4 years ago

@bbdimitriu ..yes , my service port is prefixed with http- . and it is working for HTTP_FILTER, Do I need to change it for NETWORK_FILTER .. please suggest.

ashutosh-narkar commented 4 years ago

@ajoysinha you should only need the http filter.