NetworkPolicy is bypassed while using Istio

[justinkillen]

When Istio is installed and set for gateway TLS origination, a default deny-all NetworkPolicy does not block all traffic.

Based on https://istio.io/docs/tasks/traffic-management/egress/egress-gateway-tls-origination/ with tweaks for client https -> gateway tls passthrough

Expected Behavior

Traffic should be blocked; NetworkPolicy should be enforced

Current Behavior

Traffic is being allowed; NetworkPolicy is not being enforced

Possible Solution

If I add the policy and wait a long time with no traffic, then new traffic does get blocked as expected, so perhaps it's a cache / policy refresh issue?

Steps to Reproduce (for bugs)

Install Istio via: istioctl manifest apply --set values.gateways.enabled=true --set values.gateways.istio-ingressgateway.sds.enabled=true --set values.global.k8sIngress.enabled=true --set values.global.k8sIngress.enableHttps=true --set values.global.k8sIngress.gatewayName=ingressgateway --set values.kiali.enabled=true

Apply these resources:

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: edition-cnn-com-egress
  namespace: istio-system
spec:
  host: istio-egressgateway
  subsets:
  - name: tls-origination
  - name: tls-passthrough
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: edition-cnn-com-origination
  namespace: istio-system
spec:
  host: edition.cnn.com
  subsets:
  - name: tls-origination
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 443
        tls:
          mode: SIMPLE
  - name: tls-passthrough
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 443
        tls:
          mode: DISABLE
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: edition-cnn-com
  namespace: istio-system
spec:
  selector:
    istio: egressgateway
  servers:
  - hosts:
    - edition.cnn.com
    port:
      name: http-port-for-tls-origination
      number: 80
      protocol: HTTP
  - hosts:
    - edition.cnn.com
    port:
      name: tls
      number: 443
      protocol: TLS
    tls:
      mode: PASSTHROUGH
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: edition-cnn-com
  namespace: istio-system
spec:
  hosts:
  - edition.cnn.com
  ports:
  - name: http
    number: 80
    protocol: HTTP
  - name: tls
    number: 443
    protocol: TLS
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: edition-cnn-com
  namespace: istio-system
spec:
  gateways:
  - edition-cnn-com
  - mesh
  hosts:
  - edition.cnn.com
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway
        port:
          number: 80
        subset: tls-origination
  - match:
    - gateways:
      - edition-cnn-com
      port: 80
    route:
    - destination:
        host: edition.cnn.com
        port:
          number: 443
        subset: tls-origination
      weight: 100
  tls:
  - match:
    - gateways:
      - mesh
      port: 443
      sniHosts:
      - edition.cnn.com
    route:
    - destination:
        host: istio-egressgateway
        port:
          number: 443
        subset: tls-passthrough
      weight: 100
  - match:
    - gateways:
      - edition-cnn-com
      port: 443
      sniHosts:
      - edition.cnn.com
    route:
    - destination:
        host: edition.cnn.com
        port:
          number: 443
        subset: tls-passthrough
      weight: 100

Now, create a utility pod in a separate namespace, e.g.:

---
apiVersion: v1
kind: Pod
metadata:
  name: netshoot
  namespace: default

spec:
  containers:
  - name: curlutils
    image: nicolaka/netshoot
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

exec into the pod and run:

curl -vm 5 -o /dev/null http://edition.cnn.com/politics

in a separate terminal, exec into the pod and run:

curl -vm 5 -o /dev/null https://edition.cnn.com/politics

notice both are working as expected. If you look at the proxy logs, you'll see traffic being diverted from the pod to the egress gateway and then out to edition.cnn.com on port 443

Now, apply the following NetworkPolicy, which should block all inbound traffic to the egress gateway

---
apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
  name: deny-all-egress
  namespace: istio-system
spec:
  ingress: []
  podSelector:
    matchLabels:
      istio: egressgateway
  policyTypes:
  - Ingress

You'll notice that the the https curl does in fact stop working as expected, but the http curl continues to work.

Reviewing the proxy logs, the flow stays the same: traffic being diverted from the pod to the egress gateway and then out to edition.cnn.com on port 443

Delete the NetworkPolicy, and https will start working again.

Context

Primary use cases is to apply egress restrictions to internal AWS EC2 microservices from components running in an AWS EKS cluster.

Your Environment

Calico daemonset info: Image: quay.io/calico/node:v3.3.6

Running in AWS EKS 1.15, using AWS provided images version 1.15.10-20200228. Node info (snippets): Labels: eks.amazonaws.com/nodegroup-image=ami-0582e4c984a1e848a System Info: Boot ID: 3b4adfb8-bd94-4f54-8d82-ac031550219d Kernel Version: 4.14.165-133.209.amzn2.x86_64 OS Image: Amazon Linux 2 Operating System: linux Architecture: amd64 Container Runtime Version: docker://18.9.9 Kubelet Version: v1.15.10-eks-bac369 Kube-Proxy Version: v1.15.10-eks-bac369

$ istioctl version --remote
client version: 1.5.1
control plane version: 1.5.1
data plane version: 1.5.1 (11 proxies)

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T23:41:24Z", GoVersion:"go1.14", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-af3caf", GitCommit:"af3caf6136cd355f467083651cc1010a499f59b1", GitTreeState:"clean", BuildDate:"2020-03-27T21:51:36Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

[caseydavenport]

Image: quay.io/calico/node:v3.3.6

This is a really old version of Calico that isn't in support any longer. Could you try to reproduce this on a modern version and see if the issue persists?

[justinkillen]

Updated to v3.13.0 (per EKS CNI plugin link https://docs.aws.amazon.com/eks/latest/userguide/calico.html).

Same issue

[caseydavenport]

@justinkillen just to double check, the Istio egress gateway is not host networked, correct?

[justinkillen]

It's EKS, using the AWS CNI