search

istio / old_issues_repo

Deprecated issue-tracking repo, please post new issues or feature requests to istio/istio instead.
37 stars 9 forks source link

Traffic Management: ALL https requests work (even without a ServiceEntry) #389

Closed filipre closed 5 years ago

filipre commented 6 years ago

Hi Istio-Team!

Is this a BUG or FEATURE REQUEST?: Bug

Did you review https://istio.io/help/ and existing issues to identify if this is already solved or being worked on?: Yes. Roughly related: https://github.com/istio/issues/issues/334

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:14:26Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

We use Istio 0.8

I can't use istioctl because clusters seem not to be supported yet: https://github.com/istio/istio/issues/1971. Is there another way to find out the exact Istio version number?

Is Istio Auth enabled or not ? I am not sure how istio has been installed on our cluster. However, in each pod there is a istio-proxy container. Is there a command I should run in order to find out?

What happened: For some reason, our apps can access ANY https resource, even though we did not create a ServiceRule for the right host. Example:

kubectl exec --namespace=[namespace-name] -it [pod-name] -- /bin/bash
root@[pod-name]:/app# curl -o /dev/null -s -w "%{http_code}\n" https://www.facebook.com
--
200

I checked all ServiceRules for a * host but I can't find one. We surely don't allow the host www.facebook.com somewhere.

What you expected to happen: I would have expected the request to fail. However, I receive a 200 status code back.

How to reproduce it: Our cluster has been set up by different people. Are there maybe any debug commands I could try to run in order to identify the root cause? I already tried:

kubectl get ServiceEntry --all-namespaces

and inspected each entry. Could there be a problem with our side-car-injections? Sorry for the sparse information I provide. I believe a rough direction for me to debug would already help me.

Thanks! René

louiscryan commented 6 years ago

Hmm. Its possible someone has configured the ip whitelist for bypass

https://istio.io/docs/tasks/traffic-management/egress/

filipre commented 6 years ago

Thanks for the answer! This seems likely. However, I am a bit confused by the documentation. It explains how to set the value during the Istio installation but not how to read it out. I took a look (kubectl describe) at the ConfigMap of the istio-sidecar-injector but it only shows the template and not the actual values. Is there a simple way to read out global.proxy.includeIPRanges without the usage of helm (we didn't even use helm for the installation nor any specified cloud provider there).

We use this istio-init-image: gcr.io/istio-release/proxy_init:release-0.8-20180531-09-15.

Here is the output of the describe command:

➜  kubectl describe ConfigMap istio-sidecar-injector -n istio-system -R
Name:         istio-sidecar-injector
Namespace:    istio-system
Labels:       app=istio
              chart=istio-0.8.0
              heritage=Tiller
              istio=sidecar-injector
              release=RELEASE-NAME
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","data":{"config":"policy: enabled\ntemplate: |-\n  initContainers:\n  - name: istio-init\n    image: gcr.io/istio-release/proxy_init...

Data
====
config:
----
policy: enabled
template: |-
  initContainers:
  - name: istio-init
    image: gcr.io/istio-release/proxy_init:release-0.8-20180531-09-15
    args:
    - "-p"
    - [[ .MeshConfig.ProxyListenPort ]]
    - "-u"
    - 1337
    - "-m"
    - [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
    - "-i"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges"  ]]"
    [[ else -]]
    - "*"
    [[ end -]]
    - "-x"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeOutboundIPRanges") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeOutboundIPRanges"  ]]"
    [[ else -]]
    - ""
    [[ end -]]
    - "-b"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeInboundPorts") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeInboundPorts"  ]]"
    [[ else -]]
    - [[ range .Spec.Containers -]][[ range .Ports -]][[ .ContainerPort -]], [[ end -]][[ end -]][[ end]]
    - "-d"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeInboundPorts") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeInboundPorts" ]]"
    [[ else -]]
    - ""
    [[ end -]]
    imagePullPolicy: IfNotPresent
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
      privileged: true
    restartPolicy: Always

  containers:
  - name: istio-proxy
    image: [[ if (isset .ObjectMeta.Annotations "sidecar.istio.io/proxyImage") -]]
    "[[ index .ObjectMeta.Annotations "sidecar.istio.io/proxyImage" ]]"
    [[ else -]]
    gcr.io/istio-release/proxyv2:release-0.8-20180531-09-15
    [[ end -]]
    args:
    - proxy
    - sidecar
    - --configPath
    - [[ .ProxyConfig.ConfigPath ]]
    - --binaryPath
    - [[ .ProxyConfig.BinaryPath ]]
    - --serviceCluster
    [[ if ne "" (index .ObjectMeta.Labels "app") -]]
    - [[ index .ObjectMeta.Labels "app" ]]
    [[ else -]]
    - "istio-proxy"
    [[ end -]]
    - --drainDuration
    - [[ formatDuration .ProxyConfig.DrainDuration ]]
    - --parentShutdownDuration
    - [[ formatDuration .ProxyConfig.ParentShutdownDuration ]]
    - --discoveryAddress
    - [[ .ProxyConfig.DiscoveryAddress ]]
    - --discoveryRefreshDelay
    - [[ formatDuration .ProxyConfig.DiscoveryRefreshDelay ]]
    - --zipkinAddress
    - [[ .ProxyConfig.ZipkinAddress ]]
    - --connectTimeout
    - [[ formatDuration .ProxyConfig.ConnectTimeout ]]
    - --statsdUdpAddress
    - [[ .ProxyConfig.StatsdUdpAddress ]]
    - --proxyAdminPort
    - [[ .ProxyConfig.ProxyAdminPort ]]
    - --controlPlaneAuthPolicy
    - [[ .ProxyConfig.ControlPlaneAuthPolicy ]]
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
    - name: INSTANCE_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
    - name: ISTIO_META_POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: ISTIO_META_INTERCEPTION_MODE
      value: [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
    imagePullPolicy: IfNotPresent
    securityContext:
        privileged: false
        readOnlyRootFilesystem: true
        [[ if eq (or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String) "TPROXY" -]]
        capabilities:
          add:
          - NET_ADMIN
        [[ else -]]
        runAsUser: 1337
        [[ end -]]
    restartPolicy: Always
    resources:
      requests:
        cpu: 100m
        memory: 128Mi

    volumeMounts:
    - mountPath: /etc/istio/proxy
      name: istio-envoy
    - mountPath: /etc/certs/
      name: istio-certs
      readOnly: true
  volumes:
  - emptyDir:
      medium: Memory
    name: istio-envoy
  - name: istio-certs
    secret:
      optional: true
      [[ if eq .Spec.ServiceAccountName "" -]]
      secretName: istio.default
      [[ else -]]
      secretName: [[ printf "istio.%s" .Spec.ServiceAccountName ]]
      [[ end -]]
Events:  <none>

Or do I have to inspect a different resource to find out whether someone could configure a bypass?

Greetings René

louiscryan commented 6 years ago

Hey. Take a look at

https://istio.io/docs/setup/kubernetes/helm-install/

The values for the fields come from the annotations block of the config-map itself. If you don't have an annotation there for excludeIP then we'll have to look elsewhere.

You can also see what the value is being expanded to by looking at the podspec of a pod that was mutated by the webhook

filipre commented 6 years ago

Hi,

ok, I am starting to understand it. So for each pod there will be an istio-init and istio-proxy container which will be initialised by the template defined in this ConfigMap I pasted above. If you use helm and you set global.proxy.includeIPRanges, then it will set the .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges" annotation with the value you put into it. However, if you don't provide this annotation (or use helm), then Istio will use '*' which means, ALL outbound traffic is redirected to Envoy.

Here are the initContainer-arguments of one of our services:

  initContainers:
  - args:
    - -p
    - "15001"
    - -u
    - "1337"
    - -m
    - REDIRECT
    - -i
    - '*'
    - -x
    - ""
    - -b
    - 3000,
    - -d
    - ""
    image: gcr.io/istio-release/proxy_init:release-0.8-20180531-09-15
    imagePullPolicy: IfNotPresent
    name: istio-init
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
      privileged: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File

However, that means that all outbound traffic (like you have when you do curl https://www.facebook.com/ inside that container) for this service should go to Envoy first. But, doing the curl command gives me the full HTML document of Facebook, even though we didn't whitelist it. Can you confirm that this is right what I wrote and that it is not bypassing the service?

Maybe reinstalling Istio on our cluster solves this problem. Though, it would be nice to identify the root cause.

louiscryan commented 6 years ago

That all looks right. I think I know what is going on here.

Your request is HTTPS not HTTP and because TLS is originated by curl the traffic is opaque to the proxy, i.e. it looks like TCP. By default Istio acts like a transparent TCP proxy, primarily because we don't want your apps to break when you install.

We configure Envoy to forward traffic to the original target ip & port https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/service_discovery#arch-overview-service-discovery-types-original-destination in this mode. IIRC you should be able to see metrics about TCP connections being created that line up with your testing in Prometheus

(Its also possible but less likely that someone has defined a ServiceEntry resource that captures the resolved IP for www.facebook.com in the ServiceEntry.addresses field)

Instead of curl-ing HTTPS, try HTTP instead. If you do this outside Istio you will get a 302 response. If you do this with Istio it doesn't know how to resolve facebook.com (no ServiceEntry) and you'll get either 404/503

Aside, the fact that HTTPS will pass through transparently and HTTP does not is somewhat inconsistent and something we frequently discuss.

If you're feeling adventurous you can create a ServiceEntry for ' www.facebook.com' and define its IPs, & ports. Then when you use HTTP to address it inside the mesh it should succeed.

On Wed, Jun 20, 2018 at 2:35 AM René Filip notifications@github.com wrote:

Hi,

ok, I am starting to understand it. So for each pod there will be an istio-init and istio-proxy container which will be initialised by the template defined in this ConfigMap I pasted above. If you use helm and you set global.proxy.includeIPRanges, then it will set the .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges" annotation with the value you put into it. However, if you don't provide this annotation (or use helm), then Istio will use '*' which means, ALL outbound traffic is redirected to Envoy.

Here are the initContainer-arguments of one of our services:

initContainers:

  • args:
    • -p
    • "15001"
    • -u
    • "1337"
    • -m
    • REDIRECT
    • -i
    • '*'
    • -x
    • ""
    • -b
    • 3000,
    • -d
    • "" image: gcr.io/istio-release/proxy_init:release-0.8-20180531-09-15 imagePullPolicy: IfNotPresent name: istio-init resources: {} securityContext: capabilities: add:
      • NET_ADMIN privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File

However, that means that all outbound traffic (like you have when you do curl https://www.facebook.com/ inside that container) for this service should go to Envoy first. But, doing the curl command gives me the full HTML document of Facebook, even though we didn't whitelist it. Can you confirm that this is right what I wrote and that it is not bypassing the service?

Maybe reinstalling Istio on our cluster solves this problem. Though, it would be nice to identify the root cause.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/istio/old_issues_repo/issues/389#issuecomment-398687521, or mute the thread https://github.com/notifications/unsubscribe-auth/AIoKPICyEIXHplux5Eh8RxjRObuyzHfwks5t-hdrgaJpZM4Ul_PZ .

filipre commented 6 years ago

Hi,

By default Istio acts like a transparent TCP proxy, primarily because we don't want your apps to break when you install.

Are you sure about that? Because then I don't understand the documentation. If Istio acts like a transparent TCP proxy, why do you have to add this ServiceEntry to make https://www.google.com work?

cat <<EOF | istioctl create -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: google-ext
spec:
  hosts:
  - www.google.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
EOF

In the documentation you do exactly the thing I do (notice that you use HTTPS aswell):

$ curl https://www.google.com

Also, when I tested it on Minikube, https://www.google.com only worked after I added the ServiceEntry entry above. Once I removed it, it didn't work again (which I expected!).

Regarding our problem: We surely didn't whitelist www.facebook.com. It turned out, that the curl requests only sometimes work (80% maybe?). This makes us believe, that there might be some "zombie" process running or somehow two different versions of envoy or something like that. The best solution would be to reinstall Istio. Still, it would be nice if you could resolve my confusion regarding the "transparent TCP proxy" :)

filipre commented 5 years ago

ok. For anyone having the same issue: It turned out that the resolution field of the ServiceEntry field was missing. If it is missing it defaults to NONE.

REQUIRED: Service discovery mode for the hosts. Care must be taken when setting the resolution mode to NONE for a TCP port without accompanying IP addresses. In such cases, traffic to any IP on said port will be allowed (i.e. 0.0.0.0:). https://istio.io/docs/reference/config/istio.networking.v1alpha3/