working with istio

[danutc]

Hi We intend to use istio (https://istio.io/). I could not find any resource that shows how I could combine kubeless with istio. I know it is early days but did you give any consideration in using the two together?

[sebgoa]

@danutc actually we did. I tried istio when it came out, as you say it is still early but very promising. It would be a great complement to kubeless. Specifically the auth part and the tracing part. The problem that I faced is that istio injects their own agent in your pods, and it dramatically slowed down the startup of the functions.

We need to monitor the development and see how best to integrate.

[danutc]

@sebgoa you may raise a github ticket with istio and explain the problem you faced; they may have a simple solution that can be implemented; I guess they are unaware of your experience.

[stevenceuppens]

@danutc cilium.io (a Linux native network security Project) is working on integration of Istio. Currently they already use envoy (agent that istio is injecting) to run their L7 http policies. They have a different model where they run a single envoy instance, instead of 1 envoy per pod. Soon they will be able to optimize Istio, so that this single Envoy can be used to drive Istio in combination with Cilium! They do this by redirecting traffic within the Linux kernel trough BPF programs. Check out their website for other awesome stuff they bring to container networking and kubernetes!

[arapulido]

@danutc @andresmgot wrote a blog post explaining how to use Istio with Kubeless: https://engineering.bitnami.com/articles/serverless-service-mesh-with-kubeless-and-istio.html

Let us know if this works for you.

Thanks!

[andresmgot]

Actually there are a few PRs that has been merged that solve some of the issues I found when writing the article. Maybe we should write some documentation after the next release to explain how to make both projects work smoothly.

[sslavic]

At least one known remaining Kubeless-Istio compatibility issue is that Kubeless for the function defines Deployment with httpGet liveness probe, which is not compatible with Istio that has mutual TLS on by default, liveness probe for function container keeps on failing, because of that function Pod main container repeatedly crashes and gets recreated, so is not as available as it could be. (see https://istio.io/help/faq/security.html#k8s-health-checks)

Another, functional nice to have missing is ability to disable mutual TLS for the function - e.g. by adding auth.istio.io/8080: NONE annotation to the function definition and then Kubeless controller applying it to the function's Service. (see https://istio.io/docs/tasks/security/per-service-mtls.html#disable-mutual-tls-authentication-for-httpbin-service) - current workaround is to apply the annotation directly to the Service after creating the Kubeless function but not sure how persistent that is (when can/will Kubeless change the Service definition and lose the custom modification, will it last while function is defined).

[joek]

@sslavic we are working on it and will come up with a proposal within the next few days. Today 0.5 was released to which might have an impact on our solution.

[sebgoa]

@joek can you update this issue, maybe a link to your talk ?

[joek]

Actually the issue is still there. As of now, we disabled mutualTLS to get it running on our systems. In demo scenarios we can enable mutualTLS. The containers are unstable then but as the restart is pretty fast, it is working ok.

Possible fixes to this issue are:

• Have a dedicated health and metrics port excluded from envoy (available with istio 0.8)
• Change health-checks to use curl

I personally prefere the first option, as I do not trust the curl way... Nevertheless both ways should work somehow.

[gaigepr]

I am trying to integrate kubeless and istio. The initCOntainer istio-init fails with no descriptive error message.

    Container ID:  docker://193a142e45db5e113e2bbbed0ec3d031a11a1616611a327e72d53f2db5fa761b
    Image:         gcr.io/istio-release/proxy_init:1.0.2
    Image ID:      docker-pullable://gcr.io/istio-release/proxy_init@sha256:e16a0746f46cd45a9f63c27b9e09daff5432e33a2d80c8cc0956d7d63e2f9185
    Port:          <none>
    Args:
      -p
      15001
      -u
      1337
      -m
      REDIRECT
      -i
      *
      -x

      -b
      8080,
      -d

    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    3
      Started:      Thu, 20 Sep 2018 16:12:10 -0700
      Finished:     Thu, 20 Sep 2018 16:12:11 -0700
    Ready:          False
    Restart Count:  4
    Environment:    <none>
    Mounts:         <none>```

[andresmgot]

Hi @gaigepr,

what are the logs of the container? (you can get them executing kubectl logs <pod_id> -c istio-init. Also it's possible that the error is reflected in the istio components. Are the pods in the istio-system healthy? Are there any logs in those pods that may help you debug the issue?

[d0x2f]

Hi @andresmgot,

I may be running into the same issue as @gaigepr.

Here are the pod logs:

Environment:
------------
ENVOY_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15001
INBOUND_CAPTURE_PORT=15001
PROXY_UID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=8080
INBOUND_PORTS_EXCLUDE=15020
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
KUBEVIRT_INTERFACES=

+ iptables -t nat -N ISTIO_REDIRECT
iptables v1.6.0: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
+ dump
+ iptables-save

I believe this is because kubeless pods run as uid 1000 (https://github.com/kubeless/kubeless/blob/master/pkg/utils/kubelessutil.go#L670) rather than the default root/0 that the istio-init container expects.

Is there a way to override the SecurityContext for kubeless pods? I imagine running as a non-root is for a good reason however.

The mongodb helm chart solved this by specifying the security context on the container level rather than pod wide, helm/charts#11367

ref: istio/old_issues_repo#316

p.s. let me know if I should make a separate issue for this.

[andresmgot]

hi @d0x2f,

Indeed that's a problem with Istio. See https://github.com/kubeless/kubeless/issues/796

Right now it's not possible to set a securityContext at the container level but you can disable that securityContext for all your functions if you want to. See this comment for more information about how to do that.