Provide a generic solution for the Istio CNI plugin init-container limitation

[DnPlas]

What needs to get done

Because of the Istio CNI plugin limitations, some workloads may never start as they may have init-containers that require network connectivity before the Istio sidecar proxy container is ready.

The goal of this task is to provide a solution that automatically injects annotations to the Pods in the Istio mesh to exclude the redirection of certain IP ranges. This solution has been tested in #354 and it can be extended to all workloads in a Namespace.

Proposed solution

Have a webhook that automatically add the required annotations.

Why it needs to get done

To provide a solution to the limitations that we will be introducing by installing the Istio CNI plugin by default in the Charmed Kubeflow bundle.

[syncronize-issues-to-jira[bot]]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5063.

This message was autogenerated

[DnPlas]

After initial investigation, annotating the Pods with traffic.sidecar.istio.io/excludeOutboundIPRanges: "0.0.0.0/0" will actually allow outbound traffic from init-containers. To be able to annotate every Pod we could have something like kyverno, which via a Policy (a collection of rules) that could contain mutate or validate rules to be able to act on the cluster resources. We would be interested in the mutate rule, which will allow us to automatically annotate all Pods in the cluster. kyverno also provides an option to mutate existing resources which will come handy when upgrading from a Charmed Kubeflow w/o the Istio CNI plugin.

Example usage

After installation, it's fairly easy to create policies.

By creating a kyverno ClusterPolicy, we can annotate any new or existing Pods in the cluster:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-external-outbound-annotations
spec:
  rules:
  - name: all-pods-in-cluster
    match:
      any:
      - resources:
          kinds:
          - Pod
    mutate:
      patchStrategicMerge:
        metadata:
            annotations:
              traffic.sidecar.istio.io/excludeOutboundIPRanges: "0.0.0.0/0"

What would this mean for Charmed Kubeflow?

Since kyverno is a project that hasn't been charmed, we probably want to charm this solution.

[DnPlas]

There is an effort in upstream to also make these annotations configurable by the kubeflow-profile controller, which would mean that similar to what we currently do with namespace labels we could do with annotations. This solution is being discussed upstream in https://github.com/kubeflow/kubeflow/pull/7325#issuecomment-1838465513.

[DnPlas]

An alternative to charm kyverno could be using Istio's sidecarInjectorWebhook to automatically inject the required annotation to every Pod that gets created in a Namespace with sidecar injection enabled. The sidecarInjectorWebhook can be used as follows in the istioctl commands:

istioctl <command> --set "values.sidecarInjectorWebhook.injectedAnnotations.traffic\.sidecar\.istio\.io/excludeOutboundIPRanges=0.0.0.0/0" <flags>

I have successfully tested using this flag in a cluster with KServe InferenceServices and the results are promising: the sidecar gets injected as expected and the annotation is there on the Pod where the sidecar was injected.