Automatic sidecar injection

[electrocucaracha]

Overview

Istio project provides a method to automatic inject sidecars. This approach helps developers to focus on the business logic putting on side trivial things managed by the sidecars.

This feature can be implemented on this project having an universal sidecar implementation for covering most of the scenarios and the creation of an admission controller.

The following source code examples provides an idea of this proposal.

Pod definition (defined by user)

Pods can specify their requirements through the usage of annotations, like this one:

apiVersion: v1
kind: Pod 
metadata:
  name: pgw 
  annotations:
    ns.networkservicemesh.io/endpoints: |
      {   
        "name": "lte-network",
        "networkServices": [
          {"link": "s5u", "labels": "app=pgw-s5u", "ipAddress": "172.25.0.0/24"},
          {"link": "s5c", "labels": "app=pgw-s5c", "ipAddress": "172.25.1.0/24"},
          {"link": "sgi", "labels": "app=http-server-sgi", "ipAddress": "10.0.1.0/24", "route": "10.0.3.0/24"}
        ]   
      }   
  labels:
    app.kubernetes.io/name: pgw 
    network: pdn 
spec:
  containers:
    - image: electrocucaracha/pgw:0.7.5
      name: pgw 
      securityContext:
        capabilities:
          add: ["NET_ADMIN"]
      command:
        - "sh"
      args:
        - "/opt/gw-tester/script/init.sh"
      volumeMounts:
        - name: init-script
          mountPath: /opt/gw-tester/script
  volumes:
    - name: init-script
      configMap:
        name: pgw-init-script

Pod definition (modified by admission controller)

The previous Pod definition can be modified by the Admission controller and inject the sidecar and take advantage of the downwardAPI feature to define Network Services.

apiVersion: v1
kind: Pod 
metadata:
  name: pgw 
  annotations:
    ns.networkservicemesh.io/endpoints: |
      {   
        "name": "lte-network",
        "networkServices": [
          {"link": "s5u", "labels": "app=pgw-s5u", "ipAddress": "172.25.0.0/24"},
          {"link": "s5c", "labels": "app=pgw-s5c", "ipAddress": "172.25.1.0/24"},
          {"link": "sgi", "labels": "app=http-server-sgi", "ipAddress": "10.0.1.0/24", "route": "10.0.3.0/24"}
        ]   
      }   
  labels:
    app.kubernetes.io/name: pgw 
    network: pdn 
spec:
  containers:
    - name: sidecar
      image: nse:v0.0.1
      resources:
        limits:
          networkservicemesh.io/socket: 1
      volumeMounts:
        - name: nsm-endpoints
          mountPath: /etc/nsminfo
    - image: electrocucaracha/pgw:0.7.5
      name: pgw 
      securityContext:
        capabilities:
          add: ["NET_ADMIN"]
      command:
        - "sh"
      args:
        - "/opt/gw-tester/script/init.sh"
      volumeMounts:
        - name: init-script
          mountPath: /opt/gw-tester/script
  volumes:
    - name: init-script
      configMap:
        name: pgw-init-script
    - name: nsm-endpoints
      downwardAPI:
        items:
          - path: endpoints
            fieldRef:
              fieldPath: metadata.annotations['ns.networkservicemesh.io/endpoints']

Universal sidecar

The following code is an example of the universal sidecar implementation.

Note: some lines of code were omitted to reduce the size of this document.

// Endpoint contains the information to create NSM objects
type Endpoint struct {
    Name            string           `json:"name"`
    NetworkServices []NetworkService `json:"networkServices"`
}

// NetworkService contains the information to create NSM objects
type NetworkService struct {
    Link      string `json:"link"`
    Labels    string `json:"labels"`
    IPAddress string `json:"ipAddress"`
    Route     string `json:"route"`
}

// GetEndpoint parses a stream of bytes to a Endpoint struct
func GetEndpoint(endpointsFile []byte) (Endpoint, error) {
    var endpointsConfig Endpoint
    if err := json.Unmarshal(endpointsFile, &endpointsConfig); err != nil {
        return endpointsConfig, err
    }

    return endpointsConfig, nil
}
func main() {
    c := tools.NewOSSignalChannel()

    endpointsFile, err := os.Open("/etc/nsminfo/endpoints")
    byteValue, _ := ioutil.ReadAll(endpointsFile)
    endpointsConfig, err := GetEndpoint(byteValue)

    registrations := []endpoint.Registration{}
    switchEndpoint := NewSwitchEndpoint("link")
    configuration := common.FromEnv()
    configuration.EndpointNetworkService = endpointsConfig.Name
    for _, config := range endpointsConfig.NetworkServices {
        service := common.FromEnv()
        service.EndpointNetworkService = endpointsConfig.Name
        service.EndpointLabels = config.Labels
        service.IPAddress = config.IPAddress

        endpoints := []networkservice.NetworkServiceServer{
            endpoint.NewConnectionEndpoint(service),
            endpoint.NewIpamEndpoint(service),
            endpoint.NewCustomFuncEndpoint("podName", endpoint.CreatePodNameMutator()),
        }

        if config.Route != "" {
            routeAddr := endpoint.CreateRouteMutator([]string{config.Route})
            endpoints = append(endpoints, endpoint.NewCustomFuncEndpoint("route", routeAddr))
        }

        switchEndpoint.Childs[config.Link] = endpoint.NewCompositeEndpoint(endpoints...)
        registrations = append(registrations, endpoint.MakeRegistration(service))
    }

    nsEndpoint, err := endpoint.NewNSMEndpoint(
        context.Background(),
        configuration,
        endpoint.NewCompositeEndpoint(
            endpoint.NewMonitorEndpoint(configuration),
            switchEndpoint,
        ),
        endpoint.WithRegistrations(registrations...),
    )
    defer nsEndpoint.Delete()
    nsEndpoint.Start()

    // Capture signals to cleanup before exiting
    <-c
}

[edwarnicke]

So... perhaps I'm missing something... but this seems a lot more complicated than our previous annotation scheme...

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has been inactive for 37 days. Feel free to reopen it if you feel it has been closed in error.

[electrocucaracha]

@edwarnicke I have created this Mutating Admission Webhook which injects the NSM sidecar base on the annotations. This change allows to support multiple cni-multiplexers:

apiVersion: v1
kind: Pod 
metadata:
  name: example
  annotations:
    danm.k8s.io/interfaces: |
      [
        {"clusterNetwork":"default"},
        {"clusterNetwork":"lte-s5u"},
        {"clusterNetwork":"lte-s5c"},
        {"clusterNetwork":"lte-sgi"}
      ]
    k8s.v1.cni.cncf.io/networks: |
      [
        {"name": "lte-s5u", "interface": "s5u1"},
        {"name": "lte-s5c", "interface": "s5c2"},
        {"name": "lte-sgi", "interface": "sgi3"}
      ]
    ns.networkservicemesh.io/endpoints: |
      {
        "name": "lte-network",
        "networkServices": [
          {"link": "sgi", "labels": "app=http-server-sgi", "ipaddress": "10.0.1.0/24", "route": "10.0.3.0/24"},
          {"link": "s5u", "labels": "app=pgw-s5u", "ipaddress": "172.25.0.0/24"},
          {"link": "s5c", "labels": "app=pgw-s5c", "ipaddress": "172.25.1.0/24"}
        ]
      }
spec:
  containers:
    - image: busybox:stable
      name: instance
      command:
        - sleep
      args:
        - infinity

[edwarnicke]

@electrocucaracha This is interesting... but we typically keep our annotations far simpler, basically of a form

ns.networkservicemesh.io/endpoints: ${nsmurl}

we don't generally specify IP addresses or routes on the NSC side... the NSC has no sayin that, it all comes down from the NSE.

I'm curious to understand more about your thought process though :)

[electrocucaracha]

@edwarnicke those annotations are for the Endpoint, clients use the traditional ns.networkservicemesh.io annotation

apiVersion: v1
kind: Pod
metadata:
  name: client
  annotations:
    ns.networkservicemesh.io: lte-network/s5u3?link=s5u,lte-network?link=sgi
spec:
  containers:
    - image: busybox:stable
      name: instance
      command:
        - sleep
      args:
        - infinity