Use default metrics port 10255

[ldx]

Metrics-server is usually part of monitoring stacks for Kubernetes clusters. It queries kubelets via their metrics API, which is usually exposed via port 10255.

We've been running Kip in host network mode, thus moving the metrics endpoint to a different port. However, metrics-server can only set the metrics port on a per cluster basis, thus either monitor only virtual-kubelet instances or regular kubelet instances.

Thus, we need to move the metrics port to 10255.

A possible solution is to run Kip without host network mode. There are two caveats:

• We need kube-proxy in a sidecar, to put in place nodeport and clusterIP redirects.
• Access to the metadata service of the instance. I tested it briefly, and it seems to work on GCE even from inside pods. I haven't tested AWS.

The goal here is to

1. add kube-proxy sidecar,
2. run Kip without host network mode and
3. test node ports and general functionality of Kip.

Example pod spec with kube-proxy added as a sidecar:

  containers:
  - command:
    - /bin/sh
    - -c
    - exec kube-proxy --master=https://34.XX.XX.XXX --kubeconfig=/var/lib/kube-proxy/kubeconfig
      --cluster-cidr=10.25.0.0/16 --resource-container="" --oom-score-adj=-998
      --v=2
    image: gke.gcr.io/kube-proxy:v1.14.10-gke.36
    imagePullPolicy: IfNotPresent
    name: kube-proxy
    resources:
      requests:
        cpu: 100m
    securityContext:
      privileged: true
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: etc-ssl-certs
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-ca-certs
      readOnly: true
    - mountPath: /var/lib/kube-proxy/kubeconfig
      name: kube-proxy-kubeconfig
    - mountPath: /run/xtables.lock
      name: xtables-lock
    - mountPath: /lib/modules
      name: lib-modules
      readOnly: true
  - command:
    - /virtual-kubelet
    - --provider
    - kip
    - --provider-config
    - /etc/virtual-kubelet/provider.yaml
    - --network-agent-secret
    - kube-system/vk-network-agent
    - --disable-taint
    - --klog.logtostderr
    - --klog.v=5
    - --metrics-addr=:10255
    - --debug-server
    env:
    - name: KUBELET_PORT
      value: "10666"
    - name: APISERVER_CERT_LOCATION
      value: /opt/kip/data/kubelet-pki/virtual-kubelet.crt
    - name: APISERVER_KEY_LOCATION
      value: /opt/kip/data/kubelet-pki/virtual-kubelet.key
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    image: elotl/virtual-kubelet:v0.0.4-4-gc0c246b
    imagePullPolicy: Always
    name: virtual-kubelet
    resources:
      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 100Mi
    securityContext:
      privileged: true
    volumeMounts:
    - mountPath: /opt/kip/data
      name: data
    - mountPath: /etc/virtual-kubelet
      name: provider-yaml
    - mountPath: /run/xtables.lock
      name: xtables-lock
    - mountPath: /lib/modules
      name: lib-modules
      readOnly: true
  dnsPolicy: ClusterFirst
  initContainers:
  - command:
    - bash
    - -c
    - mkdir -p $CERT_DIR && /opt/csr/get-cert.sh
    env:
    - name: NODE_NAME
      value: virtual-kubelet
    - name: CERT_DIR
      value: /data/kubelet-pki
    image: elotl/init-cert:latest
    imagePullPolicy: Always
    name: init-cert
    volumeMounts:
    - mountPath: /data
      name: data
  restartPolicy: Always
  serviceAccount: virtual-kubelet
  serviceAccountName: virtual-kubelet
  tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: provider-data
  - configMap:
      defaultMode: 420
      items:
      - key: cloudinit.yaml
        mode: 384
        path: cloudinit.yaml
      - key: provider.yaml
        mode: 384
        path: provider.yaml
      name: virtual-kubelet-config-799kbh2d6d
    name: provider-yaml
  - hostPath:
      path: /run/xtables.lock
      type: FileOrCreate
    name: xtables-lock
  - hostPath:
      path: /lib/modules
      type: ""
    name: lib-modules
  - hostPath:
      path: /usr/share/ca-certificates
      type: ""
    name: usr-ca-certs
  - hostPath:
      path: /etc/ssl/certs
      type: ""
    name: etc-ssl-certs
  - hostPath:
      path: /var/lib/kube-proxy/kubeconfig
      type: FileOrCreate
    name: kube-proxy-kubeconfig

[ldx]

Fixed via https://github.com/elotl/kip/pull/103