search

intel / gardener-extension-cri-resmgr

Gardener extension controller for the https://github.com/intel/cri-resource-manager container runtime proxy
Apache License 2.0
9 stars 5 forks source link

readme

CRI-Resource-Manager extension for Gardener*

Go build and test

Warning
Code in master branch is "work in progress" and not meant to run in production environment!

Introduction

This Gardener extension will deploy and manage lifecycle of CRI-Resource-Manager in "shoot" clusters worker nodes deployed by Gardener. CRI-Resource-Manager is a proxy between kubectl and containerd to allow more sophisticated node resource management.

Requirements

Features

Missing/TODO:

How CRI-Resource-Manager is deployed

There are two charts:

Configuring CRI-resource-manager

There are multiple options to pass configuration file to CRI-Resource-Manager systemd unit:

apiVersion: core.gardener.cloud/v1beta1
kind: ControllerDeployment
metadata:
  name: cri-resmgr-extension
type: helm
providerConfig:
  chart: H4sI....
  values:
    configs:
      default: |
        ... SOME DEFAULT CONFIG ...

in above example "default" config will be used for all the shoots,

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: local
spec:
  ...
  extensions:
  - type: cri-resmgr-extension
    providerConfig:
      configs:
        fallback: |+
        ... SOME DEFAULT CONFIG ...
        default: |+
        ... SOME DEFAULT CONFIG ...

configs specified in Shoot definition will override configs provided by ControllerDeployment.

Each key of configs will generate ConfigMap in the kube-system namespace with following name:

cri-resmgr-config.TYPE e.g. cri-resmgr-config.default

types default and fallback have special meaning:

Note
Configurations from different levels are not merged. They can merge only in case of configuration propagated by cri-resmgr-agent but when manually CRI-resource-manager is restarted the configuration from agent is first pulled as a whole and not merged with cached/previous/fallback state. Always provide full configuration for every type!

More about priorities and type of config can be found here:

Getting started

I. Deploying to Gardener.

1. Prepare images.

Build and publish docker images for extension, installation and cri-resmgr-agent with your own registry.

For example lets assume that we want to use v2.isvimgreg.com as registry with specific tag called mybranch:

make REGISTRY=v2.isvimgreg.com/ TAG=mybranch build-images push-images

Then overwrite images in ControllerDeployment and provider "fallback" config to be used before CRI-resource-manager syncs with cri-resmgr-agent:

apiVersion: core.gardener.cloud/v1beta1
kind: ControllerDeployment
metadata:
  name: cri-resmgr-extension
type: helm
providerConfig:
  chart: H4sIAAAAAAAA....
  values:
   image:
     repository: v2.isvimgreg.com/gardener-extension-cri-resmgr
     tag: mybranch
     pullPolicy: Always
   imageVectorOverwrite: |
     images:
     - name: gardener-extension-cri-resmgr-installation
       tag: mybranch
       repository: v2.isvimgreg.com/gardener-extension-cri-resmgr-installation
     - name: gardener-extension-cri-resmgr-agent
       tag: mybranch
       repository: v2.isvimgreg.com/gardener-extension-cri-resmgr-agent
    configs:
      # systemd EnvironmentFile body
      EXTRA_OPTIONS: |
        EXTRA_OPTIONS="--metrics-interval=10s"
      fallback: |
        ### This is default policy from CRI-resource-manage fallback.cfg.sample
        policy:
          Active: topology-aware
          ReservedResources:
            CPU: 750m
        logger:
          Debug: resource-manager,cache,policy,resource-control
        dump:
          Config: off:.*,full:((Create)|(Remove)|(Run)|(Update)|(Start)|(Stop)).*

2. Install extension by creating ControllerRegistration and ControllerDeployment in garden cluster.

kubectl apply -f examples/ctrldeploy-ctrlreg.yaml

note that extension is not enabled globally so you need to include it in shoot spec definition like this:

spec:
  extensions:
  - type: cri-resmgr-extension

II. Deploying locally.

Steps below show how to deploy Gardener with local-provider (kind based) and then deploy and configure gardener-extension-cri-resmgr to a Shoot.

This is fully working example that uses local deployed gardener and shoot created in kind cluster. This is based on https://github.com/gardener/gardener/blob/master/docs/deployment/getting_started_locally.md

Prerequisites

Prepare local kind-based garden cluster

1. Clone the gardener
mkdir -p ~/work/
git clone https://github.com/gardener/gardener ~/work/gardener
cd ~/work/gardener
git checkout v1.86.0
cd -
2. Prepare kind cluster
make -C ~/work/gardener kind-up

kubectl cluster-info --context kind-gardener-local --kubeconfig ~/work/gardener/example/gardener-local/kind/local/kubeconfig
# WARNING!: this overwrites your local kubeconfig
cp ~/work/gardener/example/gardener-local/kind/local/kubeconfig ~/.kube/config

Check that kind cluster is ready:

kubectl get nodes
3. Deploy local gardener
make -C ~/work/gardener/ gardener-up

Check that three gardener charts are installed:

helm list -n garden -a

Deploy cri-resmgr extension

1. Prepare images
make build-images push-images
2. (Optional) Regenerate ctrldeploy-ctrlreg.yaml file:
./hacks/generate-controller-registration.sh
3. Deploy cri-resmgr-extension as Gardener extension using ControllerRegistration/ControllerDeployment
kubectl apply -f ./examples/ctrldeploy-ctrlreg.yaml

By default generated "ControllerRegistration" is not enabled globally, so you need to include this extension in shoot definition. Check this shoot.yaml as example.

Checkout installed objects:

kubectl get controllerregistrations.core.gardener.cloud cri-resmgr-extension
kubectl get controllerdeployments.core.gardener.cloud cri-resmgr-extension

There should be 'cri-resmgr-extension Extension/cri-resmgr-extension' resources visible alongside cri-resmgr-extension deployment.

Remember that "controller installation" should not be yet available - there is not shoot cluster deployed yet and extension is disabled by default (TODO, BUG?!?!)?!?!? - is visible

kubectl get controllerinstallation.core.gardener.cloud

should no return "cri-resmgr extension" installation.

4. Deploy shoot "local" cluster.

Build an image with extension and upload to local kind cluster

make build-images push-images

by default localhost:5001 registry is used (check 'Build and publish docker images' section for more info)

Create shoot:

kubectl apply -f examples/shoot.yaml

Check that shoot cluster is ready:

kubectl get shoots -n garden-local --watch -o wide

In our shoot example, the extensions is also disabled by default and need to be enabled after shoot is created with following command:

kubectl patch shoot local -n garden-local -p '{"spec":{"extensions": [ {"type": "cri-resmgr-extension", "disabled": false} ] } }'
5. Verify that ManagedResources are properly installed in shoot 'garden' (seed class) and 'shoot--local--local' namespace
kubectl get managedresource -n garden | grep cri-resmgr-extension
kubectl get managedresource -n shoot--local--local | grep extension-runtime-cri-resmgr
6. Check shoot cluster node is ready

First get credentials to access shoot cluster:

kubectl create -f kubeconfig-request.json --raw /apis/core.gardener.cloud/v1beta1/namespaces/garden-local/shoots/local/adminkubeconfig | jq -r ".status.kubeconfig" | base64 -d > /tmp/kubeconfig-shoot-local.yaml

Please follow this guide to get access to local shoot cluster.

... and check status of the node/pods:

kubectl --kubeconfig=/tmp/kubeconfig-shoot-local.yaml get nodes
kubectl --kubeconfig=/tmp/kubeconfig-shoot-local.yaml get pods -A
7. Check CRI-resource-manager is installed properly as proxy
kubectl exec -n shoot--local--local `kubectl get pod -n shoot--local--local --no-headers | grep machine-shoot | awk '{print $1}'` -- systemctl status cri-resource-manager kubelet -n 0

We should observe that:

  1. cri-resource-manager is installed as service
● cri-resource-manager.service - A CRI proxy with (hardware) resource aware container placement policies.
     Loaded: loaded (/etc/systemd/system/cri-resource-manager.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-06-29 20:21:33 UTC; 2min 14s ago
       Docs: https://github.com/intel/cri-resource-manager
   Main PID: 10937 (cri-resmgr)
      Tasks: 6 (limit: 69411)
     Memory: 19.5M
     CGroup: /docker/c4cf6958c7757cd0d66aed793a7d19f6364d936a9761c7f49c33894a65caab66/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-pod9c7ece81_7d98_4884_b214_8f2389308241.slice/cri-containerd-70b98bdb42eec15c4df4cd520d79105b7420f7be4997f4ae8f4b17503d006df4.scope/system.slice/cri-resource-manager.service
             └─10937 /opt/intel/bin/cri-resmgr --fallback-config /etc/cri-resmgr/fallback.cfg
  1. kubelet was reconfigured to use cri-resmgr.sock as its container-runtime-endpoint command line option:
● kubelet.service - kubelet daemon
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-06-29 20:21:39 UTC; 2min 9s ago
       Docs: https://kubernetes.io/docs/admin/kubelet
    Process: 11138 ExecStartPre=/var/lib/kubelet/copy-kubernetes-binary.sh kubelet (code=exited, status=0/SUCCESS)
   Main PID: 11141 (kubelet)
      Tasks: 13 (limit: 69411)
     Memory: 57.1M
     CGroup: /docker/c4cf6958c7757cd0d66aed793a7d19f6364d936a9761c7f49c33894a65caab66/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-pod9c7ece81_7d98_4884_b214_8f2389308241.slice/cri-containerd-70b98bdb42eec15c4df4cd520d79105b7420f7be4997f4ae8f4b17503d006df4.scope/system.slice/kubelet.service
             └─11141 /opt/bin/kubelet 
             --bootstrap-kubeconfig=/var/lib/kubelet/kubeconfig-bootstrap 
             --config=/var/lib/kubelet/config/kubelet 
             --kubeconfig=/var/lib/kubelet/kubeconfig-real 
             --node-labels=worker.gardener.cloud/kubernetes-version=1.24.0 
             --container-runtime=remote 
             --v=2 
             --container-runtime-endpoint=/var/run/cri-resmgr/cri-resmgr.sock # <---
8. Uninstalling (disabling) cri-resource-manager extension

You can disable "cri-resmgr extension" in existing shoot to uninstall cri-resource-manager from shoot worker node like this:

kubectl patch shoot local -n garden-local -p '{"spec":{"extensions": [ {"type": "cri-resmgr-extension", "disabled": true} ] } }'

now after executing this:

kubectl exec -n shoot--local--local `kubectl get pod -n shoot--local--local --no-headers G machine-shoot | awk '{print $1}'` -- systemctl status cri-resource-manager kubelet -n 0

there should not be "cri-resource-manager" systemd service unit anymore 

Unit cri-resource-manager.service could not be found.

... and "kubelet" should connect to containerd.sock as it was by default.

● kubelet.service - kubelet daemon
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-06-29 20:49:31 UTC; 1min 19s ago
       Docs: https://kubernetes.io/docs/admin/kubelet
    Process: 32351 ExecStartPre=/var/lib/kubelet/copy-kubernetes-binary.sh kubelet (code=exited, status=0/SUCCESS)
   Main PID: 32354 (kubelet)
      Tasks: 12 (limit: 69411)
     Memory: 64.7M
     CGroup: /docker/c4cf6958c7757cd0d66aed793a7d19f6364d936a9761c7f49c33894a65caab66/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-pod9c7ece81_7d98_4884_b214_8f2389308241.slice/cri-containerd-70b98bdb42eec15c4df4cd520d79105b7420f7be4997f4ae8f4b17503d006df4.scope/system.slice/kubelet.service
             └─32354 /opt/bin/kubelet 
             --bootstrap-kubeconfig=/var/lib/kubelet/kubeconfig-bootstrap 
             --config=/var/lib/kubelet/config/kubelet 
             --kubeconfig=/var/lib/kubelet/kubeconfig-real 
             --node-labels=worker.gardener.cloud/kubernetes-version=1.24.0 
             --container-runtime=remote 
             --v=2 
             --container-runtime-endpoint=unix:///run/containerd/containerd.sock  # <---

Troubleshooting and debugging issues

Deployment and propagation of changes

In garden virtual and seed clusters you should monitor and watch over following process:

  1. ControllerRegistration/cri-resmgr-extension results in ControllerInstallation/cri-resmgr-extension (reconciled by gardener-controller-manager),
  2. ControllerInstallation/cri-resmgr-extension Valid/Installed/Healthy conditions are true - those conditions are based on ManagedResource/cri-resmgr-extension-* (synced by gardenlet/controllerinstallation_care controller),
  3. ManagedResource/cri-resmgr-extension-* of type=seed results in deploying Pod/cri-resmgr-extension in seed cluster (reconciled by gardener-resource-manager),
  4. (Optionally) if configs are defined, the additional ConfigMap/gardener-extension-cri-resmgr-configs should be created in extension-cri-resmgr-extension-XXXX namespace with keys representing different kind of configs (e.g. fallback/default),
  5. At the end of shoot reconciliation process, gardenlet creates Extension/cri-resource-manager watched by Pod/cri-resmgr-extension which initiates deployment of CRI-resource-manager to shoot worker nodes,
  6. CRI-Resource-Manager deployment is implemented by creating ManagedResource/extension-runtime-cri-resmgr for shoot that consists of installation chart (agent and installation DaemonSets and optional ConfigMap/cri-resmgr-config.TYPE configs),
  7. After gardener-resource-manager reconciles ManagedResources/extension-runtime-cri-resmgr for shoot - conditions of those newly created resources (2 x DaemonsSets with liveness probes) are monitored by Pod/cri-resmgr-extension and reported in Extension/cri-resmgr-extension status( SystemComponentsHealthy condition),
  8. Condition of Extensions/cri-resmgr-extension is monitored by gardenlet/shoot_care controller and affects finally Shoot.status.conditions

In a shoot, we expect two DaemonSets to be deployed:

Logging in terminal

To debug issues you can consult logs of following components (commands are for local kind-based Gardener deployment):

  1. Seed context: Pod/cri-resmgr-extension - can be accessed by: 
    kubectl logs -n extension-cri-resmgr-extension-XXXXX deploy/gardener-extension-cri-resmgr
  2. Shoot context: DaemonSet/installation: 
    kubectl logs -n kube-system ds/cri-resmgr-installation
  3. Shoot context: Pod/cri-resmgr-agent: 
    kubectl logs -n kube-system ds/cri-resmgr-agent

  4. Seed context: cri-resource-manager systemd unit:

    # Enable port-forward to shoot's loki: 
kubectl port-forward -n shoot--local--local service/loki 3100:3100 &

# Then query for metrics from loki
logcli query --org-id="operator" '{job="systemd-combine-journal"} | unpack | unit="cri-resource-manager.service"'

    Loki client logcli can be found here. Note you can specify additional label nodename="machine-shoot--local--local-local-XXXXX-XXXXX" to get logs from specific node.

    In case of issues it is worth to consult: kubelet and containerd systemd services logs following commands:

    # For kubelet
logcli query --org-id="operator" '{unit="kubelet.service"}'
# For containerd
logcli query --org-id="operator" '{unit="containerd.service"}'

    Logging in Grafana

Logs for cri-resmgr-extension are available in "Garden" Grafana in garden namespace. Logs for cri-resmgr-installation, cri-resmgr-agent and cri-resource-manager daemon are in dedicated Grafana for shoot (deployed in a seed in shoot namespace e.g. shoot--local--local).

Note that example commands below assume running local setup with kind based Gardener.

cri-resmgr-extension logs

You can check the logs in Grafana - svc/grafana namespace garden.

Expose it first with:

kubectl port-forward -n garden svc/grafana 3000:3000

Then go to http://127.0.0.1:3000 and Dashboards/Extensions or (generic Pod Logs) and choose "gardener-extension-cri-resmgr"

Grafana dashboard:

or Explore with Loki expression 

{container_name="gardener-extension-cri-resmgr"}

to filter only errors use the following query:

{container_name="gardener-extension-cri-resmgr"}|json|log_level="error"|line_format "{{.log_msg}} {{.log_error}}"

cri-resmgr-installation and cri-resmgr-agent logs

You can check the logs in Grafana for operators "grafana-operators" in Shoot namespace.

Expose it first with (for shoot local)

kubectl port-forward -n shoot--local--local svc/grafana-operators 3001:3000

Then go to http://127.0.0.1:3001 and Kubernetes DaemonSets and chose "cri-resmgr-agent" or "cri-resmgr-installation" then drill down to specific Pod

Bug in kind based local gardener Grafana operator Loki configuration
For local kind-based setup, HTTP header is missing: X-Scope-OrgID: operator. To fix it do the following, first sign in with admin/admin and then:

Or use Explorer and enter following Loki queries for installation logs:

{pod_name=~"cri-resmgr-installation-.*", container_name="installation"}

and for agent:

{pod_name=~"cri-resmgr-agent-.*"}
Example screenshot cri-resmgr-installation pod:

Example screenshot of cri-resmgr-agent pod:

cri-resource-manager daemon logs

You can check the logs in Grafana - svc/grafana-operators in seed namespace.

Loki expression 

{origin="systemd-journal",job="systemd-combine-journal"} | json | unit="cri-resource-manager.service" | unpack

with skip_headers=on in cri-resource-manager you can even parse more output from cri-resource-manager

{origin="systemd-journal",job="systemd-combine-journal"} | json | unit="cri-resource-manager.service" | unpack | label_format _entry="" | regexp "(?P<level>.): \\[ *(?P<module>.*?) *\\] (?P<msg>.*)" | line_format "{{.msg}}" | label_format msg=""

and e.g. apply the filter for level="E"

{origin="systemd-journal",job="systemd-combine-journal"} | json | unit="cri-resource-manager.service" | unpack | label_format _entry="" | regexp "(?P<level>.): \\[ *(?P<module>.*?) *\\] (?P<msg>.*)" | line_format "{{.msg}}" | label_format msg="" | level="E"

and e.g. apply the filter for module="E"

{origin="systemd-journal",job="systemd-combine-journal"} | json | unit="cri-resource-manager.service" | unpack | label_format _entry="" | regexp "(?P<level>.): \\[ *(?P<module>.*?) *\\] (?P<msg>.*)" | line_format "{{.msg}}" | label_format msg="" | module="policy"

I: [resource-manager] successfully switched to new configuration

Example screenshot of cri-resource-manager daemon:

Running e2e tests.

Assuming having gardner cloned in ~/work/gardener

and already set /etc/hosts properly with following entries:

127.0.0.1 api.e2e-default.local.external.local.gardener.cloud
127.0.0.1 api.e2e-default.local.internal.local.gardener.cloud

then:

make -C ~/work/gardener kind-up
cp ~/work/gardener/example/gardener-local/kind/local/kubeconfig ~/.kube/config
make -C ~/work/gardener gardener-up
kubectl apply -f ./examples/ctrldeploy-ctrlreg.yaml
make e2e-test KUBECONFIG=$HOME/.kube/config

Additional options available provided by test framework: when running manually with ginkgo:

KUBECONFIG=$HOME/.kube/config ginkgo -v --progress --label-filter enable ./test/e2e/... -- -verbose debug -disable-dump -project-namespace testbroken
  -disable-dump
        Disable the state dump if a test fails
  -existing-shoot-name string
        Name of an existing shoot to use instead of creating a new one.
  -kubecfg string
        the path to the kubeconfig  of the garden cluster that will be used for integration tests
  -project-namespace string
        specify the gardener project namespace to run tests
  -skip-accessing-shoot
        if set to true then the test does not try to access the shoot via its kubeconfig
  -verbose string
        verbosity level (defaults to info)

access to e2e shoot with k9s example:

k9s --kubeconfig <(kubectl view-secret -n garden-local e2e-default.kubeconfig kubeconfig)