No actions being taken by draino for the condition Ready=Unknown

[prabhatnagpal]

Please help me out because I am not able to make Draino work. Even if the node is in unknown state it doesn't drain nodes. I have used kops to spin up the cluster with 1 master and 2 nodes. I make the node reach unknown state by using this script and start on a node:-

#!/bin/bash
for (( ; ; ))
do
echo "Press CTRL+C to stop............................................."
nohup ./run.sh &
done

My draino.yaml is this:-

---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels: {component: draino}
  name: draino
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels: {component: draino}
  name: draino
rules:
- apiGroups: ['']
  resources: [events]
  verbs: [create, patch, update]
- apiGroups: ['']
  resources: [nodes]
  verbs: [get, watch, list, update]
- apiGroups: ['']
  resources: [nodes/status]
  verbs: [patch]
- apiGroups: ['']
  resources: [pods]
  verbs: [get, watch, list]
- apiGroups: ['']
  resources: [pods/eviction]
  verbs: [create]
- apiGroups: [extensions]
  resources: [daemonsets]
  verbs: [get, watch, list]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels: {component: draino}
  name: draino
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: draino}
subjects:
- {kind: ServiceAccount, name: draino, namespace: kube-system}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels: {component: draino}
  name: draino
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels: {component: draino}
  template:
    metadata:
      labels: {component: draino}
      name: draino
      namespace: kube-system
    spec:
      containers:
      - name: draino
        image: planetlabs/draino:5e07e93
        command:
        - /draino
        - Ready=Unknown
        livenessProbe:
          httpGet: {path: /healthz, port: 10002}
          initialDelaySeconds: 30
      serviceAccountName: draino

My node-problem-detector.yaml is this:-

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-problem-detector
  namespace: kube-system
  labels:
    app: node-problem-detector
spec:
  selector:
    matchLabels:
      app: node-problem-detector
  template:
    metadata:
      labels:
        app: node-problem-detector
    spec:
      containers:
      - name: node-problem-detector
        command:
        - /node-problem-detector
        - --logtostderr
        - --system-log-monitors=/config/kernel-monitor.json,/config/docker-monitor.json
        image: k8s.gcr.io/node-problem-detector:v0.6.3
        resources:
          limits:
            cpu: 10m
            memory: 80Mi
          requests:
            cpu: 10m
            memory: 80Mi
        imagePullPolicy: Always
        securityContext:
          privileged: true
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        volumeMounts:
        - name: log
          mountPath: /var/log
          readOnly: true
        - name: kmsg
          mountPath: /dev/kmsg
          readOnly: true
        # Make sure node problem detector is in the same timezone
        # with the host.
        - name: localtime
          mountPath: /etc/localtime
          readOnly: true
        - name: config
          mountPath: /config
          readOnly: true
      volumes:
      - name: log
        # Config `log` to your system log directory
        hostPath:
          path: /var/log/
      - name: kmsg
        hostPath:
          path: /dev/kmsg
      - name: localtime
        hostPath:
          path: /etc/localtime
      - name: config
        configMap:
          name: node-problem-detector-config
          items:
          - key: kernel-monitor.json
            path: kernel-monitor.json
          - key: docker-monitor.json
            path: docker-monitor.json

And finally the node-problem-detector-config.yaml is this:-

apiVersion: v1
data:
  kernel-monitor.json: |
    {
        "plugin": "kmsg",
        "logPath": "/dev/kmsg",
        "lookback": "5m",
        "bufferSize": 10,
        "source": "kernel-monitor",
        "conditions": [
            {
                "type": "KernelDeadlock",
                "reason": "KernelHasNoDeadlock",
                "message": "kernel has no deadlock"
            },
            {
                "type": "ReadonlyFilesystem",
                "reason": "FilesystemIsReadOnly",
                "message": "Filesystem is read-only"
            },
            {
                "type": "Ready",
                "reason": "NodeStatusUnknown",
                "message": "Kubelet stopped posting node status"
            }

        ],
        "rules": [

            {
                "type": "temporary",
                "reason": "NodeStatusUnknown",
                "pattern": "Kubelet stopped posting node status"
            },
            {
                "type": "temporary",
                "reason": "OOMKilling",
                "pattern": "Kill process \\d+ (.+) score \\d+ or sacrifice child\\nKilled process \\d+ (.+) total-vm:\\d+kB, anon-rss:\\d+kB, file-rss:\\d+kB.*"
            },
            {
                "type": "temporary",
                "reason": "TaskHung",
                "pattern": "task \\S+:\\w+ blocked for more than \\w+ seconds\\."
            },
            {
                "type": "temporary",
                "reason": "UnregisterNetDevice",
                "pattern": "unregister_netdevice: waiting for \\w+ to become free. Usage count = \\d+"
            },
            {
                "type": "temporary",
                "reason": "KernelOops",
                "pattern": "BUG: unable to handle kernel NULL pointer dereference at .*"
            },
            {
                "type": "temporary",
                "reason": "KernelOops",
                "pattern": "divide error: 0000 \\[#\\d+\\] SMP"
            },
            {
                "type": "permanent",
                "condition": "KernelDeadlock",
                "reason": "AUFSUmountHung",
                "pattern": "task umount\\.aufs:\\w+ blocked for more than \\w+ seconds\\."
            },
            {
                "type": "permanent",
                "condition": "KernelDeadlock",
                "reason": "DockerHung",
                "pattern": "task docker:\\w+ blocked for more than \\w+ seconds\\."
            },
            {
                "type": "permanent",
                "condition": "ReadonlyFilesystem",
                "reason": "FilesystemIsReadOnly",
                "pattern": "Remounting filesystem read-only"
            }
        ]
    }
  docker-monitor.json: |
    {
        "plugin": "journald",
        "pluginConfig": {
            "source": "dockerd"
        },
        "logPath": "/var/log/journal",
        "lookback": "5m",
        "bufferSize": 10,
        "source": "docker-monitor",
        "conditions": [],
        "rules": [
            {
                "type": "temporary",
                "reason": "CorruptDockerImage",
                "pattern": "Error trying v2 registry: failed to register layer: rename /var/lib/docker/image/(.+) /var/lib/docker/image/(.+): directory not empty.*"
            }
        ]
    }
kind: ConfigMap
metadata:
  name: node-problem-detector-config
  namespace: kube-system

[dliao-tyro-admin]

I came across something similar. Make sure to upgrade your draino and the helm chart to the newer version.

Also, for the clli args that you have supplied,Ready=Unknown. This is incorrect in the newer version and should be e.g. Ready=Unknown,10m. See https://github.com/planetlabs/draino/issues/33 for more info.

Note: I'm running release b788331 and this is working as expected.

[dbenque]

@dliao-tyro Did you have some complete draino success (cordon+drain) for such configuration Ready=Unknown,10m ? https://github.com/planetlabs/draino/issues/33#issuecomment-602466215

[dliao-tyro-admin]

I ran the following experiment to demonstrate the behaviour.

Stopping the kubelet service on one of the nodes. This will report back a NotReady status with the following:

Name:               ip-12-18-92-92.ap-southeast-2.compute.internal
Roles:              workload
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=m5.large
                    beta.kubernetes.io/os=linux
                    failure-domain.beta.kubernetes.io/region=ap-southeast-2
                    failure-domain.beta.kubernetes.io/zone=ap-southeast-2b
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-12-18-92-92.ap-southeast-2.compute.internal
                    kubernetes.io/lifecycle=spot
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/workload=true
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Mon, 23 Mar 2020 17:28:25 +1100
Taints:             node.kubernetes.io/unreachable:NoExecute
                    node.kubernetes.io/unreachable:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  ip-12-18-92-92.ap-southeast-2.compute.internal
  AcquireTime:     <unset>
  RenewTime:       Tue, 24 Mar 2020 08:17:20 +1100
Conditions:
  Type                        Status    LastHeartbeatTime                 LastTransitionTime                Reason                      Message
  ----                        ------    -----------------                 ------------------                ------                      -------
  KernelDeadlock              False     Tue, 24 Mar 2020 08:26:05 +1100   Mon, 23 Mar 2020 18:42:05 +1100   KernelHasNoDeadlock         kernel has no deadlock
  ReadonlyFilesystem          False     Tue, 24 Mar 2020 08:26:05 +1100   Mon, 23 Mar 2020 18:42:05 +1100   FilesystemIsNotReadOnly     Filesystem is not read-only
  CannotKillContainer         False     Tue, 24 Mar 2020 08:26:05 +1100   Mon, 23 Mar 2020 18:42:05 +1100   NoCannotKillContainer       System can stop containers
  FrequentKubeletRestart      False     Tue, 24 Mar 2020 08:26:05 +1100   Mon, 23 Mar 2020 18:42:07 +1100   FrequentKubeletRestart      kubelet is functioning properly
  FrequentDockerRestart       False     Tue, 24 Mar 2020 08:26:05 +1100   Mon, 23 Mar 2020 18:42:09 +1100   FrequentDockerRestart       docker is functioning properly
  FrequentContainerdRestart   False     Tue, 24 Mar 2020 08:26:05 +1100   Mon, 23 Mar 2020 18:42:11 +1100   FrequentContainerdRestart   containerd is functioning properly
  FrequentAwslogsdRestart     False     Tue, 24 Mar 2020 08:26:05 +1100   Mon, 23 Mar 2020 18:42:13 +1100   FrequentAwslogsdRestart     awslogsd is functioning properly
  MemoryPressure              Unknown   Tue, 24 Mar 2020 08:17:06 +1100   Tue, 24 Mar 2020 08:18:04 +1100   NodeStatusUnknown           Kubelet stopped posting node status.
  DiskPressure                Unknown   Tue, 24 Mar 2020 08:17:06 +1100   Tue, 24 Mar 2020 08:18:04 +1100   NodeStatusUnknown           Kubelet stopped posting node status.
  PIDPressure                 Unknown   Tue, 24 Mar 2020 08:17:06 +1100   Tue, 24 Mar 2020 08:18:04 +1100   NodeStatusUnknown           Kubelet stopped posting node status.
  Ready                       Unknown   Tue, 24 Mar 2020 08:17:06 +1100   Tue, 24 Mar 2020 08:18:04 +1100   NodeStatusUnknown           Kubelet stopped posting node status.

10 mins later, the draino logs will output:

{"level":"info","ts":1584998887.2430277,"caller":"kubernetes/eventhandler.go:139","msg":"Cordoned","node":"ip-12-18-92-92.ap-southeast-2.compute.internal"}
{"level":"info","ts":1584998887.2431808,"caller":"kubernetes/eventhandler.go:148","msg":"Scheduled drain","node":"ip-12-18-92-92.ap-southeast-2.compute.internal","after":1584999148.9898398}

I went back to start the kubelet service on the same node and the draino logs the following output:

{"level":"info","ts":1584998567.4310849,"caller":"draino/draino.go:172","msg":"node watcher is running"}
{"level":"info","ts":1584998887.2430277,"caller":"kubernetes/eventhandler.go:139","msg":"Cordoned","node":"ip-12-18-92-92.ap-southeast-2.compute.internal"}
{"level":"info","ts":1584998887.2431808,"caller":"kubernetes/eventhandler.go:148","msg":"Scheduled drain","node":"ip-10-18-92-92.ap-southeast-2.compute.internal","after":1584999148.9898398}
{"level":"info","ts":1584999250.2038314,"caller":"kubernetes/eventhandler.go:161","msg":"Drained","node":"ip-12-18-92-92.ap-southeast-2.compute.internal"}

So overall, this does seem to be working as expected with the following configuration in the container spec.

      - command:
        - /draino
        - CannotKillContainer
        - DiskPressure
        - FrequentContainerdRestart
        - FrequentDockerRestart
        - FrequentKubeletRestart
        - KernelDeadlock
        - NetworkUnavailable
        - OutOfDisk
        - PIDPressure
        - ReadonlyFilesystem
        - Ready=Unknown,10m

Let me know if you need more information.

[dbenque]

@dliao-tyro ok, but in order to have the drain completed you had to restart the kubelet, which will make the node responsive again (transition from unknown (aka notReady) to ready state). I was expecting the solution to work on top of a node that remains notReady. IMO it can due to the way the eviction API works, waiting for kubelet status to confirm pod eviction.

[dliao-tyro-admin]

@dbenque, The example above was just to demonstrate draino would attempt to drain the node if Ready=Unknown condition was met; not sure if there was a quick/simple way to set that particular node condition.

For the drain to be successful, I think there needs to be an expectation that kubelet remains operational so that it can report on status?

[yogeek]

Hi :)

Same problem here : we wanted to try draino with the Ready=Unknown,2m by stopping the kubelet and impossible for draino to finish the drain.

Does someone has a idea to solve this please ?