winnode working with kubespray cluster

pablodav commented 6 years ago

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST

I'm trying to get working a windows node with kubespray cluster, and trying to make the required changes for such support.

I have successfully run the join of windows node, but after kubelet is up I got tens or hundred of messages per second with below warning.

I think pod kube-proxy shouldn't be deployed to winnode, but maybe I'm wrong? Also I didn't notice howto exclude winnode for this kube-proxy pod.

Maybe I'm shouldn't use kubeadm for this case? (I'm not using kubeadm to join the win node yet)

Windows instructions are:https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows#network-topology

And node was started with:

./start-kubelet.ps1 -ClusterCidr 10.233.67.0/24

I0403 04:16:35.678193    3388 volume_host.go:219] using default mounter/exec for kubernetes.io/host-path
I0403 04:16:35.783664    3388 reconciler.go:237] Starting operationExecutor.MountVolume for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/b88904b0-36f2-11e8-87e5-080027e8bf1b-xtables-lock") pod "kube-proxy-v659l" (UID: "b88904b0-36f2-11e8-87e5-080027e8bf1b")
I0403 04:16:35.783664    3388 volume_host.go:219] using default mounter/exec for kubernetes.io/host-path
I0403 04:16:35.893057    3388 reconciler.go:237] Starting operationExecutor.MountVolume for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/b88904b0-36f2-11e8-87e5-080027e8bf1b-xtables-lock") pod "kube-proxy-v659l" (UID: "b88904b0-36f2-11e8-87e5-080027e8bf1b")
I0403 04:16:35.899845    3388 volume_host.go:219] using default mounter/exec for kubernetes.io/host-path
I0403 04:16:36.001422    3388 reconciler.go:237] Starting operationExecutor.MountVolume for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/b88904b0-36f2-11e8-87e5-080027e8bf1b-xtables-lock") pod "kube-proxy-v659l" (UID: "b88904b0-36f2-11e8-87e5-080027e8bf1b")
I0403 04:16:36.008255    3388 volume_host.go:219] using default mounter/exec for kubernetes.io/host-path
I0403 04:16:36.109850    3388 reconciler.go:237] Starting operationExecutor.MountVolume for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/b88904b0-36f2-11e8-87e5-080027e8bf1b-xtables-lock") pod "kube-proxy-v659l" (UID: "b88904b0-36f2-11e8-87e5-080027e8bf1b")
I0403 04:16:36.109850    3388 volume_host.go:219] using default mounter/exec for kubernetes.io/host-path
I0403 04:16:36.215308    3388 reconciler.go:237] Starting operationExecutor.MountVolume for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/b88904b0-36f2-11e8-87e5-080027e8bf1b-xtables-lock") pod "kube-proxy-v659l" (UID: "b88904b0-36f2-11e8-87e5-080027e8bf1b")
I0403 04:16:36.223068    3388 volume_host.go:219] using default mounter/exec for kubernetes.io/host-path
I0403 04:16:36.224042    3388 generic.go:183] GenericPLEG: Relisting
I0403 04:16:36.324651    3388 reconciler.go:237] Starting operationExecutor.MountVolume for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/b88904b0-36f2-11e8-87e5-080027e8bf1b-xtables-lock") pod "kube-proxy-v659l" (UID: "b88904b0-36f2-11e8-87e5-080027e8bf1b")
I0403 04:16:36.324651    3388 volume_host.go:219] using default mounter/exec for kubernetes.io/host-path
I0403 04:16:36.431086    3388 reconciler.go:237] Starting operationExecutor.MountVolume for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/b88904b0-36f2-11e8-87e5-080027e8bf1b-xtables-lock") pod "kube-proxy-v659l" (UID: "b88904b0-36f2-11e8-87e5-080027e8bf1b")
I0403 04:16:36.431086    3388 volume_host.go:219] using default mounter/exec for kubernetes.io/host-path

kubectl describe nodes:

vagrant@k8s-01:~$ kubectl describe node win-ah6pqb767nf 
Name:               win-ah6pqb767nf
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=windows
                    kubernetes.io/hostname=win-ah6pqb767nf
Annotations:        node.alpha.kubernetes.io/ttl=0
                    volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp:  Tue, 03 Apr 2018 02:42:02 +0000
Taints:             <none>
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  OutOfDisk        False   Tue, 03 Apr 2018 04:06:33 +0000   Tue, 03 Apr 2018 02:42:03 +0000   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure   False   Tue, 03 Apr 2018 04:06:33 +0000   Tue, 03 Apr 2018 03:56:57 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Tue, 03 Apr 2018 04:06:33 +0000   Tue, 03 Apr 2018 03:56:57 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  Ready            True    Tue, 03 Apr 2018 04:06:33 +0000   Tue, 03 Apr 2018 03:56:57 +0000   KubeletReady                 kubelet is posting ready status
  PIDPressure      False   Tue, 03 Apr 2018 04:06:33 +0000   Tue, 03 Apr 2018 03:54:33 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
Addresses:
  InternalIP:  172.25.128.1
  Hostname:    win-ah6pqb767nf
Capacity:
 cpu:                2
 ephemeral-storage:  0
 memory:             3145268Ki
 pods:               110
Allocatable:
 cpu:                2
 ephemeral-storage:  0
 memory:             3145268Ki
 pods:               110
System Info:
 Machine ID:      WIN-AH6PQB767NF
 System UUID:     
 Boot ID:         
 Kernel Version:  10.0.16299.309

 OS Image:                   Windows Server Standard
 Operating System:           windows
 Architecture:               amd64
 Container Runtime Version:  docker://17.6.2
 Kubelet Version:            v1.10.0
 Kube-Proxy Version:         v1.10.0
PodCIDR:                     10.233.67.0/24
ExternalID:                  win-ah6pqb767nf
Non-terminated Pods:         (1 in total)
  Namespace                  Name                CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                ------------  ----------  ---------------  -------------
  kube-system                kube-proxy-v659l    0 (0%)        0 (0%)      0 (0%)           0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ------------  ----------  ---------------  -------------
  0 (0%)        0 (0%)      0 (0%)           0 (0%)
Events:
  Type     Reason                        Age                From                         Message
  ----     ------                        ----               ----                         -------
  Warning  FailedToStartNodeHealthcheck  44m (x36 over 1h)  kube-proxy, win-ah6pqb767nf  Failed to start node healthz on 0.0.0.0:10256: listen tcp 0.0.0.0:10256: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.
  Warning  InvalidDiskCapacity           43m (x9 over 1h)   kubelet, win-ah6pqb767nf     invalid capacity 0 on image filesystem
  Warning  ImageGCFailed                 43m (x8 over 1h)   kubelet, win-ah6pqb767nf     invalid capacity 0 on image filesystem
  Normal   Starting                      12m                kubelet, win-ah6pqb767nf     Starting kubelet.
  Normal   NodeHasSufficientDisk         12m                kubelet, win-ah6pqb767nf     Node win-ah6pqb767nf status is now: NodeHasSufficientDisk
  Normal   NodeHasSufficientMemory       12m (x2 over 12m)  kubelet, win-ah6pqb767nf     Node win-ah6pqb767nf status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure         12m (x2 over 12m)  kubelet, win-ah6pqb767nf     Node win-ah6pqb767nf status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID          12m (x2 over 12m)  kubelet, win-ah6pqb767nf     Node win-ah6pqb767nf status is now: NodeHasSufficientPID
  Normal   NodeReady                     12m                kubelet, win-ah6pqb767nf     Node win-ah6pqb767nf status is now: NodeReady
  Normal   Starting                      9m                 kubelet, win-ah6pqb767nf     Starting kubelet.
  Normal   NodeHasSufficientDisk         9m                 kubelet, win-ah6pqb767nf     Node win-ah6pqb767nf status is now: NodeHasSufficientDisk
  Normal   NodeHasSufficientMemory       9m (x2 over 9m)    kubelet, win-ah6pqb767nf     Node win-ah6pqb767nf status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure         9m (x2 over 9m)    kubelet, win-ah6pqb767nf     Node win-ah6pqb767nf status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID          9m                 kubelet, win-ah6pqb767nf     Node win-ah6pqb767nf status is now: NodeHasSufficientPID
  Normal   NodeReady                     9m                 kubelet, win-ah6pqb767nf     Node win-ah6pqb767nf status is now: NodeReady
  Normal   Starting                      9m                 kube-proxy, win-ah6pqb767nf  Starting kube-proxy.
vagrant@k8s-01:~$ kubectl get nodes
NAME              STATUS    ROLES     AGE       VERSION
k8s-01            Ready     master    1h        v1.10.0
k8s-02            Ready     master    1h        v1.10.0
k8s-03            Ready     <none>    1h        v1.10.0
win-ah6pqb767nf   Ready     <none>    1h        v1.10.0

Environment:

Cloud provider or hardware configuration: Testing with vagrant up and a Windows VM.
OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"): default vagrant for cluster with some vars changed:

      ip = "#{$subnet}.#{i+100}"
      host_vars[vm_name] = {
        "ip": ip,
        "bootstrap_os": SUPPORTED_OS[$os][:bootstrap_os],
        "local_release_dir" => $local_release_dir,
        "download_run_once": "False",
        "kube_network_plugin": $network_plugin,
        "kube_node_selector_enabled": "True",
        "kubeadm_enabled": "True",
        "kube_version": "v1.10.0",
        "kubeadm_version": "v1.10.0"
      }

Version of Ansible (ansible --version): ansible 2.5

Kubespray version (commit) (git rev-parse --short HEAD): 1e1c6f54 this is: https://github.com/pablodav/kubespray/commit/1e1c6f5425afacd48a4f0e8a16c545cc74f078d2

Network plugin used: flannel for vagrant cluster.

Copy of your inventory file: default vagrant cluster

Command used to invoke ansible: vagrant up --provider=virtualbox --provision

Output of ansible run:

https://gist.github.com/pablodav/97a5b1845a927f0a33e4fb542feb9d11

pablodav commented 6 years ago

Haven't succeed with kubespray trying to get this working yet, will close it as seems that there is only me trying to do this with kubespray and I will not continue working on it.

lsolberg commented 6 years ago

Did you give up on this? Having windows support in kubespray would be very nice

pablodav commented 6 years ago

Hi @lsolberg I'm will back to kubespray once get windows support working.

I have been trying some other projects and reading hundreds of pages in books and on internet, as I had to learn more about k8s before trying the windows node itself.

Right now I'm working here: https://github.com/pablodav/kubernetes-for-windows as fork from https://github.com/ptylenda/kubernetes-for-windows. If I got it working successfully, probably will back to kubespray.

I don't know yet if main people working in kubespray are really interested in supporting win nodes, but will try in some future.

kubernetes-sigs / kubespray

winnode working with kubespray cluster #2588