search

cablespaghetti / kubeadm-aws

Really cheap Kubernetes cluster on AWS with kubeadm
Other
864 stars 58 forks source link

Kubeadm and kubelet won't install #18

Open vroad opened 5 years ago

vroad commented 5 years ago

terraform apply tfplan run fine, but kubectl is not available in master node. I found later that user data script failed to run because of package dependency issues.

Upgrading kubernetes-version to 1.15.0 from 1.13.4 fixes the issue for kubelet part, but not for kubeadm.

Relevant part from /var/log/cloud-init-output.log:

Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'modules:config' at Tue, 02 Jul 2019 02:54:23 +0000. Up 99.88 seconds.
#!/bin/bash -ve

# Disable pointless daemons
systemctl stop snapd snapd.socket lxcfs snap.amazon-ssm-agent.amazon-ssm-agent
systemctl disable snapd snapd.socket lxcfs snap.amazon-ssm-agent.amazon-ssm-agent
Synchronizing state of lxcfs.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable lxcfs
Removed /etc/systemd/system/sockets.target.wants/snapd.socket.
Removed /etc/systemd/system/multi-user.target.wants/snapd.service.
Removed /etc/systemd/system/multi-user.target.wants/snap.amazon-ssm-agent.amazon-ssm-agent.service.

# Disable swap to make K8S happy
swapoff -a
sed -i '/swap/d' /etc/fstab

# Install K8S, kubeadm and Docker
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
export DEBIAN_FRONTEND=noninteractive
apt-get update
Hit:1 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:3 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:4 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [8570 kB]
Get:6 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8993 B]
Get:7 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic/universe Translation-en [4941 kB]
Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [26.9 kB]
Get:9 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [151 kB]
Get:10 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic/multiverse Translation-en [108 kB]
Get:11 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [676 kB]
Get:12 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [249 kB]
Get:13 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [965 kB]
Get:14 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [287 kB]
Get:15 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [6644 B]
Get:16 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [3556 B]
Get:17 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [2512 B]
Get:18 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic-backports/main Translation-en [1644 B]
Get:19 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [3736 B]
Get:20 http://ap-northeast-1.ec2.archive.ubuntu.com/ubuntu bionic-backports/universe Translation-en [1696 B]
Get:21 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [447 kB]
Get:22 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [156 kB]
Get:23 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [570 kB]
Get:24 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [185 kB]
Get:25 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [4008 B]
Get:26 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2060 B]
Fetched 17.6 MB in 12s (1438 kB/s)
Reading package lists...
apt-get install -y kubelet=1.13.4-00 kubeadm=1.13.4-00 kubectl=1.13.4-00 awscli jq docker.io
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 kubeadm : Depends: kubernetes-cni (= 0.6.0) but 0.7.5-00 is to be installed
 kubelet : Depends: kubernetes-cni (= 0.6.0) but 0.7.5-00 is to be installed
E: Unable to correct problems, you have held broken packages.
Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'modules:final' at Tue, 02 Jul 2019 02:54:26 +0000. Up 103.09 seconds.
2019-07-02 02:54:53,879 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [100]
2019-07-02 02:54:53,920 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2019-07-02 02:54:53,921 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 finished at Tue, 02 Jul 2019 02:54:54 +0000. Datasource DataSourceEc2Local.  Up 131.35 seconds

Even though upgrading k8s version fixes the issue, I got an ~warning~ error on a config file:

Running kubeadm init
your configuration file uses an old API spec: "kubeadm.k8s.io/v1alpha3". Please use kubeadm v1.14 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'modules:final' at Tue, 02 Jul 2019 03:15:56 +0000. Up 84.23 seconds.
2019-07-02 03:20:29,226 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
2019-07-02 03:20:29,282 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2019-07-02 03:20:29,283 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 finished at Tue, 02 Jul 2019 03:20:29 +0000. Datasource DataSourceEc2Local.  Up 357.39 seconds

UPDATE

Running the command above on k8s 1.14.3 yielded following result:

apiVersion: kubeadm.k8s.io/v1beta1
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: ii8to1.dfyg1np0gqybdtq1
  ttl: 0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.0.100.4
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: ip-10-0-100-4.ap-northeast-1.compute.internal
---
apiServer:
  extraArgs:
    cloud-provider: aws
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
controllerManager:
  extraArgs:
    cloud-provider: aws
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.14.3
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
  qps: 5
clusterCIDR: 10.244.0.0/16
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  excludeCIDRs: null
  minSyncPeriod: 0s
  scheduler: ""
  strictARP: false
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: ""
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
resourceContainer: /kube-proxy
udpIdleTimeout: 250ms
winkernel:
  enableDSR: false
  networkName: ""
  sourceVip: ""
cablespaghetti commented 5 years ago

Hi. Thanks for looking at this. To be honest I haven't really worked on this for many months and it looks like even though I locked the versions in apt, some issues have come up over time. Due to family life and general lack of interest I won't be keeping this maintained, but feel free to fork it.