Kubernetes 安装

WangShuXian6 commented 5 years ago

安装 docker

安装docker的yum源

yum install -y yum-utils device-mapper-persistent-data lvm2

yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

查看最新的Docker版本：

yum list docker-ce.x86_64  --showduplicates |sort -r

Kubernetes 1.15当前支持的docker版本列表是1.13.1, 17.03, 17.06, 17.09, 18.06, 18.09。这里在各节点安装docker的18.09.6版本。

yum makecache fast

yum install -y --setopt=obsoletes=0 \
  docker-ce-18.09.6-3.el7 

systemctl start docker
systemctl enable docker

禁用SELinux

setenforce 0

这只能暂时禁用它（直到下一次重新启动）。要永久禁用SELinux，请编辑/etc/selinux/config 文件并将 SELINUX=enforcing 更改为SELINUX=permissive

禁用防火墙

systemctl disable firewalld &&systemctl stop firewalld

在Yum仓库中添加Kubernetes

为了让Kubernetes RPM包在yum包管理器中可用，需要参照如下代码清单中的内容，在 /etc/yum.repos.d/文件夹中添加kubernetes.repo文件。

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

国内源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装Kubelet、Kubeadm、Kubectl和Kubernetes-CNI

yum install -y kubelet kubeadm kubectl kubernetes-cni

kubelet——Kubernetes节点代理，它将为你运行一切

kubeadm——一个用于部署多节点Kubernetes集群的工具

kubectl——用于和Kubernetes交互的命令行工具

kubernetes-cni——Kubernetes容器网络接口

手动启动docker和kubelet服务

systemctl enable docker && systemctl start docker
systemctl enable kubelet && systemctl start kubelet

启用`NET.BRIDGE.BRIDGE-NF-CALL-IPTABLES`内核选项

实践过程中，注意到有些东西会禁用bridge-nf-call-iptables内核参数，而这是Kubernetes服务正常运行所必需的

sysctl -w new.bridge.bridge-nf-call-iptables=1

echo "net.bridge.bridge-nf-call-iptables=1" > /etc/sysctl.d/k8s.conf

禁用交换分区

如果启用了交换分区则无法启动Kubelet

swapoff -a && sed -i '/ swap / s/^/#/' /etc/fstab

初始化主节点

kubeadm init

在主节点上运行kubectl

在配置过kubeconfig文件之前，无法使用kubectl与集群进行通信

/etc/kubernetes/admin.conf 文件中展示了必要的一些配置。通过设置KUBECONFIG环境变量让kubectl使用它：

此处应该在admin.conf配置中设置主机公网ip [未测试]

export KUBECONFIG=/etc/kubernetes/admin.conf

写入配置


cd ~
vi .bashrc
// 输入
export KUBECONFIG=/etc/kubernetes/admin.conf
// 保存退出

// 加载新配置 source ./.bashrc echo "[ -r ~/.bashrc ] && source ~/.bashrc" >> .bash_profile


### 使用kubeadm配置工作节点
```bash

kubeadm join xxx.xxx.xxx.xxx:6443 --token xxxx.xxxxxxxx \
    --discovery-token-ca-cert-hash sha256:xxx

配置容器网络部署Weave Net插件

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

在本地使用集群

scp -P 8888 -r root@xxx.xxx.xxx.xxx:/etc/kubernetes/admin.conf ~/.kube/config2

WangShuXian6 commented 5 years ago

注意事项

`1 node(s) had taints that the pod didn't tolerate`

默认 k8s 不允许往 master 节点装东西，强行设置下允许：

kubectl taint nodes --all node-role.kubernetes.io/master-

`/proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1`

echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

WangShuXian6 commented 5 years ago

安装Helm

https://helm.sh/docs/intro/install/

Helm客户端

通常，我们将Helm客户端安装在能够执行kubectl命令的节点上

$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh

curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash

helm version验证

helm version验证

安装helm的bash命令补全脚本

helm completion bash > .helmrc

echo "source .helmrc" >> .bashrc

source .bashrc

Tab键补全helm子命令和参数了

Tiller服务器

helm init

helm search查看当前可安装的chart

WangShuXian6 commented 5 years ago

使用Helm部署Nginx Ingress

为了便于将集群中的服务暴露到集群外部，需要使用Ingress。

接下来使用Helm将Nginx Ingress部署到Kubernetes上。

Nginx Ingress Controller被部署在Kubernetes的边缘节点上

将node1(192.168.99.11)做为边缘节点，打上Label：

kubectl label node node1 node-role.kubernetes.io/edge=

`stable/nginx-ingress chart`的值文件`ingress-nginx.yaml`

controller:
  replicaCount: 1
  hostNetwork: true
  nodeSelector:
    node-role.kubernetes.io/edge: ''
  affinity:
    podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - nginx-ingress
            - key: component
              operator: In
              values:
              - controller
          topologyKey: kubernetes.io/hostname
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: PreferNoSchedule
defaultBackend:
  nodeSelector:
    node-role.kubernetes.io/edge: ''
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: PreferNoSchedule

nginx ingress controller的副本数replicaCount为1，将被调度到node1这个边缘节点上。这里并没有指定nginx ingress controller service的externalIPs，而是通过hostNetwork: true设置nginx ingress controller使用宿主机网络

helm install stable/nginx-ingress \
-n nginx-ingress \
--namespace ingress-nginx  \
-f ingress-nginx.yaml

kubectl get pod -n ingress-nginx -o wide

如果访问http://192.168.99.11返回default backend，则部署完成。

WangShuXian6 commented 5 years ago

使用Helm部署dashboard

kubernetes-dashboard.yaml：

image:
  repository: k8s.gcr.io/kubernetes-dashboard-amd64
  tag: v1.10.1
ingress:
  enabled: true
  hosts: 
    - k8s.xxx.com
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  tls:
    - secretName: frognew-com-tls-secret
      hosts:
      - k8s.xxx.com
nodeSelector:
    node-role.kubernetes.io/edge: ''
tolerations:
    - key: node-role.kubernetes.io/master
      operator: Exists
      effect: NoSchedule
    - key: node-role.kubernetes.io/master
      operator: Exists
      effect: PreferNoSchedule
rbac:
  clusterAdminRole: true

helm install stable/kubernetes-dashboard \
-n kubernetes-dashboard \
--namespace kube-system  \
-f kubernetes-dashboard.yaml

kubectl -n kube-system get secret | grep kubernetes-dashboard-token
kubernetes-dashboard-token-pkm2s                 kubernetes.io/service-account-token   3      3m7s

kubectl describe -n kube-system secret/kubernetes-dashboard-token-pkm2s

k8s.xxx.com为二级域名，在域名解析服务添加解析即可

重新部署之前先删除

helm del --purge kubernetes-dashboard

WangShuXian6 commented 5 years ago

ingress-nginx

https://github.com/kubernetes/ingress-nginx

https://kubernetes.github.io/ingress-nginx/

它围绕Kubernetes Ingress资源构建，使用ConfigMap存储NGINX配置

NGINX Ingress控制器的工作原理

NGINX配置

目标是组装配置文件（nginx.conf）

在任何配置文件发生更改后重新加载NGINX。

不会在仅影响upstream配置的更改上重新加载Nginx （即在部署应用程序时端点更改）

NGINX模型

所有部署都需要以下强制性命令

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml

裸机安装服务【LB模式配合MetalLB】【NodePort 模式修改type即可】

https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/provider/baremetal/service-nodeport.yaml

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

安装 nginx app 用以测试

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1
        ports:
        - name: http
          containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx

创建 Ingress 访问规则

apiVersion: extensions/v1
kind: Ingress
metadata:
  name: ingress
  annotations: 
    nginx.ingress.kubernetes.io/rewrite-target: /$1
  namespace: default
spec:
  rules:
  - host: nginx.xxx.cn
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 80
        path: /?(.*)

MetalLB

https://metallb.universe.tf/installation/

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.3/manifests/metallb.yaml

或者

helm install --name metallb stable/metallb

查看状态

kubectl get pod -n metallb-system  -o wide

kubectl get daemonset -n metallb-system

Layer 2模式部署

创建config.yaml提供IP池

wget https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/example-layer2-config.yaml

修改ip地址池和集群节点网段相同

kubectl apply -f example-layer2-config.yaml

example-layer2-config.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: config
  namespace: metallb-system
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.0.10-192.168.0.100

注意，地址池为真实物理节点的公网ip，必须为 xx.xxx.xx.xx-xx.xx.xx.xx 形式

若只有一个物理节点，即只有一个公网ip，则地址池为例子：1.1.1.1-1.1.1.1

创建后端应用和服务测试

wget https://raw.githubusercontent.com/google/metallb/master/manifests/tutorial-2.yaml

将 apiVersion 版本修复为 v1 ,测试版已废弃

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1
        ports:
        - name: http
          containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

kubectl apply -f tutorial-2.yaml

查看svc对象

kubectl get service

WangShuXian6 / blog

Kubernetes 安装 #91

Kubernetes 安装

安装 docker

禁用SELinux

禁用防火墙

在Yum仓库中添加Kubernetes

安装Kubelet、Kubeadm、Kubectl和Kubernetes-CNI

手动启动docker和kubelet服务

启用NET.BRIDGE.BRIDGE-NF-CALL-IPTABLES内核选项

禁用交换分区

初始化主节点

在主节点上运行kubectl

配置容器网络 部署Weave Net插件

在本地使用集群

注意事项

1 node(s) had taints that the pod didn't tolerate

/proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1

安装Helm

Helm客户端

通常，我们将Helm客户端安装在能够执行kubectl命令的节点上

helm version验证

安装helm的bash命令补全脚本

Tiller服务器

helm search查看当前可安装的chart

使用Helm部署Nginx Ingress

将node1(192.168.99.11)做为边缘节点，打上Label：

stable/nginx-ingress chart的值文件ingress-nginx.yaml

使用Helm部署dashboard

ingress-nginx

NGINX Ingress控制器的工作原理

NGINX配置

NGINX模型

所有部署都需要以下强制性命令

安装 nginx app 用以测试

创建 Ingress 访问规则

更多规则示例

MetalLB

查看状态

Layer 2模式部署

创建后端应用和服务测试

查看svc对象

在集群外部访问分配给服务的公网ip

启用`NET.BRIDGE.BRIDGE-NF-CALL-IPTABLES`内核选项

配置容器网络部署Weave Net插件

`1 node(s) had taints that the pod didn't tolerate`

`/proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1`

`stable/nginx-ingress chart`的值文件`ingress-nginx.yaml`