search

memhamwan / memhamwan.github.io

2 stars 0 forks source link

Setup k8s cluster at LEB #41

Closed turnrye closed 3 years ago

turnrye commented 3 years ago

Let’s provision a 3 node k8s cluster on our esxi host at LEB.

turnrye commented 3 years ago

Here’s the cluster.yml that I’m starting with:

# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: k8s-1.leb.memhamwan.net
  role:
  - controlplane
  - etcd
  user: turnrye
- address: k8s-2.leb.memhamwan.net
  role:
  - worker
  user: turnrye
- address: k8s-3.leb.memhamwan.net
  role:
  - worker
  user: turnrye
system_images:
  etcd: rancher/coreos-etcd:v3.4.13-rancher1
  alpine: rancher/rke-tools:v0.1.65
  nginx_proxy: rancher/rke-tools:v0.1.65
  cert_downloader: rancher/rke-tools:v0.1.65
  kubernetes_services_sidecar: rancher/rke-tools:v0.1.65
  kubedns: rancher/k8s-dns-kube-dns:1.15.10
  dnsmasq: rancher/k8s-dns-dnsmasq-nanny:1.15.10
  kubedns_sidecar: rancher/k8s-dns-sidecar:1.15.10
  kubedns_autoscaler: rancher/cluster-proportional-autoscaler:1.8.1
  coredns: rancher/coredns-coredns:1.7.0
  coredns_autoscaler: rancher/cluster-proportional-autoscaler:1.8.1
  nodelocal: rancher/k8s-dns-node-cache:1.15.13
  kubernetes: rancher/hyperkube:v1.19.3-rancher1
  flannel: rancher/coreos-flannel:v0.13.0-rancher1
  flannel_cni: rancher/flannel-cni:v0.3.0-rancher6
  calico_node: rancher/calico-node:v3.16.1
  calico_cni: rancher/calico-cni:v3.16.1
  calico_controllers: rancher/calico-kube-controllers:v3.16.1
  calico_ctl: rancher/calico-ctl:v3.16.1
  calico_flexvol: rancher/calico-pod2daemon-flexvol:v3.16.1
  canal_node: rancher/calico-node:v3.16.1
  canal_cni: rancher/calico-cni:v3.16.1
  canal_controllers: rancher/calico-kube-controllers:v3.16.1
  canal_flannel: rancher/coreos-flannel:v0.13.0-rancher1
  canal_flexvol: rancher/calico-pod2daemon-flexvol:v3.16.1
  weave_node: weaveworks/weave-kube:2.7.0
  weave_cni: weaveworks/weave-npc:2.7.0
  pod_infra_container: rancher/pause:3.2
  ingress: rancher/nginx-ingress-controller:nginx-0.35.0-rancher1
  ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.5-rancher1
  metrics_server: rancher/metrics-server:v0.3.6
  windows_pod_infra_container: rancher/kubelet-pause:v0.1.4
ssh_agent_auth: true
turnrye commented 3 years ago

Attached are the kube_config_cluster.yml and cluster.rkestate, first encrypted with openssl enc -aes-256-cbc -salt and then zipped to make github happy. kube_cluster_rke_files.zip

turnrye commented 3 years ago

Running through https://rancher.com/docs/rancher/v2.x/en/installation/install-rancher-on-k8s/ now.

turnrye commented 3 years ago

I setup a wildcard cname *.k8s.leb.memhamwan.net to use for hostnames. Here's what I configured for now for rancher:

helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=rancher.k8s.leb.memhamwan.net \
 --set ingress.tls.source=letsEncrypt \
  --set letsEncrypt.email=netops@memhamwan.org
turnrye commented 3 years ago

Running through https://metallb.universe.tf/installation/#installation-by-manifest now.

turnrye commented 3 years ago

metallb config:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 44.34.128.184-44.34.128.190

I created lb1.k8s.leb - lb7.k8s.leb entries for these in DNS.

turnrye commented 3 years ago

https://rancher.k8s.leb.memhamwan.net/ is now up with user "admin" and the standard hamwan pwd.