search

sungsoo / sungsoo.github.io

Sung-Soo Kim's Blog
30 stars 8 forks source link

Info: Kubeflow uninstallation #27

Open sungsoo opened 1 year ago

sungsoo commented 1 year ago

Kubeflow uninstallation (Microk8s with Kubeflow case)

Step#1. Juju uninstallation

(pytorch) ╭─sungsoo@z840 ~
╰─$ juju destroy-model kubeflow --yes --destroy-storage --force

(pytorch) ╭─sungsoo@z840 ~
╰─$ juju controllers
Use --refresh option with this command to see the latest information.

Controller           Model     User   Access     Cloud/Region        Models  Nodes  HA  Version
microk8s-localhost*  kubeflow  admin  superuser  microk8s/localhost       1      1   -  2.9.32

(pytorch) ╭─sungsoo@z840 ~
╰─$ juju destroy-controller microk8s-localhost

(pytorch) ╭─sungsoo@z840 ~
╰─$ juju kill-controller microk8s-localhost
WARNING! This command will destroy the "microk8s-localhost" controller.
This includes all machines, applications, data and other resources.

Continue? (y/N):y

(pytorch) ╭─sungsoo@z840 ~
╰─$ sudo snap remove juju
[sudo] password for sungsoo:
juju removed

(pytorch) ╭─sungsoo@z840 ~
╰─$ rm -rf ~/.local/share/juju

Step#2. Microk8s uninstallation

(pytorch) ╭─sungsoo@z840 ~
╰─$ sudo snap remove microk8s --purge
[sudo] password for sungsoo:
microk8s removed

(pytorch) ╭─sungsoo@z840 ~
╰─$ rm -rf ~/.kube

재설치 Microk8s 오류 생길 때

microk8s를 재설치하고 istio를 설치하려고 할 때, 아래와 같은 오류가 생긴다.

(base) ╭─sungsoo@z840 ~/kubeflow/istio-1.11.0
╰─$ bin/istioctl install
Error: fetch Kubernetes config file: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused

아래 명령을 실행하여 config 를 갱신하자.

(base) ╭─sungsoo@z840 ~/kubeflow/istio-1.11.0
╰─$ microk8s config > ~/.kube/config

Step#3. Microk8s Installation

(pytorch) ╭─sungsoo@z840 ~
╰─$ sudo snap install microk8s --classic --channel=1.21/stable

microk8s (1.21/stable) v1.21.12 from Canonical✓ installed
(pytorch) ╭─sungsoo@z840 ~
╰─$ sudo usermod -a -G microk8s $USER

(pytorch) ╭─sungsoo@z840 ~
╰─$ newgrp microk8s

(base) ╭─sungsoo@z840 ~
╰─$ sudo chown -f -R $USER ~/.kube

(base) ╭─sungsoo@z840 ~
╰─$ microk8s enable dns storage ingress metallb:10.64.140.43-10.64.140.49                                                                                         

(base) ╭─sungsoo@z840 ~
╰─$ microk8s status --wait-ready

microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    dns                  # CoreDNS
    ha-cluster           # Configure high availability on the current node
    ingress              # Ingress controller for external access
    metallb              # Loadbalancer for your Kubernetes cluster
    storage              # Storage class; allocates storage from host directory
  disabled:
    ambassador           # Ambassador API Gateway and Ingress
    cilium               # SDN, fast with full network policy
    dashboard            # The Kubernetes dashboard
    fluentd              # Elasticsearch-Fluentd-Kibana logging and monitoring
    gpu                  # Automatic enablement of Nvidia CUDA
    helm                 # Helm 2 - the package manager for Kubernetes
    helm3                # Helm 3 - Kubernetes package manager
    host-access          # Allow Pods connecting to Host services smoothly
    istio                # Core Istio service mesh services
    jaeger               # Kubernetes Jaeger operator with its simple config
    keda                 # Kubernetes-based Event Driven Autoscaling
    knative              # The Knative framework on Kubernetes.
    kubeflow             # Kubeflow for easy ML deployments
    linkerd              # Linkerd is a service mesh for Kubernetes and other frameworks
    metrics-server       # K8s Metrics Server for API access to service metrics
    multus               # Multus CNI enables attaching multiple network interfaces to pods
    openebs              # OpenEBS is the open-source storage solution for Kubernetes
    openfaas             # openfaas serverless framework
    portainer            # Portainer UI for your Kubernetes cluster
    prometheus           # Prometheus operator for monitoring and logging
    rbac                 # Role-Based Access Control for authorisation
    registry             # Private image registry exposed on localhost:32000
    traefik              # traefik Ingress controller for external access

(base) ╭─sungsoo@z840 ~
╰─$ k get pods -A
NAMESPACE        NAME                                      READY   STATUS    RESTARTS   AGE
kube-system      calico-node-qpg9x                         1/1     Running   0          10m
kube-system      coredns-7f9c69c78c-ssqw7                  1/1     Running   0          2m43s
kube-system      calico-kube-controllers-f7868dd95-sn94n   1/1     Running   0          10m
kube-system      hostpath-provisioner-566686b959-lsfc9     1/1     Running   0          87s
ingress          nginx-ingress-microk8s-controller-tsnkb   1/1     Running   0          88s
metallb-system   controller-559b68bfd8-9vcpx               1/1     Running   0          87s
metallb-system   speaker-p5fhs                             1/1     Running   0          88s

Install Istio

(base) ╭─sungsoo@z840 ~/kubeflow/istio-1.11.0
╰─$ bin/istioctl install  -y                                                                                                                                      127 ↵
This will install the Istio 1.11.0 default profile with ["Istio core" "Istiod" "Ingress gateways"] components into the cluster. Proceed? (y/N) y
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ Installation complete
Thank you for installing Istio 1.11.  Please take a few minutes to tell us about your install/upgrade experience!  https://forms.gle/kWULBRjUv7hHci7T6

Step#4 Juju Installation

(base) ╭─sungsoo@z840 ~
╰─$ sudo snap install juju --classic

juju (2.9/stable) 2.9.32 from Canonical✓ installed
(base) ╭─sungsoo@z840 ~
╰─$ juju bootstrap microk8s

(base) ╭─sungsoo@z840 ~
╰─$ juju add-model kubeflow

Added 'kubeflow' model on microk8s/localhost with credential 'microk8s' for user 'admin'

(base) ╭─sungsoo@z840 ~
╰─$ juju deploy ch:kubeflow --trust

(base) ╭─sungsoo@z840 ~
╰─$ watch -c juju status --color
sungsoo commented 1 year ago

Major Components Installation

  1. Install Istio

Please refer to the Istio install guide.

  1. Install Knative Serving

Please refer to Knative Serving install guide.

Note If you are looking to use PodSpec fields such as nodeSelector, affinity or tolerations which are now supported in the v1beta1 API spec, you need to turn on the corresponding feature flags in your Knative configuration.

  1. Install Cert Manager

The minimally required Cert Manager version is 1.3.0 and you can refer to Cert Manager.

Note Cert manager is required to provision webhook certs for production grade installation, alternatively you can run self signed certs generation script.

  1. Install KServe
kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.8.0/kserve.yaml
  1. Install KServe Built-in ClusterServingRuntimes
kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.8.0/kserve-runtimes.yaml
sungsoo commented 1 year ago

Sungsoo's Practice : Components Installation

I performed the KServe installation as the following versions in the recommended version matrix.

export ISTIO_VERSION=1.11.0
export KNATIVE_VERSION=knative-v1.0.0
export KSERVE_VERSION=v0.8.0
export CERT_MANAGER_VERSION=v1.3.0

1. Install Istio

To install Istio without sidecar injection:

istioctl install -y
(base) ╭─sungsoo@z840 ~/kubeflow/istio-1.11.0
╰─$ bin/istioctl install                                                                                                                                        127 ↵
This will install the Istio 1.11.0 default profile with ["Istio core" "Istiod" "Ingress gateways"] components into the cluster. Proceed? (y/N) y
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ Installation complete
Thank you for installing Istio 1.11.  Please take a few minutes to tell us about your install/upgrade experience!  https://forms.gle/kWULBRjUv7hHci7T6

istio 관련 pod가 제대로 실행되었는지 확인한다.

(base) ╭─sungsoo@z840 ~
╰─$ k get pods -A -w
istio-system                    istiod-75d5bf4676-tvztm                            1/1     Running   0          28s
istio-system                    istio-ingressgateway-85fbdd86f7-pl2lc              1/1     Running   0          23

2. Install Knative Serving

Knative is a serverless solution built on Kubernetes that is open source and managed by Google. Therefore, it is not tied to any cloud service and may be deployed locally if necessary.

Install Knative

이제 Knative를 설치해 보자.

아래 명령어을 통해, Knative 버전 1.0을 설치한다.

kubectl apply --filename https://github.com/knative/serving/releases/download/knative-v1.0.0/serving-crds.yaml
kubectl apply --filename https://github.com/knative/serving/releases/download/knative-v1.0.0/serving-core.yaml
kubectl apply --filename https://github.com/knative/net-istio/releases/download/knative-v1.0.0/release.yaml

3. Install Cert Manager

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.3.0/cert-manager.yaml
kubectl wait --for=condition=available --timeout=600s deployment/cert-manager-webhook -n cert-manager

4. Install KServe

kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.8.0/kserve.yaml
kubectl wait --for=condition=ready pod -l control-plane=kserve-controller-manager -n kserve --timeout=300s

5. Install KServe built-in servingruntimes

kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.8.0/kserve-runtimes.yaml

6. 설치 상태 확인

앞의 모든 내용을 실행 후, 설치가 제대로 되어있는지 POD 상태를 확인한다.

(base) ╭─sungsoo@z840 ~
╰─$ k get pods -A -w                  
NAMESPACE                       NAME                                               READY   STATUS    RESTARTS   AGE
kube-system                     calico-kube-controllers-f7868dd95-5qvc5            1/1     Running   0          22h
kube-system                     coredns-7f9c69c78c-sjtf5                           1/1     Running   0          22h
kube-system                     calico-node-d72j4                                  1/1     Running   0          22h

... 중간 생략

traindb-ml                      ml-pipeline-visualizationserver-569ccd5d86-jcmvn   1/1     Running   0          20h
traindb-ml                      ml-pipeline-ui-artifact-77dfb58d8b-lf8rt           1/1     Running   0          20h
istio-system                    istiod-75d5bf4676-tvztm                            1/1     Running   0          10m
istio-system                    istio-ingressgateway-85fbdd86f7-pl2lc              1/1     Running   0          9m55s
knative-serving                 autoscaler-6c8884d6ff-k9rkf                        1/1     Running   0          5m7s
knative-serving                 activator-68b7698d74-cn24l                         1/1     Running   0          5m8s
knative-serving                 controller-76cf997d95-95xmz                        1/1     Running   0          5m7s
knative-serving                 domain-mapping-57fdbf97b-j6sqf                     1/1     Running   0          5m6s
knative-serving                 domainmapping-webhook-66c5f7d596-h9qzf             1/1     Running   0          5m6s
knative-serving                 webhook-7df8fd847b-2wskb                           1/1     Running   0          5m5s
knative-serving                 net-istio-controller-544874485d-8n5xz              1/1     Running   0          2m58s
knative-serving                 net-istio-webhook-695d588d65-wq7mp                 1/1     Running   0          2m58s
cert-manager                    cert-manager-cainjector-655d695d74-czptn           1/1     Running   0          2m19s
cert-manager                    cert-manager-76b7c557d5-b8hl2                      1/1     Running   0          2m18s
cert-manager                    cert-manager-webhook-7955b9bb97-7pv7v              1/1     Running   0          2m18s
kserve                          kserve-controller-manager-0                        2/2     Running   0          72s