search

canonical / bundle-kubeflow

Charmed Kubeflow
Apache License 2.0
97 stars 48 forks source link

Deployment clean-up leaving multiple CRDs #525

Open Barteus opened 1 year ago

Barteus commented 1 year ago

Install and remove Charmed Kubeflow - multiple CRDs are left:

Reproduce:

  1. Deploy kubeflow on new cluster
  2. destroy model
  3. kubectl get crd
ca-scribner commented 1 year ago

There's other stuff that is left, too. I don't have a list on hand, but typically they're things like:

misohu commented 1 year ago

I run small test I deployed kubeflow (run couple of times examples with couple of notebooks). Then I removed the kubeflow juju destroy-model admin/kubeflow --destroy-storage --timeout=0 --force (first I tried without force but some appliecations could not be deleted)

checked resources 

ubuntu@ip-10-192-10-63:~$ kubectl get all --all-namespaces
NAMESPACE                    NAME                                                            READY   STATUS      RESTARTS      AGE
kube-system                  pod/calico-node-gqzfw                                           1/1     Running     0             26h
kube-system                  pod/calico-kube-controllers-5865f869b8-j5jns                    1/1     Running     0             26h
kube-system                  pod/coredns-7f9c69c78c-s2bt7                                    1/1     Running     0             26h
metallb-system               pod/speaker-ggg95                                               1/1     Running     0             26h
kube-system                  pod/hostpath-provisioner-566686b959-ghfv7                       1/1     Running     0             26h
metallb-system               pod/controller-559b68bfd8-47vmw                                 1/1     Running     0             26h
controller-uk8s-controller   pod/controller-0                                                2/2     Running     2 (26h ago)   26h
controller-uk8s-controller   pod/modeloperator-7c66fbdf76-5ncwr                              1/1     Running     0             26h
admin                        pod/ml-pipeline-ui-artifact-6c696b6499-9whlg                    2/2     Running     0             25h
admin                        pod/ml-pipeline-visualizationserver-795f7db965-mw6jp            2/2     Running     0             25h
user123                      pod/ml-pipeline-visualizationserver-795f7db965-v842t            2/2     Running     0             25h
user123                      pod/ml-pipeline-ui-artifact-6c696b6499-zbfnf                    2/2     Running     0             25h
user123                      pod/ml-workflow-in-kubeflow-nrzm9-1931348575                    0/2     Completed   0             24h
user123                      pod/ml-workflow-in-kubeflow-nrzm9-2878805880                    0/2     Completed   0             24h
user123                      pod/ml-workflow-in-kubeflow-nrzm9-3708426241                    0/2     Completed   0             24h
user123                      pod/demo-pipeline-2023-01-12-10-04-05-random-7895c4dc5f-rdsbf   1/1     Running     0             24h
user123                      pod/ml-workflow-in-kubeflow-nrzm9-3223576569                    0/2     Completed   0             24h
user123                      pod/ml-workflow-in-kubeflow-nrzm9-787185484                     0/2     Completed   0             24h
user123                      pod/demo-pipeline-2023-01-12-10-04-05-chief-0                   0/1     Completed   0             24h
user123                      pod/ml-workflow-in-kubeflow-nrzm9-4097731845                    0/2     Completed   0             24h

NAMESPACE                    NAME                                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
default                      service/kubernetes                                  ClusterIP   10.152.183.1     <none>        443/TCP                  26h
kube-system                  service/kube-dns                                    ClusterIP   10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP   26h
controller-uk8s-controller   service/controller-service                          ClusterIP   10.152.183.207   <none>        17070/TCP                26h
controller-uk8s-controller   service/modeloperator                               ClusterIP   10.152.183.122   <none>        17071/TCP                26h
admin                        service/ml-pipeline-visualizationserver             ClusterIP   10.152.183.239   <none>        8888/TCP                 25h
admin                        service/ml-pipeline-ui-artifact                     ClusterIP   10.152.183.45    <none>        80/TCP                   25h
user123                      service/ml-pipeline-visualizationserver             ClusterIP   10.152.183.48    <none>        8888/TCP                 25h
user123                      service/ml-pipeline-ui-artifact                     ClusterIP   10.152.183.11    <none>        80/TCP                   25h
user123                      service/demo                                        ClusterIP   10.152.183.244   <none>        80/TCP                   25h
user123                      service/demo-pipeline-2023-01-12-10-04-05-random    ClusterIP   10.152.183.116   <none>        6789/TCP                 24h
user123                      service/demo-pipeline-2023-01-12-10-04-05-chief-0   ClusterIP   None             <none>        2222/TCP                 24h

NAMESPACE        NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-system      daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux        26h
metallb-system   daemonset.apps/speaker       1         1         1       1            1           beta.kubernetes.io/os=linux   26h

NAMESPACE                    NAME                                                       READY   UP-TO-DATE   AVAILABLE   AGE
kube-system                  deployment.apps/calico-kube-controllers                    1/1     1            1           26h
kube-system                  deployment.apps/coredns                                    1/1     1            1           26h
kube-system                  deployment.apps/hostpath-provisioner                       1/1     1            1           26h
metallb-system               deployment.apps/controller                                 1/1     1            1           26h
controller-uk8s-controller   deployment.apps/modeloperator                              1/1     1            1           26h
admin                        deployment.apps/ml-pipeline-ui-artifact                    1/1     1            1           25h
admin                        deployment.apps/ml-pipeline-visualizationserver            1/1     1            1           25h
user123                      deployment.apps/ml-pipeline-visualizationserver            1/1     1            1           25h
user123                      deployment.apps/ml-pipeline-ui-artifact                    1/1     1            1           25h
user123                      deployment.apps/demo-pipeline-2023-01-12-10-04-05-random   1/1     1            1           24h

NAMESPACE                    NAME                                                                  DESIRED   CURRENT   READY   AGE
kube-system                  replicaset.apps/calico-kube-controllers-69d7f794d9                    0         0         0       26h
kube-system                  replicaset.apps/calico-kube-controllers-5865f869b8                    1         1         1       26h
kube-system                  replicaset.apps/coredns-7f9c69c78c                                    1         1         1       26h
kube-system                  replicaset.apps/hostpath-provisioner-566686b959                       1         1         1       26h
metallb-system               replicaset.apps/controller-559b68bfd8                                 1         1         1       26h
controller-uk8s-controller   replicaset.apps/modeloperator-7c66fbdf76                              1         1         1       26h
admin                        replicaset.apps/ml-pipeline-ui-artifact-6c696b6499                    1         1         1       25h
admin                        replicaset.apps/ml-pipeline-visualizationserver-795f7db965            1         1         1       25h
user123                      replicaset.apps/ml-pipeline-visualizationserver-795f7db965            1         1         1       25h
user123                      replicaset.apps/ml-pipeline-ui-artifact-6c696b6499                    1         1         1       25h
user123                      replicaset.apps/demo-pipeline-2023-01-12-10-04-05-random-7895c4dc5f   1         1         1       24h

NAMESPACE                    NAME                          READY   AGE
controller-uk8s-controller   statefulset.apps/controller   1/1     26h
user123                      statefulset.apps/demo         0/0     25h

Checked crds 

ubuntu@ip-10-192-10-63:~$ kubectl get crd
NAME                                                  CREATED AT
bgpconfigurations.crd.projectcalico.org               2023-01-12T08:35:48Z
bgppeers.crd.projectcalico.org                        2023-01-12T08:35:48Z
blockaffinities.crd.projectcalico.org                 2023-01-12T08:35:48Z
clusterinformations.crd.projectcalico.org             2023-01-12T08:35:48Z
felixconfigurations.crd.projectcalico.org             2023-01-12T08:35:48Z
globalnetworkpolicies.crd.projectcalico.org           2023-01-12T08:35:48Z
globalnetworksets.crd.projectcalico.org               2023-01-12T08:35:48Z
hostendpoints.crd.projectcalico.org                   2023-01-12T08:35:48Z
ipamblocks.crd.projectcalico.org                      2023-01-12T08:35:48Z
ipamconfigs.crd.projectcalico.org                     2023-01-12T08:35:48Z
ipamhandles.crd.projectcalico.org                     2023-01-12T08:35:48Z
ippools.crd.projectcalico.org                         2023-01-12T08:35:48Z
kubecontrollersconfigurations.crd.projectcalico.org   2023-01-12T08:35:48Z
networkpolicies.crd.projectcalico.org                 2023-01-12T08:35:48Z
networksets.crd.projectcalico.org                     2023-01-12T08:35:48Z
authcodes.dex.coreos.com                              2023-01-12T08:41:19Z
authrequests.dex.coreos.com                           2023-01-12T08:41:19Z
oauth2clients.dex.coreos.com                          2023-01-12T08:41:19Z
signingkeies.dex.coreos.com                           2023-01-12T08:41:19Z
refreshtokens.dex.coreos.com                          2023-01-12T08:41:19Z
passwords.dex.coreos.com                              2023-01-12T08:41:19Z
offlinesessionses.dex.coreos.com                      2023-01-12T08:41:19Z
connectors.dex.coreos.com                             2023-01-12T08:41:19Z
devicerequests.dex.coreos.com                         2023-01-12T08:41:19Z
devicetokens.dex.coreos.com                           2023-01-12T08:41:19Z
destinationrules.networking.istio.io                  2023-01-12T08:41:21Z
envoyfilters.networking.istio.io                      2023-01-12T08:41:21Z
gateways.networking.istio.io                          2023-01-12T08:41:21Z
istiooperators.install.istio.io                       2023-01-12T08:41:22Z
peerauthentications.security.istio.io                 2023-01-12T08:41:22Z
requestauthentications.security.istio.io              2023-01-12T08:41:22Z
serviceentries.networking.istio.io                    2023-01-12T08:41:22Z
sidecars.networking.istio.io                          2023-01-12T08:41:22Z
telemetries.telemetry.istio.io                        2023-01-12T08:41:22Z
virtualservices.networking.istio.io                   2023-01-12T08:41:22Z
workloadentries.networking.istio.io                   2023-01-12T08:41:22Z
workloadgroups.networking.istio.io                    2023-01-12T08:41:22Z
compositecontrollers.metacontroller.k8s.io            2023-01-12T08:42:15Z
controllerrevisions.metacontroller.k8s.io             2023-01-12T08:42:15Z
decoratorcontrollers.metacontroller.k8s.io            2023-01-12T08:42:16Z
xgboostjobs.kubeflow.org                              2023-01-12T08:42:43Z
tfjobs.kubeflow.org                                   2023-01-12T08:42:44Z
pytorchjobs.kubeflow.org                              2023-01-12T08:42:44Z
mxjobs.kubeflow.org                                   2023-01-12T08:42:44Z
mpijobs.kubeflow.org                                  2023-01-12T08:42:45Z
seldondeployments.machinelearning.seldon.io           2023-01-12T09:01:54Z

Webhook conf

ubuntu@ip-10-192-10-63:~$ kubectl get MutatingWebhookConfiguration
NAME                                              WEBHOOKS   AGE
juju-model-admission-controller-uk8s-controller   1          26h
istio-sidecar-injector-kubeflow                   4          26h

kubeflow namespace is empty just userspace plus those crds. We may open issues per repos with those resources