Error from server (InternalError): error when creating "STDIN"

MISSEY commented 2 years ago

While installing Kubeflow using the following command:

while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

I am having following errors in a loop :

Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org configured
validatingwebhookconfiguration.admissionregistration.k8s.io/istiod-istio-system configured
validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kubeflow.org configured
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.96.162.128:443: connect: connection refused
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.96.162.128:443: connect: connection refused
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.96.162.128:443: connect: connection refused

kubectl version :

kubectl version --v=10
I1210 10:30:01.517955  151960 loader.go:375] Config loaded from file:  /home/sami02/.kube/config
I1210 10:30:01.519059  151960 round_trippers.go:424] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.19.16 (linux/amd64) kubernetes/e37e4ab" 'https://10.249.6.9:6443/version?timeout=32s'
I1210 10:30:01.529846  151960 round_trippers.go:444] GET https://10.249.6.9:6443/version?timeout=32s 200 OK in 10 milliseconds
I1210 10:30:01.529867  151960 round_trippers.go:450] Response Headers:
I1210 10:30:01.529875  151960 round_trippers.go:453]     Cache-Control: no-cache, private
I1210 10:30:01.529885  151960 round_trippers.go:453]     Content-Type: application/json
I1210 10:30:01.529894  151960 round_trippers.go:453]     Content-Length: 265
I1210 10:30:01.529900  151960 round_trippers.go:453]     Date: Fri, 10 Dec 2021 09:30:01 GMT
I1210 10:30:01.533506  151960 request.go:1097] Response Body: {
  "major": "1",
  "minor": "19",
  "gitVersion": "v1.19.16",
  "gitCommit": "e37e4ab4cc8dcda84f1344dda47a97bb1927d074",
  "gitTreeState": "clean",
  "buildDate": "2021-10-27T16:20:18Z",
  "goVersion": "go1.15.15",
  "compiler": "gc",
  "platform": "linux/amd64"
}
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2021-10-27T16:25:59Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2021-10-27T16:20:18Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

Any help would be great !!

Thank you Best

Saurabh

jimthompson5802 commented 2 years ago

@MISSEY I'm new to kubeflow as well so factor that in re: the following. 😄

I basically do the same thing and for the most part it works. I think the messages you are seeing indicate that the custom resources are being constructed/initialized. If you let it run long enough (couple of minutes) it usually completes. Or it as been my experience to-date.

Couple of things to look for or try.

In a separate terminal window have you tried running a kubectl get pod -A to see what pods are initializing.

For the pods initializing, try kubectl describe <pod_name> -n <namespace>. If you may see a message about pulling images. The delay may be due to pulling in the required images.

MISSEY commented 2 years ago

@jimthompson5802

This is what I am getting

      (base) sami02@sami02:/media/sami02/1E4B5BB258FA2EEC/cluster/k8/manifests-1.4.0$ kubectl get pod -A
      NAMESPACE          NAME                                       READY   STATUS    RESTARTS   AGE
      auth               dex-5ddf47d88d-66d8v                       0/1     Pending   0          15m
      cert-manager       cert-manager-7dd5854bb4-r5c2g              0/1     Pending   0          15m
      cert-manager       cert-manager-cainjector-64c949654c-v45bf   0/1     Pending   0          15m
      cert-manager       cert-manager-webhook-6bdffc7c9d-ssc8v      0/1     Pending   0          15m
      istio-system       authservice-0                              0/1     Pending   0          15m
      istio-system       cluster-local-gateway-7bf6b98855-f9mnb     0/1     Pending   0          15m
      istio-system       istio-ingressgateway-78bc678876-cn48w      0/1     Pending   0          15m
      istio-system       istiod-755f4cc457-hrb4z                    0/1     Pending   0          15m
      knative-eventing   eventing-controller-64d97555b-nh62d        0/1     Pending   0          15m
      knative-eventing   eventing-webhook-5c5b8d5c6d-cxzx6          0/1     Pending   0          15m
      knative-eventing   imc-controller-688df5bdb4-45z7c            0/1     Pending   0          15m
      knative-eventing   imc-dispatcher-5dbb47f555-lwqp7            0/1     Pending   0          15m
      knative-eventing   mt-broker-controller-856784c8ff-jj9hj      0/1     Pending   0          15m
      knative-eventing   mt-broker-filter-68fcfcc6c8-rjgpn          0/1     Pending   0          15m
      knative-eventing   mt-broker-ingress-bd54bc995-cdvls          0/1     Pending   0          15m
      kube-system        calico-kube-controllers-558995777d-k7kvj   1/1     Running   0          6h18m
      kube-system        calico-node-fdwc4                          1/1     Running   0          6h18m
      kube-system        coredns-f9fd979d6-tw6qv                    1/1     Running   0          4d20h
      kube-system        coredns-f9fd979d6-x25hr                    1/1     Running   0          4d20h
      kube-system        etcd-sami02                                1/1     Running   0          4d20h
      kube-system        kube-apiserver-sami02                      1/1     Running   0          4d20h
      kube-system        kube-controller-manager-sami02             1/1     Running   0          4d20h
      kube-system        kube-proxy-k69vq                           1/1     Running   0          4d20h
      kube-system        kube-scheduler-sami02                      1/1     Running   0          4d20h
      (base) sami02@sami02:/media/sami02/1E4B5BB258FA2EEC/cluster/k8/manifests-1.4.0$ kubectl get namespace
      NAME               STATUS   AGE
      auth               Active   15m
      cert-manager       Active   15m
      default            Active   4d20h
      istio-system       Active   15m
      knative-eventing   Active   15m
      knative-serving    Active   15m
      kube-node-lease    Active   4d20h
      kube-public        Active   4d20h
      kube-system        Active   4d20h
      kubeflow           Active   15m
      (base) sami02@sami02:/media/sami02/1E4B5BB258FA2EEC/cluster/k8/manifests-1.4.0$ kubectl get pods -n kubeflow
      No resources found in kubeflow namespace.

jimthompson5802 commented 2 years ago

@MISSEY Have you tried a kubectl describe on the pending pods to see what they are waiting on? Example,

kubectl describe -n cert-manager pod cert-manager-7dd5854bb4-r5c2g

I believe you are using this to install kubeflow

while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

This is the same command I use. I've not experience any issues with deploying pods in the cert-manager, istio-system or knative-eventing namespaces. I've had issue with the kubeflow namespace but not these three.

What is your k8s distribution? Do you know the version?

MISSEY commented 2 years ago

@jimthompson5802

The K8s version is :

       kubeadm version
      kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2021-10-27T16:24:44Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

Yes I used below command to install Kubeflow 1.4

while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

The describe shows that 0 nodes are available. How to solve this ?


      sami02@sami02:/media/sami02/1E4B5BB258FA2EEC/cluster/k8/manifests-1.4.0$ kubectl describe -n cert-manager pod cert-manager-7dd5854bb4-r5c2g
      Name:           cert-manager-7dd5854bb4-r5c2g
      Namespace:      cert-manager
      Priority:       0
      Node:           <none>
      Labels:         app=cert-manager
                      app.kubernetes.io/component=controller
                      app.kubernetes.io/instance=cert-manager
                      app.kubernetes.io/name=cert-manager
                      pod-template-hash=7dd5854bb4
      Annotations:    prometheus.io/path: /metrics
                      prometheus.io/port: 9402
                      prometheus.io/scrape: true
      Status:         Pending
      IP:             
      IPs:            <none>
      Controlled By:  ReplicaSet/cert-manager-7dd5854bb4
      Containers:
        cert-manager:
          Image:      quay.io/jetstack/cert-manager-controller:v1.3.1
          Port:       9402/TCP
          Host Port:  0/TCP
          Args:
            --v=2
            --cluster-resource-namespace=$(POD_NAMESPACE)
            --leader-election-namespace=kube-system
          Environment:
            POD_NAMESPACE:  cert-manager (v1:metadata.namespace)
          Mounts:
            /var/run/secrets/kubernetes.io/serviceaccount from cert-manager-token-pjh6n (ro)
      Conditions:
        Type           Status
        PodScheduled   False 
      Volumes:
        cert-manager-token-pjh6n:
          Type:        Secret (a volume populated by a Secret)
          SecretName:  cert-manager-token-pjh6n
          Optional:    false
      QoS Class:       BestEffort
      Node-Selectors:  <none>
      Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                       node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
      Events:
        Type     Reason            Age                   From               Message
        ----     ------            ----                  ----               -------
        Warning  FailedScheduling  108s (x814 over 20h)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.

jimthompson5802 commented 2 years ago

@MISSEY We're reaching the limit of what I know about k8s administration. 🤷 I've not used kubeadm so not familiar with how it works in setting up a k8s cluster. So If you don't mind some "shots in the dark", I can suggest what I would do and see what happens.

Send output from kubectl version. This is to confirm that kubectl and the server are at compatible versions.
Send output from kustomize versions. Again just to confirm compatible version.
Send output from kubectl get node -o wide
Send output from kubectl describe node for each of the nodes shown in the previous command. I'm assuming you have only a few nodes (one or two nodes). I'm trying to see if this is could be the issue: https://github.com/calebhailey/homelab/issues/3
I know when you run while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done, it generates a lot of output. Could you capture all of the output in a text file and attach the file to the reply of this comment. I'd like to see the entire output.

Let's start with this and see what happens.

MISSEY commented 2 years ago

@jimthompson5802 Thank you so much for your efforts.

Kubectl version

    Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2021-10-27T16:25:59Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2021-10-27T16:20:18Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

kustomize version

Version: {KustomizeVersion:3.2.0 GitCommit:a3103f1e62ddb5b696daa3fd359bb6f2e8333b49 BuildDate:2019-09-18T16:26:36Z GoOs:linux GoArch:amd64}
kubectl get node -o wide

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME sami02 Ready master 5d21h v1.19.16 10.249.6.9 Ubuntu 20.04.3 LTS 5.11.0-41-generic docker://20.10.11

Yes, you are right I only have one master node. I tried to initiate a virtual slave node but couldn't.

kubectl describe node

Name:               sami02
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=sami02
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/master=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    projectcalico.org/IPv4Address: 10.249.6.9/20
                    projectcalico.org/IPv4IPIPTunnelAddr: 172.16.112.128
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Fri, 10 Dec 2021 17:03:09 +0100
Taints:             node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  sami02
  AcquireTime:     <unset>
  RenewTime:       Thu, 16 Dec 2021 15:00:39 +0100
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Wed, 15 Dec 2021 07:13:50 +0100   Wed, 15 Dec 2021 07:13:50 +0100   CalicoIsUp                   Calico is running on this node
  MemoryPressure       False   Thu, 16 Dec 2021 14:59:37 +0100   Fri, 10 Dec 2021 17:03:05 +0100   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 16 Dec 2021 14:59:37 +0100   Fri, 10 Dec 2021 17:03:05 +0100   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 16 Dec 2021 14:59:37 +0100   Fri, 10 Dec 2021 17:03:05 +0100   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 16 Dec 2021 14:59:37 +0100   Wed, 15 Dec 2021 07:13:56 +0100   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  10.249.6.9
  Hostname:    sami02
Capacity:
  cpu:                8
  ephemeral-storage:  479152840Ki
  hugepages-2Mi:      0
  memory:             16339260Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  441587256613
  hugepages-2Mi:      0
  memory:             16236860Ki
  pods:               110
System Info:
  Machine ID:                 e3002615a44e41a6b475a44a354c588c
  System UUID:                471bc580-d831-11dd-9b01-10bf48804aaa
  Boot ID:                    84142f1d-fee5-4643-bda9-c3eead916cbb
  Kernel Version:             5.11.0-41-generic
  OS Image:                   Ubuntu 20.04.3 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.11
  Kubelet Version:            v1.19.16
  Kube-Proxy Version:         v1.19.16
Non-terminated Pods:          (9 in total)
  Namespace                   Name                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                                        ------------  ----------  ---------------  -------------  ---
  kube-system                 calico-kube-controllers-558995777d-k7kvj    0 (0%)        0 (0%)      0 (0%)           0 (0%)         31h
  kube-system                 calico-node-fdwc4                           250m (3%)     0 (0%)      0 (0%)           0 (0%)         31h
  kube-system                 coredns-f9fd979d6-tw6qv                     100m (1%)     0 (0%)      70Mi (0%)        170Mi (1%)     5d21h
  kube-system                 coredns-f9fd979d6-x25hr                     100m (1%)     0 (0%)      70Mi (0%)        170Mi (1%)     5d21h
  kube-system                 etcd-sami02                                 0 (0%)        0 (0%)      0 (0%)           0 (0%)         5d21h
  kube-system                 kube-apiserver-sami02                       250m (3%)     0 (0%)      0 (0%)           0 (0%)         5d21h
  kube-system                 kube-controller-manager-sami02              200m (2%)     0 (0%)      0 (0%)           0 (0%)         5d21h
  kube-system                 kube-proxy-k69vq                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         5d21h
  kube-system                 kube-scheduler-sami02                       100m (1%)     0 (0%)      0 (0%)           0 (0%)         5d21h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                1 (12%)     0 (0%)
  memory             140Mi (0%)  340Mi (2%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:              <none>

Output of while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

2021/12/16 15:05:38 nil value at `valueFrom.configMapKeyRef.name` ignored in mutation attempt
2021/12/16 15:05:38 nil value at `valueFrom.secretKeyRef.name` ignored in mutation attempt
2021/12/16 15:05:38 well-defined vars that were never replaced: kfp-app-name,kfp-app-version
namespace/auth unchanged
namespace/cert-manager unchanged
namespace/istio-system unchanged
namespace/knative-eventing unchanged
namespace/knative-serving unchanged
namespace/kubeflow unchanged
customresourcedefinition.apiextensions.k8s.io/apiserversources.sources.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/brokers.eventing.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/certificates.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/channels.messaging.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/configurations.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/containersources.sources.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/eventtypes.eventing.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/experiments.kubeflow.org unchanged
customresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/ingresses.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/inmemorychannels.messaging.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/metrics.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/mxjobs.kubeflow.org configured
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/parallels.flows.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/pingsources.sources.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/podautoscalers.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/profiles.kubeflow.org configured
customresourcedefinition.apiextensions.k8s.io/pytorchjobs.kubeflow.org configured
customresourcedefinition.apiextensions.k8s.io/revisions.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/routes.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/sequences.flows.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/serverlessservices.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/services.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/sinkbindings.sources.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/subscriptions.messaging.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/suggestions.kubeflow.org unchanged
customresourcedefinition.apiextensions.k8s.io/tfjobs.kubeflow.org configured
customresourcedefinition.apiextensions.k8s.io/trials.kubeflow.org unchanged
customresourcedefinition.apiextensions.k8s.io/triggers.eventing.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/xgboostjobs.kubeflow.org configured
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/authcodes.dex.coreos.com unchanged
customresourcedefinition.apiextensions.k8s.io/authorizationpolicies.security.istio.io unchanged
customresourcedefinition.apiextensions.k8s.io/compositecontrollers.metacontroller.k8s.io unchanged
customresourcedefinition.apiextensions.k8s.io/controllerrevisions.metacontroller.k8s.io unchanged
customresourcedefinition.apiextensions.k8s.io/decoratorcontrollers.metacontroller.k8s.io unchanged
customresourcedefinition.apiextensions.k8s.io/destinationrules.networking.istio.io unchanged
customresourcedefinition.apiextensions.k8s.io/envoyfilters.networking.istio.io unchanged
customresourcedefinition.apiextensions.k8s.io/gateways.networking.istio.io unchanged
customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kubeflow.org configured
customresourcedefinition.apiextensions.k8s.io/istiooperators.install.istio.io unchanged
customresourcedefinition.apiextensions.k8s.io/mpijobs.kubeflow.org unchanged
customresourcedefinition.apiextensions.k8s.io/notebooks.kubeflow.org configured
customresourcedefinition.apiextensions.k8s.io/peerauthentications.security.istio.io unchanged
customresourcedefinition.apiextensions.k8s.io/poddefaults.kubeflow.org unchanged
customresourcedefinition.apiextensions.k8s.io/requestauthentications.security.istio.io unchanged
customresourcedefinition.apiextensions.k8s.io/scheduledworkflows.kubeflow.org unchanged
customresourcedefinition.apiextensions.k8s.io/serviceentries.networking.istio.io unchanged
customresourcedefinition.apiextensions.k8s.io/sidecars.networking.istio.io unchanged
customresourcedefinition.apiextensions.k8s.io/tensorboards.tensorboard.kubeflow.org configured
customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kubeflow.org configured
customresourcedefinition.apiextensions.k8s.io/viewers.kubeflow.org unchanged
customresourcedefinition.apiextensions.k8s.io/virtualservices.networking.istio.io unchanged
customresourcedefinition.apiextensions.k8s.io/workloadentries.networking.istio.io unchanged
customresourcedefinition.apiextensions.k8s.io/workloadgroups.networking.istio.io unchanged
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
mutatingwebhookconfiguration.admissionregistration.k8s.io/katib.kubeflow.org configured
mutatingwebhookconfiguration.admissionregistration.k8s.io/sinkbindings.webhook.sources.knative.dev unchanged
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.eventing.knative.dev unchanged
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.istio.networking.internal.knative.dev unchanged
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.serving.knative.dev unchanged
Warning: admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
mutatingwebhookconfiguration.admissionregistration.k8s.io/admission-webhook-mutating-webhook-configuration configured
mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org configured
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector configured
serviceaccount/dex unchanged
serviceaccount/cert-manager unchanged
serviceaccount/cert-manager-cainjector unchanged
serviceaccount/cert-manager-webhook unchanged
serviceaccount/cluster-local-gateway-service-account unchanged
serviceaccount/istio-ingressgateway-service-account unchanged
serviceaccount/istio-reader-service-account unchanged
serviceaccount/istiod-service-account unchanged
serviceaccount/eventing-controller unchanged
serviceaccount/eventing-webhook unchanged
serviceaccount/imc-controller unchanged
serviceaccount/imc-dispatcher unchanged
serviceaccount/mt-broker-filter unchanged
serviceaccount/mt-broker-ingress unchanged
serviceaccount/pingsource-mt-adapter unchanged
serviceaccount/controller unchanged
serviceaccount/admission-webhook-service-account unchanged
serviceaccount/argo unchanged
serviceaccount/centraldashboard unchanged
serviceaccount/jupyter-web-app-service-account unchanged
serviceaccount/katib-controller unchanged
serviceaccount/katib-ui unchanged
serviceaccount/kfserving-models-web-app unchanged
serviceaccount/kubeflow-pipelines-cache unchanged
serviceaccount/kubeflow-pipelines-cache-deployer-sa unchanged
serviceaccount/kubeflow-pipelines-container-builder unchanged
serviceaccount/kubeflow-pipelines-metadata-writer unchanged
serviceaccount/kubeflow-pipelines-viewer unchanged
serviceaccount/meta-controller-service unchanged
serviceaccount/metadata-grpc-server unchanged
serviceaccount/ml-pipeline unchanged
serviceaccount/ml-pipeline-persistenceagent unchanged
serviceaccount/ml-pipeline-scheduledworkflow unchanged
serviceaccount/ml-pipeline-ui unchanged
serviceaccount/ml-pipeline-viewer-crd-service-account unchanged
serviceaccount/ml-pipeline-visualizationserver unchanged
serviceaccount/mpi-operator unchanged
serviceaccount/mysql unchanged
serviceaccount/notebook-controller-service-account unchanged
serviceaccount/pipeline-runner unchanged
serviceaccount/profiles-controller-service-account unchanged
serviceaccount/tensorboard-controller unchanged
serviceaccount/tensorboards-web-app-service-account unchanged
serviceaccount/training-operator unchanged
serviceaccount/volumes-web-app-service-account unchanged
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving unchanged
role.rbac.authorization.k8s.io/cluster-local-gateway-sds unchanged
role.rbac.authorization.k8s.io/istio-ingressgateway-sds unchanged
role.rbac.authorization.k8s.io/istiod-istio-system unchanged
role.rbac.authorization.k8s.io/knative-eventing-webhook unchanged
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection unchanged
role.rbac.authorization.k8s.io/cert-manager:leaderelection unchanged
role.rbac.authorization.k8s.io/argo-role unchanged
role.rbac.authorization.k8s.io/centraldashboard unchanged
role.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-role unchanged
role.rbac.authorization.k8s.io/kubeflow-pipelines-cache-role unchanged
role.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-role unchanged
role.rbac.authorization.k8s.io/leader-election-role unchanged
role.rbac.authorization.k8s.io/ml-pipeline unchanged
role.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-role unchanged
role.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-role unchanged
role.rbac.authorization.k8s.io/ml-pipeline-ui unchanged
role.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role unchanged
role.rbac.authorization.k8s.io/notebook-controller-leader-election-role unchanged
role.rbac.authorization.k8s.io/pipeline-runner unchanged
role.rbac.authorization.k8s.io/profiles-leader-election-role unchanged
role.rbac.authorization.k8s.io/tensorboard-controller-leader-election-role unchanged
Warning: rbac.authorization.k8s.io/v1beta1 Role is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 Role
role.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role unchanged
clusterrole.rbac.authorization.k8s.io/addressable-resolver configured
clusterrole.rbac.authorization.k8s.io/admission-webhook-cluster-role unchanged
clusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-admin configured
clusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-edit configured
clusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-view unchanged
clusterrole.rbac.authorization.k8s.io/aggregate-to-kubeflow-pipelines-edit unchanged
clusterrole.rbac.authorization.k8s.io/aggregate-to-kubeflow-pipelines-view unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view unchanged
clusterrole.rbac.authorization.k8s.io/argo-cluster-role unchanged
clusterrole.rbac.authorization.k8s.io/broker-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/builtin-podspecable-binding unchanged
clusterrole.rbac.authorization.k8s.io/centraldashboard unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-edit unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-view unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews unchanged
clusterrole.rbac.authorization.k8s.io/channel-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/channelable-manipulator configured
clusterrole.rbac.authorization.k8s.io/eventing-broker-filter unchanged
clusterrole.rbac.authorization.k8s.io/eventing-broker-ingress unchanged
clusterrole.rbac.authorization.k8s.io/eventing-config-reader unchanged
clusterrole.rbac.authorization.k8s.io/eventing-sources-source-observer unchanged
clusterrole.rbac.authorization.k8s.io/flows-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/imc-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/imc-channelable-manipulator unchanged
clusterrole.rbac.authorization.k8s.io/imc-controller unchanged
clusterrole.rbac.authorization.k8s.io/imc-dispatcher unchanged
clusterrole.rbac.authorization.k8s.io/istio-reader-istio-system unchanged
clusterrole.rbac.authorization.k8s.io/istiod-istio-system unchanged
clusterrole.rbac.authorization.k8s.io/jupyter-web-app-cluster-role unchanged
clusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-admin configured
clusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-edit unchanged
clusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-view unchanged
clusterrole.rbac.authorization.k8s.io/katib-controller unchanged
clusterrole.rbac.authorization.k8s.io/katib-ui unchanged
clusterrole.rbac.authorization.k8s.io/kfserving-manager-role configured
clusterrole.rbac.authorization.k8s.io/kfserving-models-web-app-cluster-role unchanged
clusterrole.rbac.authorization.k8s.io/kfserving-proxy-role unchanged
clusterrole.rbac.authorization.k8s.io/knative-bindings-namespaced-admin unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-controller unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-mt-broker-filter unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-mt-broker-ingress unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-mt-channel-broker-controller unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-namespaced-admin unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-namespaced-edit unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-namespaced-view unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-pingsource-mt-adapter unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-sources-controller unchanged
clusterrole.rbac.authorization.k8s.io/knative-eventing-webhook unchanged
clusterrole.rbac.authorization.k8s.io/knative-flows-namespaced-admin unchanged
clusterrole.rbac.authorization.k8s.io/knative-messaging-namespaced-admin unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-admin configured
clusterrole.rbac.authorization.k8s.io/knative-serving-core unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-istio unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-admin unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-edit unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-view unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-podspecable-binding unchanged
clusterrole.rbac.authorization.k8s.io/knative-sources-namespaced-admin unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-admin configured
clusterrole.rbac.authorization.k8s.io/kubeflow-edit configured
clusterrole.rbac.authorization.k8s.io/kubeflow-istio-admin configured
clusterrole.rbac.authorization.k8s.io/kubeflow-istio-edit unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-istio-view unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-katib-admin configured
clusterrole.rbac.authorization.k8s.io/kubeflow-katib-edit unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-katib-view unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-admin configured
clusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-edit unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-view unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-admin unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-edit unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-view unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-mpijobs-admin configured
clusterrole.rbac.authorization.k8s.io/kubeflow-mpijobs-edit unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-mpijobs-view unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-clusterrole unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-cache-role unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-edit configured
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-role unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-view configured
clusterrole.rbac.authorization.k8s.io/kubeflow-training-admin configured
clusterrole.rbac.authorization.k8s.io/kubeflow-training-edit unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-training-view unchanged
clusterrole.rbac.authorization.k8s.io/kubeflow-view configured
clusterrole.rbac.authorization.k8s.io/messaging-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/meta-channelable-manipulator unchanged
clusterrole.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-role unchanged
clusterrole.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-role unchanged
clusterrole.rbac.authorization.k8s.io/ml-pipeline-ui unchanged
clusterrole.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role unchanged
clusterrole.rbac.authorization.k8s.io/mpi-operator unchanged
clusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-admin configured
clusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-edit unchanged
clusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-view unchanged
clusterrole.rbac.authorization.k8s.io/notebook-controller-role configured
clusterrole.rbac.authorization.k8s.io/podspecable-binding configured
clusterrole.rbac.authorization.k8s.io/service-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/serving-addressable-resolver unchanged
clusterrole.rbac.authorization.k8s.io/source-observer configured
clusterrole.rbac.authorization.k8s.io/tensorboard-controller-manager-role configured
clusterrole.rbac.authorization.k8s.io/tensorboard-controller-proxy-role unchanged
clusterrole.rbac.authorization.k8s.io/tensorboards-web-app-cluster-role unchanged
clusterrole.rbac.authorization.k8s.io/tensorboards-web-app-kubeflow-tensorboard-ui-admin configured
clusterrole.rbac.authorization.k8s.io/tensorboards-web-app-kubeflow-tensorboard-ui-edit unchanged
clusterrole.rbac.authorization.k8s.io/tensorboards-web-app-kubeflow-tensorboard-ui-view unchanged
clusterrole.rbac.authorization.k8s.io/training-operator unchanged
clusterrole.rbac.authorization.k8s.io/volumes-web-app-cluster-role unchanged
clusterrole.rbac.authorization.k8s.io/volumes-web-app-kubeflow-volume-ui-admin configured
clusterrole.rbac.authorization.k8s.io/volumes-web-app-kubeflow-volume-ui-edit unchanged
clusterrole.rbac.authorization.k8s.io/volumes-web-app-kubeflow-volume-ui-view unchanged
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/dex unchanged
clusterrole.rbac.authorization.k8s.io/ml-pipeline unchanged
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving configured
rolebinding.rbac.authorization.k8s.io/cluster-local-gateway-sds unchanged
rolebinding.rbac.authorization.k8s.io/istio-ingressgateway-sds unchanged
rolebinding.rbac.authorization.k8s.io/istiod-istio-system unchanged
rolebinding.rbac.authorization.k8s.io/eventing-webhook unchanged
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection unchanged
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection configured
rolebinding.rbac.authorization.k8s.io/argo-binding unchanged
rolebinding.rbac.authorization.k8s.io/centraldashboard unchanged
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-binding unchanged
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-rolebinding unchanged
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-binding unchanged
rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding unchanged
rolebinding.rbac.authorization.k8s.io/ml-pipeline unchanged
rolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-binding unchanged
rolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-binding unchanged
rolebinding.rbac.authorization.k8s.io/ml-pipeline-ui unchanged
rolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-binding unchanged
rolebinding.rbac.authorization.k8s.io/notebook-controller-leader-election-rolebinding unchanged
rolebinding.rbac.authorization.k8s.io/pipeline-runner-binding unchanged
rolebinding.rbac.authorization.k8s.io/profiles-leader-election-rolebinding unchanged
rolebinding.rbac.authorization.k8s.io/tensorboard-controller-leader-election-rolebinding unchanged
Warning: rbac.authorization.k8s.io/v1beta1 RoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 RoleBinding
rolebinding.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/admission-webhook-cluster-role-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/argo-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/centraldashboard unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews configured
clusterrolebinding.rbac.authorization.k8s.io/eventing-controller unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-controller-manipulator unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-controller-resolver unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-controller-source-observer unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-controller-sources-controller unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-mt-channel-broker-controller unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-webhook unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-webhook-podspecable-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/eventing-webhook-resolver unchanged
clusterrolebinding.rbac.authorization.k8s.io/imc-controller unchanged
clusterrolebinding.rbac.authorization.k8s.io/imc-dispatcher unchanged
clusterrolebinding.rbac.authorization.k8s.io/istio-reader-istio-system unchanged
clusterrolebinding.rbac.authorization.k8s.io/istiod-istio-system unchanged
clusterrolebinding.rbac.authorization.k8s.io/jupyter-web-app-cluster-role-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/katib-controller unchanged
clusterrolebinding.rbac.authorization.k8s.io/katib-ui unchanged
clusterrolebinding.rbac.authorization.k8s.io/kfserving-manager-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/kfserving-models-web-app-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/kfserving-proxy-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/knative-eventing-mt-broker-filter unchanged
clusterrolebinding.rbac.authorization.k8s.io/knative-eventing-mt-broker-ingress unchanged
clusterrolebinding.rbac.authorization.k8s.io/knative-eventing-pingsource-mt-adapter unchanged
clusterrolebinding.rbac.authorization.k8s.io/knative-serving-controller-admin unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-clusterrolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/meta-controller-cluster-role-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-ui unchanged
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/mpi-operator unchanged
clusterrolebinding.rbac.authorization.k8s.io/notebook-controller-role-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/profiles-cluster-role-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/tensorboard-controller-manager-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/tensorboard-controller-proxy-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/tensorboards-web-app-cluster-role-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/training-operator unchanged
clusterrolebinding.rbac.authorization.k8s.io/volumes-web-app-cluster-role-binding unchanged
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/dex unchanged
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline unchanged
configmap/dex unchanged
configmap/istio unchanged
configmap/istio-sidecar-injector unchanged
configmap/oidc-authservice-parameters unchanged
configmap/config-br-default-channel unchanged
configmap/config-br-defaults unchanged
configmap/config-imc-event-dispatcher unchanged
configmap/config-leader-election unchanged
configmap/config-logging unchanged
configmap/config-observability unchanged
configmap/config-ping-defaults unchanged
configmap/config-tracing unchanged
configmap/default-ch-webhook unchanged
configmap/config-autoscaler unchanged
configmap/config-defaults unchanged
configmap/config-deployment unchanged
configmap/config-domain unchanged
configmap/config-features unchanged
configmap/config-gc unchanged
configmap/config-istio unchanged
configmap/config-leader-election unchanged
configmap/config-logging unchanged
configmap/config-network unchanged
configmap/config-observability unchanged
configmap/config-tracing unchanged
configmap/centraldashboard-config unchanged
configmap/centraldashboard-parameters unchanged
configmap/inferenceservice-config unchanged
configmap/jupyter-web-app-config-c6mgtkmgh4 unchanged
configmap/jupyter-web-app-logos unchanged
configmap/jupyter-web-app-parameters-h7dbhff8cb unchanged
configmap/katib-config unchanged
configmap/kfp-launcher unchanged
configmap/kfserving-config unchanged
configmap/kfserving-models-web-app-config-mtgm8bbd98 unchanged
configmap/kubeflow-pipelines-profile-controller-code-dhhfgkff5f unchanged
configmap/kubeflow-pipelines-profile-controller-env-mgh6th2gff unchanged
configmap/metadata-grpc-configmap unchanged
configmap/ml-pipeline-ui-configmap unchanged
configmap/mpi-operator-config unchanged
configmap/namespace-labels-data-m76mmk6ckt unchanged
configmap/notebook-controller-config-kkt88f2bhb unchanged
configmap/pipeline-api-server-config-f4t72426kt unchanged
configmap/pipeline-install-config unchanged
configmap/profiles-config-2g29dd2mbd unchanged
configmap/tensorboard-controller-config-mcd64bhg84 unchanged
configmap/tensorboards-web-app-parameters-bdb48k45kd unchanged
configmap/trial-templates unchanged
configmap/volumes-web-app-parameters-d9b9bmk7ch unchanged
configmap/workflow-controller-configmap unchanged
configmap/default-install-config-5cbhhbttg4 unchanged
secret/dex-oidc-client unchanged
secret/oidc-authservice-client unchanged
secret/eventing-webhook-certs unchanged
secret/istio-webhook-certs unchanged
secret/webhook-certs unchanged
secret/katib-mysql-secrets unchanged
secret/kfserving-webhook-server-secret unchanged
secret/mlpipeline-minio-artifact configured
secret/mysql-secret configured
service/dex unchanged
service/cert-manager unchanged
service/cert-manager-webhook unchanged
service/authservice unchanged
service/cluster-local-gateway unchanged
service/istio-ingressgateway unchanged
service/istiod unchanged
service/knative-local-gateway unchanged
service/broker-filter unchanged
service/broker-ingress unchanged
service/eventing-webhook unchanged
service/imc-dispatcher unchanged
service/activator-service unchanged
service/autoscaler unchanged
service/controller unchanged
service/istio-webhook unchanged
service/webhook unchanged
service/admission-webhook-service unchanged
service/cache-server unchanged
service/centraldashboard unchanged
service/jupyter-web-app-service unchanged
service/katib-controller unchanged
service/katib-db-manager unchanged
service/katib-mysql unchanged
service/katib-ui unchanged
service/kfserving-controller-manager-metrics-service unchanged
service/kfserving-controller-manager-service unchanged
service/kfserving-models-web-app unchanged
service/kfserving-webhook-server-service unchanged
service/kubeflow-pipelines-profile-controller unchanged
service/metadata-envoy-service unchanged
service/metadata-grpc-service unchanged
service/minio-service unchanged
service/ml-pipeline unchanged
service/ml-pipeline-ui unchanged
service/ml-pipeline-visualizationserver unchanged
service/mysql unchanged
service/notebook-controller-service unchanged
service/profiles-kfam unchanged
service/tensorboard-controller-controller-manager-metrics-service unchanged
service/tensorboards-web-app-service unchanged
service/training-operator unchanged
service/volumes-web-app-service unchanged
service/workflow-controller-metrics unchanged
deployment.apps/dex unchanged
deployment.apps/cert-manager unchanged
deployment.apps/cert-manager-cainjector unchanged
deployment.apps/cert-manager-webhook unchanged
deployment.apps/cluster-local-gateway configured
deployment.apps/istio-ingressgateway configured
deployment.apps/istiod configured
deployment.apps/eventing-controller unchanged
deployment.apps/eventing-webhook unchanged
deployment.apps/imc-controller unchanged
deployment.apps/imc-dispatcher unchanged
deployment.apps/mt-broker-controller unchanged
deployment.apps/mt-broker-filter unchanged
deployment.apps/mt-broker-ingress unchanged
deployment.apps/pingsource-mt-adapter configured
deployment.apps/activator configured
deployment.apps/autoscaler configured
deployment.apps/controller configured
deployment.apps/istio-webhook unchanged
deployment.apps/networking-istio unchanged
deployment.apps/webhook unchanged
deployment.apps/admission-webhook-deployment unchanged
deployment.apps/cache-deployer-deployment unchanged
deployment.apps/cache-server configured
deployment.apps/centraldashboard configured
deployment.apps/jupyter-web-app-deployment configured
deployment.apps/katib-controller unchanged
deployment.apps/katib-db-manager unchanged
deployment.apps/katib-mysql unchanged
deployment.apps/katib-ui unchanged
deployment.apps/kfserving-models-web-app unchanged
deployment.apps/kubeflow-pipelines-profile-controller unchanged
deployment.apps/metadata-envoy-deployment unchanged
deployment.apps/metadata-grpc-deployment unchanged
deployment.apps/metadata-writer configured
deployment.apps/minio unchanged
deployment.apps/ml-pipeline configured
deployment.apps/ml-pipeline-persistenceagent configured
deployment.apps/ml-pipeline-scheduledworkflow configured
deployment.apps/ml-pipeline-ui configured
deployment.apps/ml-pipeline-viewer-crd configured
deployment.apps/ml-pipeline-visualizationserver unchanged
deployment.apps/mpi-operator unchanged
deployment.apps/mysql unchanged
deployment.apps/notebook-controller-deployment unchanged
deployment.apps/profiles-deployment unchanged
deployment.apps/tensorboard-controller-controller-manager unchanged
deployment.apps/tensorboards-web-app-deployment configured
deployment.apps/training-operator unchanged
deployment.apps/volumes-web-app-deployment configured
deployment.apps/workflow-controller unchanged
statefulset.apps/authservice unchanged
statefulset.apps/kfserving-controller-manager unchanged
statefulset.apps/metacontroller configured
poddisruptionbudget.policy/eventing-webhook unchanged
poddisruptionbudget.policy/activator-pdb unchanged
poddisruptionbudget.policy/webhook-pdb unchanged
horizontalpodautoscaler.autoscaling/broker-filter-hpa unchanged
horizontalpodautoscaler.autoscaling/broker-ingress-hpa unchanged
horizontalpodautoscaler.autoscaling/eventing-webhook unchanged
horizontalpodautoscaler.autoscaling/activator unchanged
horizontalpodautoscaler.autoscaling/webhook unchanged
image.caching.internal.knative.dev/queue-proxy unchanged
profile.kubeflow.org/kubeflow-user-example-com unchanged
compositecontroller.metacontroller.k8s.io/kubeflow-pipelines-profile-controller unchanged
destinationrule.networking.istio.io/knative unchanged
destinationrule.networking.istio.io/metadata-grpc-service unchanged
destinationrule.networking.istio.io/ml-pipeline unchanged
destinationrule.networking.istio.io/ml-pipeline-minio unchanged
destinationrule.networking.istio.io/ml-pipeline-mysql unchanged
destinationrule.networking.istio.io/ml-pipeline-ui unchanged
destinationrule.networking.istio.io/ml-pipeline-visualizationserver unchanged
envoyfilter.networking.istio.io/authn-filter unchanged
envoyfilter.networking.istio.io/metadata-exchange-1.8 unchanged
envoyfilter.networking.istio.io/metadata-exchange-1.9 unchanged
envoyfilter.networking.istio.io/stats-filter-1.8 unchanged
envoyfilter.networking.istio.io/stats-filter-1.9 unchanged
envoyfilter.networking.istio.io/tcp-metadata-exchange-1.8 unchanged
envoyfilter.networking.istio.io/tcp-metadata-exchange-1.9 unchanged
envoyfilter.networking.istio.io/tcp-stats-filter-1.8 unchanged
envoyfilter.networking.istio.io/tcp-stats-filter-1.9 unchanged
envoyfilter.networking.istio.io/x-forwarded-host unchanged
gateway.networking.istio.io/cluster-local-gateway unchanged
gateway.networking.istio.io/istio-ingressgateway unchanged
gateway.networking.istio.io/knative-local-gateway unchanged
gateway.networking.istio.io/kubeflow-gateway unchanged
virtualservice.networking.istio.io/dex unchanged
virtualservice.networking.istio.io/centraldashboard unchanged
virtualservice.networking.istio.io/jupyter-web-app-jupyter-web-app unchanged
virtualservice.networking.istio.io/katib-ui unchanged
virtualservice.networking.istio.io/metadata-grpc unchanged
virtualservice.networking.istio.io/ml-pipeline-ui unchanged
virtualservice.networking.istio.io/profiles-kfam unchanged
virtualservice.networking.istio.io/tensorboards-web-app-tensorboards-web-app unchanged
virtualservice.networking.istio.io/volumes-web-app-volumes-web-app unchanged
virtualservice.networking.istio.io/kfserving-models-web-app unchanged
authorizationpolicy.security.istio.io/cluster-local-gateway unchanged
authorizationpolicy.security.istio.io/global-deny-all unchanged
authorizationpolicy.security.istio.io/istio-ingressgateway unchanged
authorizationpolicy.security.istio.io/activator-service unchanged
authorizationpolicy.security.istio.io/autoscaler unchanged
authorizationpolicy.security.istio.io/controller unchanged
authorizationpolicy.security.istio.io/istio-webhook unchanged
authorizationpolicy.security.istio.io/webhook unchanged
authorizationpolicy.security.istio.io/kfserving-models-web-app unchanged
authorizationpolicy.security.istio.io/metadata-grpc-service unchanged
authorizationpolicy.security.istio.io/minio-service unchanged
authorizationpolicy.security.istio.io/ml-pipeline unchanged
authorizationpolicy.security.istio.io/ml-pipeline-ui unchanged
authorizationpolicy.security.istio.io/ml-pipeline-visualizationserver unchanged
authorizationpolicy.security.istio.io/mysql unchanged
authorizationpolicy.security.istio.io/service-cache-server unchanged
peerauthentication.security.istio.io/domainmapping-webhook unchanged
peerauthentication.security.istio.io/istio-webhook unchanged
peerauthentication.security.istio.io/webhook unchanged
persistentvolumeclaim/authservice-pvc unchanged
persistentvolumeclaim/katib-mysql unchanged
persistentvolumeclaim/minio-pvc unchanged
persistentvolumeclaim/mysql-pv-claim unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.eventing.knative.dev unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.istio.networking.internal.knative.dev unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.serving.knative.dev unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/katib.kubeflow.org configured
validatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.eventing.knative.dev unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.serving.knative.dev unchanged
Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org configured
validatingwebhookconfiguration.admissionregistration.k8s.io/istiod-istio-system configured
validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kubeflow.org configured
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.102.197.131:443: connect: connection refused
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.102.197.131:443: connect: connection refused
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.102.197.131:443: connect: connection refused
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.102.197.131:443: connect: connection refused
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.102.197.131:443: connect: connection refused
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.102.197.131:443: connect: connection refused
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.102.197.131:443: connect: connection refused
Retrying to apply resources

Thank you again ;)

jimthompson5802 commented 2 years ago

@MISSEY Looks like the kubectl describe node shows the issue. This is the relevant output

Taints:             node-role.kubernetes.io/master:NoSchedule

I understand this tells k8s not to schedule any pods on this node. You cluster has only one node, so nothing will start. This is the issue in the stackoverflow issue I pointed out in my previous post.

The corrective action is found in this comment. Give this a try and let me know if this resolved your issue.

MISSEY commented 2 years ago

Hi @jimthompson5802, it worked, thank you so much. Sorry for the late reply. Was off for a few days.

jimthompson5802 commented 2 years ago

@MISSEY No problem. Hope you had an enjoyable time off. I'm glad the suggestion worked. I appreciate the opportunity to learn something new.

yangy996 commented 2 years ago

Hi @jimthompson5802, it worked, thank you so much. Sorry for the late reply. Was off for a few days.

hello,it's not worked.How to solve it?

ziyou987 commented 2 years ago

Hi @MISSEY , I also encountered the same problem. Please tell me your successful method. Thank you very much!

r-matsuzaka commented 2 years ago

@jimthompson5802 @MISSEY Hello. I had the same issue and checked Taints but the value is none .
I could not understand the reason of this issue and the solution.
Can you explain about that?

$ kubectl describe node minikube
Name:               minikube
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=minikube
                    kubernetes.io/os=linux
                    minikube.k8s.io/commit=62e108c3dfdec8029a890ad6d8ef96b6461426dc
                    minikube.k8s.io/name=minikube
                    minikube.k8s.io/primary=true
                    minikube.k8s.io/updated_at=2022_08_14T13_40_38_0700
                    minikube.k8s.io/version=v1.26.1
                    node-role.kubernetes.io/control-plane=
                    node-role.kubernetes.io/master=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sun, 14 Aug 2022 13:40:34 +0900
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  minikube
  AcquireTime:     <unset>
  RenewTime:       Sun, 14 Aug 2022 15:26:32 +0900
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Sun, 14 Aug 2022 15:21:59 +0900   Sun, 14 Aug 2022 13:40:32 +0900   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Sun, 14 Aug 2022 15:21:59 +0900   Sun, 14 Aug 2022 13:40:32 +0900   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Sun, 14 Aug 2022 15:21:59 +0900   Sun, 14 Aug 2022 13:40:32 +0900   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Sun, 14 Aug 2022 15:21:59 +0900   Sun, 14 Aug 2022 13:40:35 +0900   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.39.71
  Hostname:    minikube
Capacity:
  cpu:                4
  ephemeral-storage:  45604772Ki
  hugepages-2Mi:      0
  memory:             20484460Ki
  pods:               110
Allocatable:
  cpu:                4
  ephemeral-storage:  45604772Ki
  hugepages-2Mi:      0
  memory:             20484460Ki
  pods:               110
System Info:
  Machine ID:                 5ce75aba991c4cfdb5dc4ffc8b4a5a2c
  System UUID:                5ce75aba-991c-4cfd-b5dc-4ffc8b4a5a2c
  Boot ID:                    af27df4b-b3c8-4373-8a68-07e806ecbbcd
  Kernel Version:             5.10.57
  OS Image:                   Buildroot 2021.02.12
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.17
  Kubelet Version:            v1.21.14
  Kube-Proxy Version:         v1.21.14
PodCIDR:                      10.244.0.0/24
PodCIDRs:                     10.244.0.0/24
Non-terminated Pods:          (58 in total)
  Namespace                   Name                                                      CPU Requests  CPU Limits   Memory Requests  Memory Limits         Age
  ---------                   ----                                                      ------------  ----------   ---------------  -------------         ---
  auth                        dex-559dbcd758-cxcmk                                      0 (0%)        0 (0%)       0 (0%)           0 (0%)                105m
  cert-manager                cert-manager-7b8c77d4bd-pr7nq                             0 (0%)        0 (0%)       0 (0%)           0 (0%)                105m
  cert-manager                cert-manager-cainjector-7c744f57b5-4gz7n                  0 (0%)        0 (0%)       0 (0%)           0 (0%)                105m
  cert-manager                cert-manager-webhook-fcd445bc4-slmls                      0 (0%)        0 (0%)       0 (0%)           0 (0%)                105m
  istio-system                authservice-0                                             0 (0%)        0 (0%)       0 (0%)           0 (0%)                105m
  istio-system                cluster-local-gateway-55ff4696f4-x6flv                    100m (2%)     2 (50%)      128Mi (0%)       1Gi (5%)              105m
  istio-system                istio-ingressgateway-6668f9548d-lczhl                     10m (0%)      2 (50%)      40Mi (0%)        1Gi (5%)              105m
  istio-system                istiod-6678f9548b-6ss5r                                   10m (0%)      0 (0%)       100Mi (0%)       0 (0%)                105m
  knative-eventing            eventing-controller-8457bd9747-s6bsf                      100m (2%)     0 (0%)       100Mi (0%)       0 (0%)                105m
  knative-eventing            eventing-webhook-69986cfb5d-7wtrh                         100m (2%)     200m (5%)    50Mi (0%)        200Mi (0%)            105m
  knative-serving             activator-7c5cd78566-kjdhv                                310m (7%)     3 (75%)      100Mi (0%)       1624Mi (8%)           104m
  knative-serving             autoscaler-98487645d-67dtx                                110m (2%)     3 (75%)      140Mi (0%)       2024Mi (10%)          104m
  knative-serving             controller-7546f544b7-hzmxd                               110m (2%)     3 (75%)      140Mi (0%)       2024Mi (10%)          104m
  knative-serving             domain-mapping-5d56bfc7d-kh9h9                            40m (1%)      2300m (57%)  80Mi (0%)        1424Mi (7%)           104m
  knative-serving             domainmapping-webhook-696559d49c-ql886                    110m (2%)     2500m (62%)  140Mi (0%)       1524Mi (7%)           104m
  knative-serving             net-istio-controller-c4d469c-w8dgq                        40m (1%)      2300m (57%)  80Mi (0%)        1424Mi (7%)           104m
  knative-serving             net-istio-webhook-855bcb6747-hcv45                        30m (0%)      2200m (55%)  60Mi (0%)        1224Mi (6%)           104m
  knative-serving             webhook-59f9fdd446-zc7st                                  110m (2%)     2500m (62%)  140Mi (0%)       1524Mi (7%)           104m
  kube-system                 coredns-558bd4d5db-8qfql                                  100m (2%)     0 (0%)       70Mi (0%)        170Mi (0%)            105m
  kube-system                 etcd-minikube                                             100m (2%)     0 (0%)       100Mi (0%)       0 (0%)                105m
  kube-system                 kube-apiserver-minikube                                   250m (6%)     0 (0%)       0 (0%)           0 (0%)                105m
  kube-system                 kube-controller-manager-minikube                          200m (5%)     0 (0%)       0 (0%)           0 (0%)                105m
  kube-system                 kube-proxy-6p5bl                                          0 (0%)        0 (0%)       0 (0%)           0 (0%)                105m
  kube-system                 kube-scheduler-minikube                                   100m (2%)     0 (0%)       0 (0%)           0 (0%)                105m
  kube-system                 storage-provisioner                                       0 (0%)        0 (0%)       0 (0%)           0 (0%)                105m
  kubeflow-user-example-com   hogehogehoge-0                                            510m (12%)    2600m (65%)  1064Mi (5%)      2362232012800m (11%)  86m
  kubeflow-user-example-com   ml-pipeline-ui-artifact-7cd897c59f-t5p7b                  20m (0%)      2100m (52%)  110Mi (0%)       1524Mi (7%)           93m
  kubeflow-user-example-com   ml-pipeline-visualizationserver-795f7db965-rcjfx          60m (1%)      2500m (62%)  240Mi (1%)       2Gi (10%)             93m
  kubeflow                    admission-webhook-deployment-c77d48bbb-4qm24              0 (0%)        0 (0%)       0 (0%)           0 (0%)                104m
  kubeflow                    cache-server-56d94f5d78-pxtxm                             10m (0%)      2 (50%)      40Mi (0%)        1Gi (5%)              104m
  kubeflow                    centraldashboard-5864f74d99-dsfxz                         10m (0%)      2 (50%)      40Mi (0%)        1Gi (5%)              104m
  kubeflow                    jupyter-web-app-deployment-5bc998bcb5-tr4rc               0 (0%)        0 (0%)       0 (0%)           0 (0%)                104m
  kubeflow                    katib-controller-6848d4dd9f-nbjmt                         0 (0%)        0 (0%)       0 (0%)           0 (0%)                104m
  kubeflow                    katib-db-manager-665954948-g6zxd                          0 (0%)        0 (0%)       0 (0%)           0 (0%)                104m
  kubeflow                    katib-mysql-5bf95ddfcc-bq8ft                              0 (0%)        0 (0%)       0 (0%)           0 (0%)                104m
  kubeflow                    katib-ui-56ccff658f-tq5tl                                 0 (0%)        0 (0%)       0 (0%)           0 (0%)                104m
  kubeflow                    kserve-controller-manager-0                               100m (2%)     100m (2%)    200Mi (0%)       300Mi (1%)            104m
  kubeflow                    kserve-models-web-app-5878544ffd-bblpt                    10m (0%)      2 (50%)      40Mi (0%)        1Gi (5%)              104m
  kubeflow                    kubeflow-pipelines-profile-controller-5d98fd7b4f-ldhjh    0 (0%)        0 (0%)       0 (0%)           0 (0%)                104m
  kubeflow                    metacontroller-0                                          100m (2%)     1 (25%)      256Mi (1%)       1Gi (5%)              104m
  kubeflow                    metadata-envoy-deployment-5b685dfb7f-fchc7                0 (0%)        0 (0%)       0 (0%)           0 (0%)                104m
  kubeflow                    metadata-grpc-deployment-f8d68f687-shgsb                  10m (0%)      2 (50%)      40Mi (0%)        1Gi (5%)              104m
  kubeflow                    metadata-writer-d6498d6b4-chg98                           10m (0%)      2 (50%)      40Mi (0%)        1Gi (5%)              104m
  kubeflow                    minio-5b65df66c9-842s9                                    30m (0%)      2 (50%)      140Mi (0%)       1Gi (5%)              104m
  kubeflow                    ml-pipeline-844c786c48-lbvqq                              260m (6%)     2 (50%)      540Mi (2%)       1Gi (5%)              104m
  kubeflow                    ml-pipeline-persistenceagent-5854f86f8b-h8tq9             130m (3%)     2 (50%)      540Mi (2%)       1Gi (5%)              104m
  kubeflow                    ml-pipeline-scheduledworkflow-5dddbf664f-rpccp            10m (0%)      2 (50%)      40Mi (0%)        1Gi (5%)              104m
  kubeflow                    ml-pipeline-ui-6bdfc6dbcd-btd75                           20m (0%)      2 (50%)      110Mi (0%)       1Gi (5%)              104m
  kubeflow                    ml-pipeline-viewer-crd-85f6fd557b-ddddp                   10m (0%)      2 (50%)      40Mi (0%)        1Gi (5%)              104m
  kubeflow                    ml-pipeline-visualizationserver-7c4885999-srld9           40m (1%)      2 (50%)      540Mi (2%)       1Gi (5%)              104m
  kubeflow                    mysql-5c7f79f986-mq9r7                                    110m (2%)     2 (50%)      840Mi (4%)       1Gi (5%)              104m
  kubeflow                    notebook-controller-deployment-6478d4858c-qd9c9           10m (0%)      2 (50%)      40Mi (0%)        1Gi (5%)              104m
  kubeflow                    profiles-deployment-7bc47446fb-mhrxj                      10m (0%)      2 (50%)      40Mi (0%)        1Gi (5%)              104m
  kubeflow                    tensorboard-controller-deployment-f4f555b95-7xdj8         25m (0%)      3 (75%)      168Mi (0%)       1280Mi (6%)           104m
  kubeflow                    tensorboards-web-app-deployment-7578c885f7-f76r6          0 (0%)        0 (0%)       0 (0%)           0 (0%)                104m
  kubeflow                    training-operator-6c9f6fd894-cz2g6                        100m (2%)     100m (2%)    20Mi (0%)        30Mi (0%)             104m
  kubeflow                    volumes-web-app-deployment-7bc5754bd4-mtjv4               0 (0%)        0 (0%)       0 (0%)           0 (0%)                104m
  kubeflow                    workflow-controller-6b9b6c5b46-jzh92                      110m (2%)     2 (50%)      540Mi (2%)       1Gi (5%)              104m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                3635m (90%)   68400m (1710%)
  memory             7136Mi (35%)  41998404812800m (200%)
  ephemeral-storage  0 (0%)        0 (0%)
  hugepages-2Mi      0 (0%)        0 (0%)
Events:              <none>

petr-larin commented 2 weeks ago

I ran into the same issue on minikube and the solution was - per kubeflow documentation - increase the memory:

$ minikube start --cpus 4 --memory 8096 --disk-size=40g Notes:

These are the minimum recommended settings on the VM created by minikube for kubeflow deployment

https://v0-2.kubeflow.org/docs/started/getting-started-minikube/

juliusvonkohout commented 2 weeks ago

Please check https://github.com/kubeflow/manifests/tree/master?tab=readme-ov-file#prerequisites-1

kubeflow / manifests

Error from server (InternalError): error when creating "STDIN" #2086