1.21.4版本中安装kubeflow未成功

peterkainj commented 3 years ago

在安装k8s1.21.4版本后运行python3 install.py，pod状态如下

[root@dataplatform-1 kubeflow]# kubectl get pods -A
NAMESPACE            NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager         cert-manager-649f8dfd4b-wxrls              1/1     Running   0          8h
cert-manager         cert-manager-cainjector-75cd8bbf6d-xvr4d   1/1     Running   1          7h41m
cert-manager         cert-manager-webhook-5b5cd9bd6f-kg7m5      1/1     Running   0          7h42m
istio-system         authservice-0                              1/1     Running   0          3h21m
knative-eventing     broker-controller-cfb5ccb77-6grfx          1/1     Running   0          7h41m
knative-eventing     eventing-controller-8657cd4b8-jqrrx        1/1     Running   0          7h42m
knative-eventing     eventing-webhook-67f86f4d4d-sql4q          1/1     Running   0          7h42m
knative-eventing     imc-controller-68bd666784-llrzc            1/1     Running   0          7h41m
knative-eventing     imc-dispatcher-78ff9dd847-bw8sp            1/1     Running   0          7h42m
knative-serving      activator-6f7c49884c-hmvm7                 1/1     Running   0          8h
knative-serving      autoscaler-5cc745b987-jggmn                1/1     Running   0          8h
knative-serving      controller-dd65cb4b7-kvhsn                 1/1     Running   0          8h
knative-serving      istio-webhook-5f545fc44b-jq222             1/1     Running   0          7h41m
knative-serving      networking-istio-6b6df495d6-88cpn          1/1     Running   0          8h
knative-serving      webhook-9ff656f95-5hq5j                    1/1     Running   0          7h42m
kube-system          coredns-558bd4d5db-hgcwm                   1/1     Running   0          8h
kube-system          coredns-558bd4d5db-xfdj7                   1/1     Running   0          8h
kube-system          etcd-dataplatform-1                        1/1     Running   0          8h
kube-system          kube-apiserver-dataplatform-1              1/1     Running   0          8h
kube-system          kube-controller-manager-dataplatform-1     1/1     Running   2          8h
kube-system          kube-flannel-ds-297cc                      1/1     Running   0          8h
kube-system          kube-flannel-ds-cgqjd                      1/1     Running   0          8h
kube-system          kube-flannel-ds-mx6dz                      1/1     Running   0          8h
kube-system          kube-flannel-ds-rql92                      1/1     Running   0          8h
kube-system          kube-flannel-ds-t4jhk                      1/1     Running   0          8h
kube-system          kube-proxy-26kk6                           1/1     Running   0          8h
kube-system          kube-proxy-4cnxt                           1/1     Running   0          8h
kube-system          kube-proxy-8nmtx                           1/1     Running   0          8h
kube-system          kube-proxy-fx8kq                           1/1     Running   0          8h
kube-system          kube-proxy-kx6f9                           1/1     Running   0          8h
kube-system          kube-scheduler-dataplatform-1              1/1     Running   2          8h
local-path-storage   local-path-provisioner-556d4466c8-w27pc    1/1     Running   0          3h22m

没有找到kubeflow的namespace下的pod 看安装过程在执行018,019 ,021的时候有报错

Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Warning: admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
Error from server (InternalError): error when creating "./manifest1.3/018-kfserving-overlays-kubeflow.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request
Error from server (InternalError): error when creating "./manifest1.3/018-kfserving-overlays-kubeflow.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request
b'customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kubeflow.org configured\ncustomresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kubeflow.org configured\nrole.rbac.authorization.k8s.io/leader-election-role unchanged\nclusterrole.rbac.authorization.k8s.io/kfserving-manager-role configured\nclusterrole.rbac.authorization.k8s.io/kfserving-proxy-role unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-admin configured\nclusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-edit unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-view unchanged\nrolebinding.rbac.authorization.k8s.io/leader-election-rolebinding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/kfserving-manager-rolebinding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/kfserving-proxy-rolebinding unchanged\nconfigmap/inferenceservice-config configured\nconfigmap/kfserving-config unchanged\nsecret/kfserving-webhook-server-secret unchanged\nservice/kfserving-controller-manager-metrics-service unchanged\nservice/kfserving-controller-manager-service unchanged\nservice/kfserving-webhook-server-service unchanged\nstatefulset.apps/kfserving-controller-manager unchanged\nmutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org configured\nvalidatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org configured\nvalidatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kubeflow.org configured\n'
kubectl apply -f ./manifest1.3/019-katib-installs-katib-with-kubeflow-cert-manager.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Error from server (InternalError): error when creating "./manifest1.3/019-katib-installs-katib-with-kubeflow-cert-manager.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request
Error from server (InternalError): error when creating "./manifest1.3/019-katib-installs-katib-with-kubeflow-cert-manager.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request
b'customresourcedefinition.apiextensions.k8s.io/experiments.kubeflow.org unchanged\ncustomresourcedefinition.apiextensions.k8s.io/suggestions.kubeflow.org unchanged\ncustomresourcedefinition.apiextensions.k8s.io/trials.kubeflow.org unchanged\nserviceaccount/katib-controller unchanged\nserviceaccount/katib-ui unchanged\nclusterrole.rbac.authorization.k8s.io/katib-controller unchanged\nclusterrole.rbac.authorization.k8s.io/katib-ui unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-katib-admin configured\nclusterrole.rbac.authorization.k8s.io/kubeflow-katib-edit unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-katib-view unchanged\nclusterrolebinding.rbac.authorization.k8s.io/katib-controller unchanged\nclusterrolebinding.rbac.authorization.k8s.io/katib-ui unchanged\nconfigmap/katib-config unchanged\nconfigmap/trial-template unchanged\nsecret/katib-mysql-secrets unchanged\nservice/katib-controller unchanged\nservice/katib-db-manager unchanged\nservice/katib-mysql unchanged\nservice/katib-ui unchanged\npersistentvolumeclaim/katib-mysql unchanged\ndeployment.apps/katib-controller unchanged\ndeployment.apps/katib-db-manager unchanged\ndeployment.apps/katib-mysql unchanged\ndeployment.apps/katib-ui unchanged\nvirtualservice.networking.istio.io/katib-ui unchanged\nmutatingwebhookconfiguration.admissionregistration.k8s.io/katib.kubeflow.org configured\nvalidatingwebhookconfiguration.admissionregistration.k8s.io/katib.kubeflow.org configured\n'```

```kubectl apply -f ./manifest1.3/021-admission-webhook-overlays-cert-manager.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Warning: admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
Error from server (InternalError): error when creating "./manifest1.3/021-admission-webhook-overlays-cert-manager.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request
Error from server (InternalError): error when creating "./manifest1.3/021-admission-webhook-overlays-cert-manager.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request
b'customresourcedefinition.apiextensions.k8s.io/poddefaults.kubeflow.org unchanged\nserviceaccount/admission-webhook-service-account unchanged\nclusterrole.rbac.authorization.k8s.io/admission-webhook-cluster-role unchanged\nclusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-admin configured\nclusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-edit configured\nclusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-view unchanged\nclusterrolebinding.rbac.authorization.k8s.io/admission-webhook-cluster-role-binding unchanged\nservice/admission-webhook-service unchanged\ndeployment.apps/admission-webhook-deployment unchanged\nmutatingwebhookconfiguration.admissionregistration.k8s.io/admission-webhook-mutating-webhook-configuration configured\n'

并且最后start to patch中也有报错

b'deployment.apps/jupyter-web-app-deployment created\n'
Error from server (NotFound): error when deleting "./patch/kfserving.yaml": configmaps "config-deployment" not found
b'image.caching.internal.knative.dev "queue-proxy" deleted\nconfigmap "inferenceservice-config" deleted\n'
Error from server (InternalError): error when creating "./patch/kfserving.yaml": Internal error occurred: failed calling webhook "config.webhook.serving.knative.dev": Post "https://webhook.knative-serving.svc:443/config-validation?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
b'image.caching.internal.knative.dev/queue-proxy created\nconfigmap/inferenceservice-config created\n'
b'deployment.apps "mysql" deleted\n'

我打开看了下这些文件，看其中也有下载gcr地址的镜像这些。想问下这种情况我可以怎么补救？应该怎么办？跪求指教！！

peterkainj commented 3 years ago

完整的执行日志是这样的

[root@dataplatform-1 kubeflow-manifests]# python3 install.py 
kubectl apply -f ./manifest1.3/001-cert-manager-cert-manager-kube-system-resources-base.yaml
Warning: rbac.authorization.k8s.io/v1beta1 Role is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 Role
Warning: rbac.authorization.k8s.io/v1beta1 RoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 RoleBinding
b'role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection unchanged\nrole.rbac.authorization.k8s.io/cert-manager:leaderelection unchanged\nrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection configured\nrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:webhook-authentication-reader configured\nrolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection configured\nconfigmap/cert-manager-kube-params-parameters unchanged\n'
kubectl apply -f ./manifest1.3/002-cert-manager-cert-manager-crds-base.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
b'customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io unchanged\n'
kubectl apply -f ./manifest1.3/003-cert-manager-overlays-self-signed.yaml
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
Warning: apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
Warning: admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
b'namespace/cert-manager unchanged\nserviceaccount/cert-manager unchanged\nserviceaccount/cert-manager-cainjector unchanged\nserviceaccount/cert-manager-webhook unchanged\nclusterrole.rbac.authorization.k8s.io/cert-manager-edit unchanged\nclusterrole.rbac.authorization.k8s.io/cert-manager-view unchanged\nclusterrole.rbac.authorization.k8s.io/cert-manager-webhook:webhook-requester unchanged\nclusterrole.rbac.authorization.k8s.io/cert-manager-cainjector unchanged\nclusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged\nclusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged\nclusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged\nclusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged\nclusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged\nclusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged\nclusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector unchanged\nclusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged\nclusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged\nclusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged\nclusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged\nclusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged\nclusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged\nclusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:auth-delegator configured\nconfigmap/cert-manager-parameters unchanged\nservice/cert-manager unchanged\nservice/cert-manager-webhook unchanged\ndeployment.apps/cert-manager unchanged\ndeployment.apps/cert-manager-cainjector configured\ndeployment.apps/cert-manager-webhook configured\napiservice.apiregistration.k8s.io/v1beta1.webhook.cert-manager.io unchanged\nclusterissuer.cert-manager.io/kubeflow-self-signing-issuer unchanged\nmutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured\nvalidatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured\n'
kubectl apply -f ./manifest1.3/004-istio-1-9-0-istio-crds-base.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
b'customresourcedefinition.apiextensions.k8s.io/authorizationpolicies.security.istio.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/destinationrules.networking.istio.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/envoyfilters.networking.istio.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/gateways.networking.istio.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/istiooperators.install.istio.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/peerauthentications.security.istio.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/requestauthentications.security.istio.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/serviceentries.networking.istio.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/sidecars.networking.istio.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/virtualservices.networking.istio.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/workloadentries.networking.istio.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/workloadgroups.networking.istio.io unchanged\n'
kubectl apply -f ./manifest1.3/005-istio-1-9-0-istio-namespace-base.yaml
b'namespace/istio-system unchanged\n'
kubectl apply -f ./manifest1.3/006-istio-1-9-0-istio-install-base.yaml
Warning: admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
b'serviceaccount/istio-ingressgateway-service-account unchanged\nserviceaccount/istio-reader-service-account unchanged\nserviceaccount/istiod-service-account unchanged\nrole.rbac.authorization.k8s.io/istio-ingressgateway-sds unchanged\nrole.rbac.authorization.k8s.io/istiod-istio-system unchanged\nclusterrole.rbac.authorization.k8s.io/istio-reader-istio-system unchanged\nclusterrole.rbac.authorization.k8s.io/istiod-istio-system unchanged\nrolebinding.rbac.authorization.k8s.io/istio-ingressgateway-sds unchanged\nrolebinding.rbac.authorization.k8s.io/istiod-istio-system unchanged\nclusterrolebinding.rbac.authorization.k8s.io/istio-reader-istio-system unchanged\nclusterrolebinding.rbac.authorization.k8s.io/istiod-istio-system unchanged\nconfigmap/istio unchanged\nconfigmap/istio-sidecar-injector configured\nservice/istio-ingressgateway unchanged\nservice/istiod unchanged\ndeployment.apps/istio-ingressgateway configured\ndeployment.apps/istiod configured\nenvoyfilter.networking.istio.io/metadata-exchange-1.8 unchanged\nenvoyfilter.networking.istio.io/metadata-exchange-1.9 unchanged\nenvoyfilter.networking.istio.io/stats-filter-1.8 unchanged\nenvoyfilter.networking.istio.io/stats-filter-1.9 unchanged\nenvoyfilter.networking.istio.io/tcp-metadata-exchange-1.8 unchanged\nenvoyfilter.networking.istio.io/tcp-metadata-exchange-1.9 unchanged\nenvoyfilter.networking.istio.io/tcp-stats-filter-1.8 unchanged\nenvoyfilter.networking.istio.io/tcp-stats-filter-1.9 unchanged\nenvoyfilter.networking.istio.io/x-forwarded-host unchanged\ngateway.networking.istio.io/istio-ingressgateway unchanged\nauthorizationpolicy.security.istio.io/global-deny-all unchanged\nauthorizationpolicy.security.istio.io/istio-ingressgateway unchanged\nmutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector configured\nvalidatingwebhookconfiguration.admissionregistration.k8s.io/istiod-istio-system configured\n'
kubectl apply -f ./manifest1.3/007-oidc-authservice-oidc-authservice-base.yaml
b'configmap/oidc-authservice-parameters unchanged\nsecret/oidc-authservice-client unchanged\nservice/authservice unchanged\npersistentvolumeclaim/authservice-pvc unchanged\nstatefulset.apps/authservice unchanged\nenvoyfilter.networking.istio.io/authn-filter unchanged\n'
kubectl apply -f ./manifest1.3/008-dex-overlays-istio.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
b'namespace/auth unchanged\ncustomresourcedefinition.apiextensions.k8s.io/authcodes.dex.coreos.com unchanged\nserviceaccount/dex unchanged\nclusterrole.rbac.authorization.k8s.io/dex unchanged\nclusterrolebinding.rbac.authorization.k8s.io/dex unchanged\nconfigmap/dex configured\nsecret/dex-oidc-client unchanged\nservice/dex unchanged\ndeployment.apps/dex configured\nvirtualservice.networking.istio.io/dex unchanged\n'
kubectl apply -f ./manifest1.3/009-knative-knative-serving-crds-base.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
b'namespace/knative-serving unchanged\ncustomresourcedefinition.apiextensions.k8s.io/certificates.networking.internal.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/configurations.serving.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/ingresses.networking.internal.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/metrics.autoscaling.internal.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/podautoscalers.autoscaling.internal.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/revisions.serving.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/routes.serving.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/serverlessservices.networking.internal.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/services.serving.knative.dev unchanged\n'
kubectl apply -f ./manifest1.3/010-knative-knative-serving-install-base.yaml
Warning: admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
Error from server (InternalError): error when creating "./manifest1.3/010-knative-knative-serving-install-base.yaml": Internal error occurred: failed calling webhook "config.webhook.serving.knative.dev": Post "https://webhook.knative-serving.svc:443/config-validation?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
b'serviceaccount/controller unchanged\nclusterrole.rbac.authorization.k8s.io/knative-serving-addressable-resolver unchanged\nclusterrole.rbac.authorization.k8s.io/knative-serving-admin configured\nclusterrole.rbac.authorization.k8s.io/knative-serving-core unchanged\nclusterrole.rbac.authorization.k8s.io/knative-serving-istio unchanged\nclusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-admin unchanged\nclusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-edit unchanged\nclusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-view unchanged\nclusterrole.rbac.authorization.k8s.io/knative-serving-podspecable-binding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/knative-serving-controller-admin unchanged\nconfigmap/config-autoscaler unchanged\nconfigmap/config-defaults unchanged\nconfigmap/config-domain unchanged\nconfigmap/config-gc unchanged\nconfigmap/config-istio unchanged\nconfigmap/config-leader-election unchanged\nconfigmap/config-logging unchanged\nconfigmap/config-network unchanged\nconfigmap/config-observability unchanged\nconfigmap/config-tracing unchanged\nsecret/istio-webhook-certs unchanged\nsecret/webhook-certs unchanged\nservice/activator-service unchanged\nservice/autoscaler unchanged\nservice/controller unchanged\nservice/istio-webhook unchanged\nservice/webhook unchanged\ndeployment.apps/activator configured\ndeployment.apps/autoscaler unchanged\ndeployment.apps/controller configured\ndeployment.apps/istio-webhook unchanged\ndeployment.apps/networking-istio unchanged\ndeployment.apps/webhook unchanged\nhorizontalpodautoscaler.autoscaling/activator unchanged\nimage.caching.internal.knative.dev/queue-proxy configured\ndestinationrule.networking.istio.io/knative unchanged\ngateway.networking.istio.io/cluster-local-gateway unchanged\nauthorizationpolicy.security.istio.io/activator-service unchanged\nauthorizationpolicy.security.istio.io/autoscaler unchanged\nauthorizationpolicy.security.istio.io/controller unchanged\nauthorizationpolicy.security.istio.io/istio-webhook unchanged\nauthorizationpolicy.security.istio.io/webhook unchanged\nmutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.istio.networking.internal.knative.dev unchanged\nmutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.serving.knative.dev unchanged\nvalidatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.istio.networking.internal.knative.dev unchanged\nvalidatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.serving.knative.dev unchanged\nvalidatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.serving.knative.dev unchanged\n'
kubectl apply -f ./manifest1.3/011-knative-knative-eventing-crds-base.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
b'namespace/knative-eventing unchanged\ncustomresourcedefinition.apiextensions.k8s.io/apiserversources.sources.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/brokers.eventing.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/channels.messaging.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/configmappropagations.configs.internal.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/containersources.sources.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/eventtypes.eventing.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/inmemorychannels.messaging.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/parallels.flows.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/pingsources.sources.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/sequences.flows.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/sinkbindings.sources.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/subscriptions.messaging.knative.dev unchanged\ncustomresourcedefinition.apiextensions.k8s.io/triggers.eventing.knative.dev unchanged\n'
kubectl apply -f ./manifest1.3/012-knative-knative-eventing-install-base.yaml
Warning: admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
b'serviceaccount/eventing-controller unchanged\nserviceaccount/eventing-webhook unchanged\nserviceaccount/imc-controller unchanged\nserviceaccount/imc-dispatcher unchanged\nserviceaccount/pingsource-jobrunner unchanged\nclusterrole.rbac.authorization.k8s.io/addressable-resolver configured\nclusterrole.rbac.authorization.k8s.io/broker-addressable-resolver unchanged\nclusterrole.rbac.authorization.k8s.io/builtin-podspecable-binding unchanged\nclusterrole.rbac.authorization.k8s.io/channel-addressable-resolver unchanged\nclusterrole.rbac.authorization.k8s.io/channelable-manipulator configured\nclusterrole.rbac.authorization.k8s.io/eventing-broker-filter unchanged\nclusterrole.rbac.authorization.k8s.io/eventing-broker-ingress unchanged\nclusterrole.rbac.authorization.k8s.io/eventing-config-reader unchanged\nclusterrole.rbac.authorization.k8s.io/eventing-sources-source-observer unchanged\nclusterrole.rbac.authorization.k8s.io/flows-addressable-resolver unchanged\nclusterrole.rbac.authorization.k8s.io/imc-addressable-resolver unchanged\nclusterrole.rbac.authorization.k8s.io/imc-channelable-manipulator unchanged\nclusterrole.rbac.authorization.k8s.io/imc-controller unchanged\nclusterrole.rbac.authorization.k8s.io/imc-dispatcher unchanged\nclusterrole.rbac.authorization.k8s.io/knative-eventing-channel-broker-controller unchanged\nclusterrole.rbac.authorization.k8s.io/knative-eventing-controller unchanged\nclusterrole.rbac.authorization.k8s.io/knative-eventing-jobrunner unchanged\nclusterrole.rbac.authorization.k8s.io/knative-eventing-namespaced-admin unchanged\nclusterrole.rbac.authorization.k8s.io/knative-eventing-namespaced-edit unchanged\nclusterrole.rbac.authorization.k8s.io/knative-eventing-namespaced-view unchanged\nclusterrole.rbac.authorization.k8s.io/knative-eventing-sources-controller unchanged\nclusterrole.rbac.authorization.k8s.io/knative-eventing-webhook unchanged\nclusterrole.rbac.authorization.k8s.io/knative-flows-namespaced-admin unchanged\nclusterrole.rbac.authorization.k8s.io/knative-messaging-namespaced-admin unchanged\nclusterrole.rbac.authorization.k8s.io/knative-sources-namespaced-admin unchanged\nclusterrole.rbac.authorization.k8s.io/messaging-addressable-resolver unchanged\nclusterrole.rbac.authorization.k8s.io/podspecable-binding configured\nclusterrole.rbac.authorization.k8s.io/service-addressable-resolver unchanged\nclusterrole.rbac.authorization.k8s.io/serving-addressable-resolver unchanged\nclusterrole.rbac.authorization.k8s.io/source-observer configured\nclusterrolebinding.rbac.authorization.k8s.io/eventing-channel-broker-controller unchanged\nclusterrolebinding.rbac.authorization.k8s.io/eventing-controller unchanged\nclusterrolebinding.rbac.authorization.k8s.io/eventing-controller-manipulator unchanged\nclusterrolebinding.rbac.authorization.k8s.io/eventing-controller-resolver unchanged\nclusterrolebinding.rbac.authorization.k8s.io/eventing-controller-source-observer unchanged\nclusterrolebinding.rbac.authorization.k8s.io/eventing-controller-sources-controller unchanged\nclusterrolebinding.rbac.authorization.k8s.io/eventing-webhook unchanged\nclusterrolebinding.rbac.authorization.k8s.io/eventing-webhook-podspecable-binding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/eventing-webhook-resolver unchanged\nclusterrolebinding.rbac.authorization.k8s.io/imc-controller unchanged\nclusterrolebinding.rbac.authorization.k8s.io/imc-dispatcher unchanged\nclusterrolebinding.rbac.authorization.k8s.io/pingsource-jobrunner unchanged\nconfigmap/config-br-default-channel unchanged\nconfigmap/config-br-defaults unchanged\nconfigmap/config-imc-event-dispatcher unchanged\nconfigmap/config-leader-election unchanged\nconfigmap/config-logging unchanged\nconfigmap/config-observability unchanged\nconfigmap/config-tracing unchanged\nconfigmap/default-ch-webhook unchanged\nsecret/eventing-webhook-certs unchanged\nservice/broker-filter unchanged\nservice/broker-ingress unchanged\nservice/eventing-webhook unchanged\nservice/imc-dispatcher unchanged\ndeployment.apps/broker-controller configured\ndeployment.apps/eventing-controller unchanged\ndeployment.apps/eventing-webhook unchanged\ndeployment.apps/imc-controller unchanged\ndeployment.apps/imc-dispatcher unchanged\nmutatingwebhookconfiguration.admissionregistration.k8s.io/sinkbindings.webhook.sources.knative.dev unchanged\nmutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.eventing.knative.dev unchanged\nvalidatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.eventing.knative.dev unchanged\nvalidatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.eventing.knative.dev unchanged\n'
kubectl apply -f ./manifest1.3/013-istio-1-9-0-cluster-local-gateway-base.yaml
b'serviceaccount/cluster-local-gateway-service-account unchanged\nrole.rbac.authorization.k8s.io/cluster-local-gateway-sds unchanged\nrolebinding.rbac.authorization.k8s.io/cluster-local-gateway-sds unchanged\nservice/cluster-local-gateway unchanged\ndeployment.apps/cluster-local-gateway configured\ngateway.networking.istio.io/cluster-local-gateway unchanged\nauthorizationpolicy.security.istio.io/cluster-local-gateway unchanged\n'
kubectl apply -f ./manifest1.3/014-kubeflow-namespace-kubeflow-namespace-base.yaml
b'namespace/kubeflow unchanged\n'
kubectl apply -f ./manifest1.3/015-kubeflow-roles-kubeflow-roles-base.yaml
b'clusterrole.rbac.authorization.k8s.io/kubeflow-admin configured\nclusterrole.rbac.authorization.k8s.io/kubeflow-edit configured\nclusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-admin unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-edit unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-view unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-view configured\n'
kubectl apply -f ./manifest1.3/016-istio-1-9-0-kubeflow-istio-resources-base.yaml
b'clusterrole.rbac.authorization.k8s.io/kubeflow-istio-admin configured\nclusterrole.rbac.authorization.k8s.io/kubeflow-istio-edit unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-istio-view unchanged\ngateway.networking.istio.io/kubeflow-gateway unchanged\n'
kubectl apply -f ./manifest1.3/017-pipeline-env-platform-agnostic-multi-user.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
b'customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/compositecontrollers.metacontroller.k8s.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/controllerrevisions.metacontroller.k8s.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/decoratorcontrollers.metacontroller.k8s.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/scheduledworkflows.kubeflow.org unchanged\ncustomresourcedefinition.apiextensions.k8s.io/viewers.kubeflow.org unchanged\ncustomresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io unchanged\ncustomresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io unchanged\nserviceaccount/argo unchanged\nserviceaccount/kubeflow-pipelines-cache unchanged\nserviceaccount/kubeflow-pipelines-cache-deployer-sa unchanged\nserviceaccount/kubeflow-pipelines-container-builder unchanged\nserviceaccount/kubeflow-pipelines-metadata-writer unchanged\nserviceaccount/kubeflow-pipelines-viewer unchanged\nserviceaccount/meta-controller-service unchanged\nserviceaccount/metadata-grpc-server unchanged\nserviceaccount/ml-pipeline unchanged\nserviceaccount/ml-pipeline-persistenceagent unchanged\nserviceaccount/ml-pipeline-scheduledworkflow unchanged\nserviceaccount/ml-pipeline-ui unchanged\nserviceaccount/ml-pipeline-viewer-crd-service-account unchanged\nserviceaccount/ml-pipeline-visualizationserver unchanged\nserviceaccount/mysql unchanged\nserviceaccount/pipeline-runner unchanged\nrole.rbac.authorization.k8s.io/argo-role unchanged\nrole.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-role unchanged\nrole.rbac.authorization.k8s.io/kubeflow-pipelines-cache-role unchanged\nrole.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-role unchanged\nrole.rbac.authorization.k8s.io/ml-pipeline unchanged\nrole.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-role unchanged\nrole.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-role unchanged\nrole.rbac.authorization.k8s.io/ml-pipeline-ui unchanged\nrole.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role unchanged\nrole.rbac.authorization.k8s.io/pipeline-runner unchanged\nclusterrole.rbac.authorization.k8s.io/aggregate-to-kubeflow-pipelines-edit unchanged\nclusterrole.rbac.authorization.k8s.io/aggregate-to-kubeflow-pipelines-view unchanged\nclusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin unchanged\nclusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit unchanged\nclusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view unchanged\nclusterrole.rbac.authorization.k8s.io/argo-cluster-role unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-clusterrole unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-cache-role unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-edit configured\nclusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-role unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-view configured\nclusterrole.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-role unchanged\nclusterrole.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-role unchanged\nclusterrole.rbac.authorization.k8s.io/ml-pipeline-ui unchanged\nclusterrole.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role unchanged\nclusterrole.rbac.authorization.k8s.io/ml-pipeline unchanged\nrolebinding.rbac.authorization.k8s.io/argo-binding unchanged\nrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-binding unchanged\nrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-rolebinding unchanged\nrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-binding unchanged\nrolebinding.rbac.authorization.k8s.io/ml-pipeline unchanged\nrolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-binding unchanged\nrolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-binding unchanged\nrolebinding.rbac.authorization.k8s.io/ml-pipeline-ui unchanged\nrolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-binding unchanged\nrolebinding.rbac.authorization.k8s.io/pipeline-runner-binding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/argo-binding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-binding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-clusterrolebinding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-binding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/meta-controller-cluster-role-binding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-binding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-binding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-ui unchanged\nclusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-binding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/ml-pipeline unchanged\nconfigmap/kubeflow-pipelines-profile-controller-code-c2cd68d9k4 configured\nconfigmap/kubeflow-pipelines-profile-controller-env-5252m69c4c unchanged\nconfigmap/metadata-grpc-configmap unchanged\nconfigmap/ml-pipeline-ui-configmap unchanged\nconfigmap/pipeline-api-server-config-dc9hkg52h6 unchanged\nconfigmap/pipeline-install-config configured\nconfigmap/workflow-controller-configmap configured\nsecret/mlpipeline-minio-artifact configured\nsecret/mysql-secret configured\nservice/cache-server unchanged\nservice/kubeflow-pipelines-profile-controller unchanged\nservice/metadata-envoy-service unchanged\nservice/metadata-grpc-service unchanged\nservice/minio-service unchanged\nservice/ml-pipeline unchanged\nservice/ml-pipeline-ui unchanged\nservice/ml-pipeline-visualizationserver unchanged\nservice/mysql unchanged\nservice/workflow-controller-metrics unchanged\npersistentvolumeclaim/minio-pvc unchanged\npersistentvolumeclaim/mysql-pv-claim unchanged\ndeployment.apps/cache-deployer-deployment unchanged\ndeployment.apps/cache-server configured\ndeployment.apps/kubeflow-pipelines-profile-controller unchanged\ndeployment.apps/metadata-envoy-deployment unchanged\ndeployment.apps/metadata-grpc-deployment unchanged\ndeployment.apps/metadata-writer configured\ndeployment.apps/minio configured\ndeployment.apps/ml-pipeline configured\ndeployment.apps/ml-pipeline-persistenceagent configured\ndeployment.apps/ml-pipeline-scheduledworkflow configured\ndeployment.apps/ml-pipeline-ui configured\ndeployment.apps/ml-pipeline-viewer-crd configured\ndeployment.apps/ml-pipeline-visualizationserver unchanged\ndeployment.apps/mysql configured\ndeployment.apps/workflow-controller configured\nstatefulset.apps/metacontroller configured\ncompositecontroller.metacontroller.k8s.io/kubeflow-pipelines-profile-controller unchanged\ndestinationrule.networking.istio.io/ml-pipeline unchanged\ndestinationrule.networking.istio.io/ml-pipeline-minio unchanged\ndestinationrule.networking.istio.io/ml-pipeline-mysql unchanged\ndestinationrule.networking.istio.io/ml-pipeline-ui unchanged\ndestinationrule.networking.istio.io/ml-pipeline-visualizationserver unchanged\nvirtualservice.networking.istio.io/metadata-grpc unchanged\nvirtualservice.networking.istio.io/ml-pipeline-ui unchanged\nauthorizationpolicy.security.istio.io/metadata-grpc-service unchanged\nauthorizationpolicy.security.istio.io/minio-service unchanged\nauthorizationpolicy.security.istio.io/ml-pipeline unchanged\nauthorizationpolicy.security.istio.io/ml-pipeline-ui unchanged\nauthorizationpolicy.security.istio.io/ml-pipeline-visualizationserver unchanged\nauthorizationpolicy.security.istio.io/mysql unchanged\nauthorizationpolicy.security.istio.io/service-cache-server unchanged\n'
kubectl apply -f ./manifest1.3/018-kfserving-overlays-kubeflow.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Warning: admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
Error from server (InternalError): error when creating "./manifest1.3/018-kfserving-overlays-kubeflow.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request
Error from server (InternalError): error when creating "./manifest1.3/018-kfserving-overlays-kubeflow.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request
b'customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kubeflow.org configured\ncustomresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kubeflow.org configured\nrole.rbac.authorization.k8s.io/leader-election-role unchanged\nclusterrole.rbac.authorization.k8s.io/kfserving-manager-role configured\nclusterrole.rbac.authorization.k8s.io/kfserving-proxy-role unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-admin configured\nclusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-edit unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-view unchanged\nrolebinding.rbac.authorization.k8s.io/leader-election-rolebinding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/kfserving-manager-rolebinding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/kfserving-proxy-rolebinding unchanged\nconfigmap/inferenceservice-config configured\nconfigmap/kfserving-config unchanged\nsecret/kfserving-webhook-server-secret unchanged\nservice/kfserving-controller-manager-metrics-service unchanged\nservice/kfserving-controller-manager-service unchanged\nservice/kfserving-webhook-server-service unchanged\nstatefulset.apps/kfserving-controller-manager unchanged\nmutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org configured\nvalidatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org configured\nvalidatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kubeflow.org configured\n'
kubectl apply -f ./manifest1.3/019-katib-installs-katib-with-kubeflow-cert-manager.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Error from server (InternalError): error when creating "./manifest1.3/019-katib-installs-katib-with-kubeflow-cert-manager.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request
Error from server (InternalError): error when creating "./manifest1.3/019-katib-installs-katib-with-kubeflow-cert-manager.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request
b'customresourcedefinition.apiextensions.k8s.io/experiments.kubeflow.org unchanged\ncustomresourcedefinition.apiextensions.k8s.io/suggestions.kubeflow.org unchanged\ncustomresourcedefinition.apiextensions.k8s.io/trials.kubeflow.org unchanged\nserviceaccount/katib-controller unchanged\nserviceaccount/katib-ui unchanged\nclusterrole.rbac.authorization.k8s.io/katib-controller unchanged\nclusterrole.rbac.authorization.k8s.io/katib-ui unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-katib-admin configured\nclusterrole.rbac.authorization.k8s.io/kubeflow-katib-edit unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-katib-view unchanged\nclusterrolebinding.rbac.authorization.k8s.io/katib-controller unchanged\nclusterrolebinding.rbac.authorization.k8s.io/katib-ui unchanged\nconfigmap/katib-config unchanged\nconfigmap/trial-template unchanged\nsecret/katib-mysql-secrets unchanged\nservice/katib-controller unchanged\nservice/katib-db-manager unchanged\nservice/katib-mysql unchanged\nservice/katib-ui unchanged\npersistentvolumeclaim/katib-mysql unchanged\ndeployment.apps/katib-controller unchanged\ndeployment.apps/katib-db-manager unchanged\ndeployment.apps/katib-mysql unchanged\ndeployment.apps/katib-ui unchanged\nvirtualservice.networking.istio.io/katib-ui unchanged\nmutatingwebhookconfiguration.admissionregistration.k8s.io/katib.kubeflow.org configured\nvalidatingwebhookconfiguration.admissionregistration.k8s.io/katib.kubeflow.org configured\n'
kubectl apply -f ./manifest1.3/020-centraldashboard-overlays-istio.yaml
b'serviceaccount/centraldashboard unchanged\nrole.rbac.authorization.k8s.io/centraldashboard unchanged\nclusterrole.rbac.authorization.k8s.io/centraldashboard unchanged\nrolebinding.rbac.authorization.k8s.io/centraldashboard unchanged\nclusterrolebinding.rbac.authorization.k8s.io/centraldashboard unchanged\nconfigmap/centraldashboard-config unchanged\nconfigmap/centraldashboard-parameters unchanged\nservice/centraldashboard unchanged\ndeployment.apps/centraldashboard configured\nvirtualservice.networking.istio.io/centraldashboard unchanged\n'
kubectl apply -f ./manifest1.3/021-admission-webhook-overlays-cert-manager.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Warning: admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
Error from server (InternalError): error when creating "./manifest1.3/021-admission-webhook-overlays-cert-manager.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request
Error from server (InternalError): error when creating "./manifest1.3/021-admission-webhook-overlays-cert-manager.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request
b'customresourcedefinition.apiextensions.k8s.io/poddefaults.kubeflow.org unchanged\nserviceaccount/admission-webhook-service-account unchanged\nclusterrole.rbac.authorization.k8s.io/admission-webhook-cluster-role unchanged\nclusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-admin configured\nclusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-edit configured\nclusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-view unchanged\nclusterrolebinding.rbac.authorization.k8s.io/admission-webhook-cluster-role-binding unchanged\nservice/admission-webhook-service unchanged\ndeployment.apps/admission-webhook-deployment unchanged\nmutatingwebhookconfiguration.admissionregistration.k8s.io/admission-webhook-mutating-webhook-configuration configured\n'
kubectl apply -f ./manifest1.3/022-jupyter-overlays-istio.yaml
Warning: rbac.authorization.k8s.io/v1beta1 Role is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 Role
Warning: rbac.authorization.k8s.io/v1beta1 RoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 RoleBinding
b'serviceaccount/jupyter-web-app-service-account unchanged\nrole.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role unchanged\nclusterrole.rbac.authorization.k8s.io/jupyter-web-app-cluster-role unchanged\nclusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-admin configured\nclusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-edit unchanged\nclusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-view unchanged\nrolebinding.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role-binding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/jupyter-web-app-cluster-role-binding unchanged\nconfigmap/jupyter-web-app-config-tkhtgh5mcm unchanged\nconfigmap/jupyter-web-app-parameters-chmg88cm48 unchanged\nservice/jupyter-web-app-service unchanged\ndeployment.apps/jupyter-web-app-deployment configured\nvirtualservice.networking.istio.io/jupyter-web-app-jupyter-web-app unchanged\n'
kubectl apply -f ./manifest1.3/023-jupyter-overlays-kubeflow.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
b'customresourcedefinition.apiextensions.k8s.io/notebooks.kubeflow.org configured\nserviceaccount/notebook-controller-service-account unchanged\nrole.rbac.authorization.k8s.io/notebook-controller-leader-election-role unchanged\nclusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-admin configured\nclusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-edit unchanged\nclusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-view unchanged\nclusterrole.rbac.authorization.k8s.io/notebook-controller-role configured\nrolebinding.rbac.authorization.k8s.io/notebook-controller-leader-election-rolebinding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/notebook-controller-role-binding unchanged\nconfigmap/notebook-controller-config-m44cmb547t unchanged\nservice/notebook-controller-service unchanged\ndeployment.apps/notebook-controller-deployment unchanged\n'
kubectl apply -f ./manifest1.3/024-profiles-overlays-kubeflow.yaml
b'customresourcedefinition.apiextensions.k8s.io/profiles.kubeflow.org configured\nserviceaccount/profiles-controller-service-account unchanged\nrole.rbac.authorization.k8s.io/profiles-leader-election-role unchanged\nrolebinding.rbac.authorization.k8s.io/profiles-leader-election-rolebinding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/profiles-cluster-role-binding unchanged\nconfigmap/profiles-config-46c7tgh6fd unchanged\nservice/profiles-kfam unchanged\ndeployment.apps/profiles-deployment unchanged\nvirtualservice.networking.istio.io/profiles-kfam unchanged\n'
kubectl apply -f ./manifest1.3/025-volumes-web-app-overlays-istio.yaml
b'serviceaccount/volumes-web-app-service-account unchanged\nclusterrole.rbac.authorization.k8s.io/volumes-web-app-cluster-role unchanged\nclusterrole.rbac.authorization.k8s.io/volumes-web-app-kubeflow-volume-ui-admin configured\nclusterrole.rbac.authorization.k8s.io/volumes-web-app-kubeflow-volume-ui-edit unchanged\nclusterrole.rbac.authorization.k8s.io/volumes-web-app-kubeflow-volume-ui-view unchanged\nclusterrolebinding.rbac.authorization.k8s.io/volumes-web-app-cluster-role-binding unchanged\nconfigmap/volumes-web-app-parameters-4gg8cm2gmk unchanged\nservice/volumes-web-app-service unchanged\ndeployment.apps/volumes-web-app-deployment configured\nvirtualservice.networking.istio.io/volumes-web-app-volumes-web-app unchanged\n'
kubectl apply -f ./manifest1.3/026-tensorboard-overlays-kubeflow.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
b'customresourcedefinition.apiextensions.k8s.io/tensorboards.tensorboard.kubeflow.org configured\nserviceaccount/tensorboard-controller unchanged\nrole.rbac.authorization.k8s.io/tensorboard-controller-leader-election-role unchanged\nclusterrole.rbac.authorization.k8s.io/tensorboard-controller-manager-role configured\nclusterrole.rbac.authorization.k8s.io/tensorboard-controller-proxy-role unchanged\nrolebinding.rbac.authorization.k8s.io/tensorboard-controller-leader-election-rolebinding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/tensorboard-controller-manager-rolebinding unchanged\nclusterrolebinding.rbac.authorization.k8s.io/tensorboard-controller-proxy-rolebinding unchanged\nconfigmap/tensorboard-controller-config-bf88mm96c8 unchanged\nservice/tensorboard-controller-controller-manager-metrics-service unchanged\ndeployment.apps/tensorboard-controller-controller-manager unchanged\n'
kubectl apply -f ./manifest1.3/027-tensorboard-overlays-istio.yaml
b'serviceaccount/tensorboards-web-app-service-account unchanged\nclusterrole.rbac.authorization.k8s.io/tensorboards-web-app-cluster-role unchanged\nclusterrole.rbac.authorization.k8s.io/tensorboards-web-app-kubeflow-tensorboard-ui-admin configured\nclusterrole.rbac.authorization.k8s.io/tensorboards-web-app-kubeflow-tensorboard-ui-edit unchanged\nclusterrole.rbac.authorization.k8s.io/tensorboards-web-app-kubeflow-tensorboard-ui-view unchanged\nclusterrolebinding.rbac.authorization.k8s.io/tensorboards-web-app-cluster-role-binding unchanged\nconfigmap/tensorboards-web-app-parameters-g28fbd6cch unchanged\nservice/tensorboards-web-app-service unchanged\ndeployment.apps/tensorboards-web-app-deployment configured\nvirtualservice.networking.istio.io/tensorboards-web-app-tensorboards-web-app unchanged\n'
kubectl apply -f ./manifest1.3/028-tf-training-overlays-kubeflow.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
b'customresourcedefinition.apiextensions.k8s.io/tfjobs.kubeflow.org unchanged\nserviceaccount/tf-job-operator unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-tfjobs-admin configured\nclusterrole.rbac.authorization.k8s.io/kubeflow-tfjobs-edit unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-tfjobs-view unchanged\nclusterrole.rbac.authorization.k8s.io/tf-job-operator unchanged\nclusterrolebinding.rbac.authorization.k8s.io/tf-job-operator unchanged\nservice/tf-job-operator unchanged\ndeployment.apps/tf-job-operator unchanged\n'
kubectl apply -f ./manifest1.3/029-pytorch-job-overlays-kubeflow.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
b'customresourcedefinition.apiextensions.k8s.io/pytorchjobs.kubeflow.org unchanged\nserviceaccount/pytorch-operator unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-pytorchjobs-admin configured\nclusterrole.rbac.authorization.k8s.io/kubeflow-pytorchjobs-edit unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-pytorchjobs-view unchanged\nclusterrole.rbac.authorization.k8s.io/pytorch-operator unchanged\nclusterrolebinding.rbac.authorization.k8s.io/pytorch-operator unchanged\nservice/pytorch-operator unchanged\ndeployment.apps/pytorch-operator unchanged\n'
kubectl apply -f ./manifest1.3/030-mpi-job-overlays-kubeflow.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
b'customresourcedefinition.apiextensions.k8s.io/mpijobs.kubeflow.org unchanged\nserviceaccount/mpi-operator unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-mpijobs-admin configured\nclusterrole.rbac.authorization.k8s.io/kubeflow-mpijobs-edit unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-mpijobs-view unchanged\nclusterrole.rbac.authorization.k8s.io/mpi-operator unchanged\nclusterrolebinding.rbac.authorization.k8s.io/mpi-operator unchanged\nconfigmap/mpi-operator-config unchanged\ndeployment.apps/mpi-operator unchanged\n'
kubectl apply -f ./manifest1.3/031-mxnet-job-overlays-kubeflow.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
b'customresourcedefinition.apiextensions.k8s.io/mxjobs.kubeflow.org unchanged\nserviceaccount/mxnet-operator unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-mxjobs-admin configured\nclusterrole.rbac.authorization.k8s.io/kubeflow-mxjobs-edit unchanged\nclusterrole.rbac.authorization.k8s.io/kubeflow-mxjobs-view unchanged\nclusterrole.rbac.authorization.k8s.io/mxnet-operator unchanged\nclusterrolebinding.rbac.authorization.k8s.io/mxnet-operator unchanged\ndeployment.apps/mxnet-operator unchanged\n'
kubectl apply -f ./manifest1.3/032-xgboost-job-overlays-kubeflow.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
b'customresourcedefinition.apiextensions.k8s.io/xgboostjobs.xgboostjob.kubeflow.org configured\nserviceaccount/xgboost-operator-service-account unchanged\nclusterrole.rbac.authorization.k8s.io/xgboost-operator-cluster-role unchanged\nclusterrolebinding.rbac.authorization.k8s.io/xgboost-operator-cluster-role-binding unchanged\nconfigmap/xgboost-operator-xgboost-operator-config-6ct58987ht unchanged\nservice/xgboost-operator-service unchanged\ndeployment.apps/xgboost-operator-deployment unchanged\n'
kubectl apply -f ./manifest1.3/033-user-namespace-user-namespace-base.yaml
b'configmap/default-install-config-9h2h2b6hbk configured\nprofile.kubeflow.org/kubeflow-user-example-com configured\n'
start to patch...
b'configmap "dex" deleted\ndeployment.apps "dex" deleted\nconfigmap "default-install-config-9h2h2b6hbk" deleted\nprofile.kubeflow.org "kubeflow-user-example-com" deleted\n'
b'configmap/dex created\ndeployment.apps/dex created\nconfigmap/default-install-config-9h2h2b6hbk created\nprofile.kubeflow.org/kubeflow-user-example-com created\n'
b'deployment.apps "cluster-local-gateway" deleted\n'
b'deployment.apps/cluster-local-gateway created\n'
b'deployment.apps "minio" deleted\n'
b'deployment.apps/minio created\n'
b'envoyfilter.networking.istio.io "authn-filter" deleted\n'
b'envoyfilter.networking.istio.io/authn-filter created\n'
b'deployment.apps "istio-ingressgateway" deleted\n'
b'deployment.apps/istio-ingressgateway created\n'
b'deployment.apps "istiod" deleted\nconfigmap "istio-sidecar-injector" deleted\n'
b'deployment.apps/istiod created\nconfigmap/istio-sidecar-injector created\n'
b'deployment.apps "jupyter-web-app-deployment" deleted\n'
b'deployment.apps/jupyter-web-app-deployment created\n'
Error from server (NotFound): error when deleting "./patch/kfserving.yaml": configmaps "config-deployment" not found
b'image.caching.internal.knative.dev "queue-proxy" deleted\nconfigmap "inferenceservice-config" deleted\n'
Error from server (InternalError): error when creating "./patch/kfserving.yaml": Internal error occurred: failed calling webhook "config.webhook.serving.knative.dev": Post "https://webhook.knative-serving.svc:443/config-validation?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
b'image.caching.internal.knative.dev/queue-proxy created\nconfigmap/inferenceservice-config created\n'
b'deployment.apps "mysql" deleted\n'
b'deployment.apps/mysql created\n'
b'configmap "kubeflow-pipelines-profile-controller-code-c2cd68d9k4" deleted\nconfigmap "pipeline-install-config" deleted\ndeployment.apps "workflow-controller" deleted\ndeployment.apps "kubeflow-pipelines-profile-controller" deleted\n'
b'configmap/kubeflow-pipelines-profile-controller-code-c2cd68d9k4 created\nconfigmap/pipeline-install-config created\ndeployment.apps/workflow-controller created\ndeployment.apps/kubeflow-pipelines-profile-controller created\n'
b'deployment.apps "tensorboards-web-app-deployment" deleted\n'
b'deployment.apps/tensorboards-web-app-deployment created\n'
b'deployment.apps "volumes-web-app-deployment" deleted\n'
b'deployment.apps/volumes-web-app-deployment created\n'
b'configmap "workflow-controller-configmap" deleted\ndeployment.apps "workflow-controller" deleted\ndeployment.apps "cache-server" deleted\n'
b'configmap/workflow-controller-configmap created\ndeployment.apps/workflow-controller created\ndeployment.apps/cache-server created\n'

shikanon commented 3 years ago

@peterkainj 你的k8s版本是多少，我看你运行日志有很多unchanged，说明你应该是之前有安装过相关组件，你可以先尝试卸载所有的组件后再次安装，卸载命令：

kubectl delete -f manifest1.3/

peterkainj commented 3 years ago

我之前安装了1.22版本的缘故，在网上找不到很好的下载包的方法，直到看到您的帖子，clone下来试了试，发现1.22版本中很多beta版本的api取消了，再翻帖子回来看原来是针对1.21版本的，所以后面退了版本，退成1.21.4版本。但也确实一直再试apply（install.py），并没有试delete。我先卸载下再安装试试

peterkainj commented 3 years ago

@peterkainj 你的k8s版本是多少，我看你运行日志有很多unchanged，说明你应该是之前有安装过相关组件，你可以先尝试卸载所有的组件后再次安装，卸载命令：
kubectl delete -f manifest1.3/

目前已经能继续往下进行了，然而执行完脚本发现有些pod起不来

[root@dataplatform-1 kubeflow-manifests]# kubectl get pods -A
NAMESPACE            NAME                                                        READY   STATUS             RESTARTS   AGE
auth                 dex-6d8cd4fccb-4qgfh                                        1/1     Running            0          2m24s
cert-manager         cert-manager-649f8dfd4b-zrvb6                               1/1     Running            0          8m25s
cert-manager         cert-manager-cainjector-75cd8bbf6d-sqwc5                    1/1     Running            0          8m25s
cert-manager         cert-manager-webhook-5b5cd9bd6f-4sq6p                       1/1     Running            0          8m25s
istio-system         authservice-0                                               1/1     Running            0          7m39s
istio-system         cluster-local-gateway-5c9656c74f-g9cwd                      0/1     Running            0          2m23s
istio-system         istio-ingressgateway-58d9c489b4-2hfxk                       0/1     Running            0          2m19s
istio-system         istiod-74c965dc6f-5qjw9                                     1/1     Running            0          2m18s
knative-eventing     broker-controller-cfb5ccb77-5tt2s                           1/1     Running            0          6m41s
knative-eventing     eventing-controller-8657cd4b8-npsmf                         1/1     Running            0          6m41s
knative-eventing     eventing-webhook-67f86f4d4d-2zxgn                           1/1     Running            0          6m41s
knative-eventing     imc-controller-68bd666784-nf886                             1/1     Running            0          6m41s
knative-eventing     imc-dispatcher-78ff9dd847-rp2f5                             1/1     Running            0          6m41s
knative-serving      activator-6f7c49884c-bczp4                                  1/1     Running            0          7m5s
knative-serving      autoscaler-5cc745b987-jrq46                                 1/1     Running            0          7m5s
knative-serving      controller-dd65cb4b7-cctb5                                  1/1     Running            0          7m5s
knative-serving      istio-webhook-5f545fc44b-m2skc                              1/1     Running            0          7m5s
knative-serving      networking-istio-6b6df495d6-85vx8                           1/1     Running            0          7m5s
knative-serving      webhook-9ff656f95-g6m44                                     1/1     Running            0          7m5s
kube-system          calico-kube-controllers-58497c65d5-mgqs6                    1/1     Running            14         25h
kube-system          calico-node-8p5bb                                           1/1     Running            0          24h
kube-system          calico-node-9jq8r                                           1/1     Running            0          25h
kube-system          calico-node-9klc6                                           1/1     Running            0          24h
kube-system          calico-node-hpqx9                                           1/1     Running            0          24h
kube-system          calico-node-xwfqt                                           1/1     Running            0          24h
kube-system          coredns-558bd4d5db-b4k89                                    1/1     Running            0          25h
kube-system          coredns-558bd4d5db-ftqqf                                    1/1     Running            0          25h
kube-system          etcd-dataplatform-1                                         1/1     Running            0          25h
kube-system          kube-apiserver-dataplatform-1                               1/1     Running            0          20m
kube-system          kube-controller-manager-dataplatform-1                      1/1     Running            3          133m
kube-system          kube-proxy-2lfzr                                            1/1     Running            0          24h
kube-system          kube-proxy-4jcv4                                            1/1     Running            0          24h
kube-system          kube-proxy-fcgrl                                            1/1     Running            0          24h
kube-system          kube-proxy-kzgql                                            1/1     Running            0          24h
kube-system          kube-proxy-rlf8k                                            1/1     Running            0          25h
kube-system          kube-scheduler-dataplatform-1                               1/1     Running            3          133m
kubeflow             admission-webhook-deployment-5f5cc7968b-s6d9x               1/1     Running            0          4m52s
kubeflow             cache-deployer-deployment-64598b6c87-kncms                  2/2     Running            1          5m44s
kubeflow             cache-server-59d67c7584-vb5lf                               1/2     Running            0          2m8s
kubeflow             centraldashboard-576d9c6b5d-mst76                           1/1     Running            0          5m5s
kubeflow             jupyter-web-app-deployment-7c6974bb88-9gqsk                 1/1     Running            0          2m11s
kubeflow             katib-controller-7b784c44dd-dtppq                           1/1     Running            0          5m18s
kubeflow             katib-db-manager-7b8fcf9c78-h9hxl                           1/1     Running            0          5m18s
kubeflow             katib-mysql-db94bf5f4-5zsds                                 1/1     Running            0          5m18s
kubeflow             katib-ui-69f5b6795d-j2xst                                   1/1     Running            0          5m18s
kubeflow             kfserving-controller-manager-0                              2/2     Running            0          5m30s
kubeflow             kubeflow-pipelines-profile-controller-76c45c8c6b-dp59g      1/1     Running            0          2m11s
kubeflow             metacontroller-0                                            1/1     Running            0          5m43s
kubeflow             metadata-envoy-deployment-56f745f7fb-6rhk6                  1/1     Running            0          5m44s
kubeflow             metadata-grpc-deployment-895c86c94-pl7rk                    2/2     Running            2          5m44s
kubeflow             metadata-writer-b7ff9787-hnv4w                              2/2     Running            0          5m44s
kubeflow             minio-cc8f7c6d-75whk                                        2/2     Running            0          2m21s
kubeflow             ml-pipeline-55989f45f4-qkjnf                                2/2     Running            3          5m43s
kubeflow             ml-pipeline-persistenceagent-7fb8f6dc68-j76v8               2/2     Running            0          5m43s
kubeflow             ml-pipeline-scheduledworkflow-64bcfd6596-88xr8              2/2     Running            0          5m43s
kubeflow             ml-pipeline-ui-7c67689947-9nb48                             2/2     Running            0          5m43s
kubeflow             ml-pipeline-viewer-crd-565fb9b5c5-5z4r9                     2/2     Running            1          5m43s
kubeflow             ml-pipeline-visualizationserver-545895fddd-qnrnb            2/2     Running            0          5m42s
kubeflow             mpi-operator-794849c566-xp26w                               1/1     Running            0          3m8s
kubeflow             mxnet-operator-6668d797d4-sl8gv                             1/1     Running            0          2m57s
kubeflow             mysql-c8d548489-bfmll                                       1/2     Running            0          2m11s
kubeflow             notebook-controller-deployment-7b4895474c-9j22l             1/1     Running            0          4m29s
kubeflow             profiles-deployment-6f7d8bbb7c-wpfvf                        2/2     Running            0          4m18s
kubeflow             pytorch-operator-6887749499-cgtjs                           2/2     Running            0          3m21s
kubeflow             tensorboard-controller-controller-manager-dd896c8df-jjw2m   3/3     Running            1          3m55s
kubeflow             tensorboards-web-app-deployment-5969cd5b68-nvk5j            1/1     Running            0          2m11s
kubeflow             tf-job-operator-ccb48b77b-dtkvz                             1/1     Running            0          3m32s
kubeflow             volumes-web-app-deployment-867dfb5b5c-g6m5k                 1/1     Running            0          2m9s
kubeflow             workflow-controller-7d7886dfd4-cv26w                        0/2     CrashLoopBackOff   4          2m8s
kubeflow             xgboost-operator-deployment-665cf9bf8d-8rpmf                2/2     Running            1          2m45s
local-path-storage   local-path-provisioner-7c4f6f75cd-xgrfw                     1/1     Running            15         6h34m

看了workflow-controller的logs日志

[root@dataplatform-1 kubeflow-manifests]# kubectl logs workflow-controller-7d7886dfd4-cv26w -n kubeflow
Using deprecated annotation `kubectl.kubernetes.io/default-logs-container` in pod/workflow-controller-7d7886dfd4-cv26w. Please use `kubectl.kubernetes.io/default-container` instead
time="2021-08-31T10:02:34.648Z" level=info msg="config map" name=workflow-controller-configmap
time="2021-08-31T10:02:34.649Z" level=fatal msg="Failed to register watch for controller config map: Get https://10.1.0.1:443/api/v1/namespaces/kubeflow/configmaps/workflow-controller-configmap: dial tcp 10.1.0.1:443: connect: connection refused"
[root@dataplatform-1 kubeflow-manifests]# kubectl get svc -A
NAMESPACE          NAME                                                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                                                      AGE
auth               dex                                                         NodePort    10.1.160.67    <none>        5556:32000/TCP                                                               9m46s
cert-manager       cert-manager                                                ClusterIP   10.1.69.82     <none>        9402/TCP                                                                     10m
cert-manager       cert-manager-webhook                                        ClusterIP   10.1.217.86    <none>        443/TCP                                                                      10m
default            kubernetes                                                  ClusterIP   10.1.0.1       <none>        443/TCP                                                                      25h
istio-system       authservice                                                 ClusterIP   10.1.9.121     <none>        8080/TCP                                                                     9m56s
istio-system       cluster-local-gateway                                       ClusterIP   10.1.68.155    <none>        15020/TCP,80/TCP                                                             8m47s
istio-system       istio-ingressgateway                                        NodePort    10.1.16.125    <none>        15021:32494/TCP,80:30000/TCP,443:30880/TCP,31400:32206/TCP,15443:32702/TCP   10m
istio-system       istiod                                                      ClusterIP   10.1.253.154   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP                                        10m
knative-eventing   broker-filter                                               ClusterIP   10.1.105.104   <none>        80/TCP,9090/TCP  
...

10.1.0.1:443是kubernetes服务的地址，然后我看了istio相关的几个pod，发现两个READY状态是0/1的pod报同样的问题

[root@dataplatform-1 kubeflow-manifests]# kubectl describe pod cluster-local-gateway-5c9656c74f-g9cwd -n istio-system
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  36m                   default-scheduler  Successfully assigned istio-system/cluster-local-gateway-5c9656c74f-g9cwd to dataplatform-3
  Normal   Pulled     36m                   kubelet            Container image "registry.cn-shenzhen.aliyuncs.com/tensorbytes/istio-proxyv2:1.9.0-e8a74" already present on machine
  Normal   Created    36m                   kubelet            Created container istio-proxy
  Normal   Started    36m                   kubelet            Started container istio-proxy
  Warning  Unhealthy  35m (x16 over 36m)    kubelet            Readiness probe failed: Get "http://10.244.149.223:15021/healthz/ready": dial tcp 10.244.149.223:15021: connect: connection refused
  Warning  Unhealthy  67s (x1035 over 35m)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503

[root@dataplatform-1 kubeflow-manifests]# kubectl logs cluster-local-gateway-5c9656c74f-g9cwd -n istio-system
...
2021-08-31T10:12:52.886175Z warn    sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unauthenticated desc = request authenticate failure
2021-08-31T10:12:53.011621Z warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-08-31T10:12:53.267797Z warn    sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unauthenticated desc = request authenticate failure

istio-ingressgateway也是类似的日志而authservice-0报

time="2021-08-31T09:50:24Z" level=error msg="OIDC provider setup failed, retrying in 10 seconds: Get http://dex.auth.svc.cluster.local:5556/dex/.well-known/openid-configuration: dial tcp 10.1.160.67:5556: connect: connection refused"
time="2021-08-31T09:50:35Z" level=error msg="OIDC provider setup failed, retrying in 10 seconds: Get http://dex.auth.svc.cluster.local:5556/dex/.well-known/openid-configuration: dial tcp 10.1.160.67:5556: connect: connection refused"
time="2021-08-31T09:50:46Z" level=error msg="OIDC provider setup failed, retrying in 10 seconds: Get http://dex.auth.svc.cluster.local:5556/dex/.well-known/openid-configuration: dial tcp 10.1.160.67:5556: connect: connection refused"
time="2021-08-31T09:50:57Z" level=error msg="OIDC provider setup failed, retrying in 10 seconds: Get http://dex.auth.svc.cluster.local:5556/dex/.well-known/openid-configuration: dial tcp 10.1.160.67:5556: connect: connection refused"

dex的日志正常

[root@dataplatform-1 kubeflow-manifests]# kubectl logs dex-6d8cd4fccb-4qgfh -n auth
...
time="2021-08-31T09:51:03Z" level=info msg="checking if custom resource offlinesessionses.dex.coreos.com has been created already..."
time="2021-08-31T09:51:03Z" level=info msg="The custom resource offlinesessionses.dex.coreos.com already available, skipping create"
time="2021-08-31T09:51:03Z" level=info msg="checking if custom resource connectors.dex.coreos.com has been created already..."
time="2021-08-31T09:51:03Z" level=info msg="The custom resource connectors.dex.coreos.com already available, skipping create"
time="2021-08-31T09:51:03Z" level=info msg="config storage: kubernetes"
time="2021-08-31T09:51:03Z" level=info msg="config static client: Dex Login Application"
time="2021-08-31T09:51:03Z" level=info msg="config connector: local passwords enabled"
time="2021-08-31T09:51:03Z" level=info msg="config skipping approval screen"
time="2021-08-31T09:51:03Z" level=info msg="keys expired, rotating"
time="2021-08-31T09:51:03Z" level=info msg="keys rotated, next rotation: 2021-08-31 15:51:03.466601156 +0000 UTC"
time="2021-08-31T09:51:03Z" level=info msg="listening (http) on 0.0.0.0:5556"

感觉上是istio认证出了问题，网络不通，但怎么处理还是一头雾水。。。项目也重新安装过多次，都是同样的问题

shikanon commented 3 years ago

@peterkainj 先确保网络是否有问题，可以找一个没有 sidecar 的pod ，进到容器里面安装 telnet ，然后 telnet 目标ip + 端口看看，如果istio证书有问题，可以看看 istiod 的日志，这个会有帮助：

kubectl logs -f <isitod的pod名称> -nistio-system

peterkainj commented 3 years ago

@peterkainj 先确保网络是否有问题，可以找一个没有 sidecar 的pod ，进到容器里面安装 telnet ，然后 telnet 目标ip + 端口看看，如果istio证书有问题，可以看看 istiod 的日志，这个会有帮助：

kubectl logs -f <isitod的pod名称> -nistio-system

看istiod的日志

[root@dataplatform-1 kubeflow-manifests]# kubectl logs --tail 5 istiod-74c965dc6f-n45r7 -n istio-system 
2021-09-01T10:05:58.606175Z warn    serverca    Authentication failed for 10.244.149.247:36274: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT from cluster "Kubernetes": the service account authentication returns an error: [invalid bearer token, token audiences ["https://kubernetes.default.svc.cluster.local"] is invalid for the target audiences ["istio-ca"]]. 
2021-09-01T10:05:58.674644Z error   ads Failed to authenticate client from 10.244.15.86:47236: Authenticator ClientCertAuthenticator: no verified chain is found; Authenticator KubeJWTAuthenticator: failed to validate the JWT from cluster "Kubernetes": the service account authentication returns an error: [invalid bearer token, token audiences ["https://kubernetes.default.svc.cluster.local"] is invalid for the target audiences ["istio-ca"]]
2021-09-01T10:05:58.727997Z warn    serverca    Authentication failed for 10.244.56.31:45468: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT from cluster "Kubernetes": the service account authentication returns an error: [invalid bearer token, token audiences ["https://kubernetes.default.svc.cluster.local"] is invalid for the target audiences ["istio-ca"]]. 
2021-09-01T10:05:58.740784Z error   ads Failed to authenticate client from 10.244.149.194:59002: Authenticator ClientCertAuthenticator: no verified chain is found; Authenticator KubeJWTAuthenticator: failed to validate the JWT from cluster "Kubernetes": the service account authentication returns an error: [invalid bearer token, token audiences ["https://kubernetes.default.svc.cluster.local"] is invalid for the target audiences ["istio-ca"]]
2021-09-01T10:05:58.768051Z warn    serverca    Authentication failed for 10.244.149.194:41624: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT from cluster "Kubernetes": the service account authentication returns an error: [invalid bearer token, token audiences ["https://kubernetes.default.svc.cluster.local"] is invalid for the target audiences ["istio-ca"]].

shikanon commented 3 years ago

@peterkainj 和这个issue一样：https://github.com/istio/istio/issues/23879 试试不用patch版本的看是否可以:

kubectl delete -f manifest1.13/006-istio-1-9-0-istio-install-base.yaml
kubectl apply -f manifest1.13/006-istio-1-9-0-istio-install-base.yaml

peterkainj commented 3 years ago

@peterkainj 和这个issue一样：istio/istio#23879 试试不用patch版本的看是否可以:

确实是可以的，我看istio相关的yaml有006和013，分别delete和apply以后还是不太行，但对应pod状态是正常了；然后我又重新卸载安装了一次，这次patch时在目录里移去文件istiod.yaml，cluster-local-gateway.yaml，istio-ingressgateway.yaml，其他文件正常patch结束后，就可以正常访问并使用了。爆赞！！

shikanon commented 3 years ago

@peterkainj 嗯，那应该是你那个k8s集群不支持旧版的jwt模式了，我之前加这个istio patch就是因为 kind 老版本k8s不支持istio的这个功能。

drinktee commented 2 years ago

@peterkainj 和这个issue一样：istio/istio#23879 试试不用patch版本的看是否可以:

确实是可以的，我看istio相关的yaml有006和013，分别delete和apply以后还是不太行，但对应pod状态是正常了；然后我又重新卸载安装了一次，这次patch时在目录里移去文件istiod.yaml，cluster-local-gateway.yaml，istio-ingressgateway.yaml，其他文件正常patch结束后，就可以正常访问并使用了。爆赞！！

这个后来怎么解决呢，我也遇到了

shikanon commented 2 years ago

@peterkainj 和这个issue一样：istio/istio#23879 试试不用patch版本的看是否可以:

确实是可以的，我看istio相关的yaml有006和013，分别delete和apply以后还是不太行，但对应pod状态是正常了；然后我又重新卸载安装了一次，这次patch时在目录里移去文件istiod.yaml，cluster-local-gateway.yaml，istio-ingressgateway.yaml，其他文件正常patch结束后，就可以正常访问并使用了。爆赞！！

这个后来怎么解决呢，我也遇到了

就是删除 patch/ 文件夹下面的 istiod.yaml，cluster-local-gateway.yaml，istio-ingressgateway.yaml 这三个和istio相关的文件，因为 kind 老版本k8s不支持istio的jwt功能，如果你们不需要可以删除，这样就不会被patch了。

brness commented 1 year ago

@peterkainj 和这个issue一样：istio/istio#23879 试试不用patch版本的看是否可以:

确实是可以的，我看istio相关的yaml有006和013，分别delete和apply以后还是不太行，但对应pod状态是正常了；然后我又重新卸载安装了一次，这次patch时在目录里移去文件istiod.yaml，cluster-local-gateway.yaml，istio-ingressgateway.yaml，其他文件正常patch结束后，就可以正常访问并使用了。爆赞！！

这个后来怎么解决呢，我也遇到了

就是删除 patch/ 文件夹下面的 istiod.yaml，cluster-local-gateway.yaml，istio-ingressgateway.yaml 这三个和istio相关的文件，因为 kind 老版本k8s不支持istio的jwt功能，如果你们不需要可以删除，这样就不会被patch了。

但是只是简单的删除三个文件会导致很多的功能没法正常使用，还是说不用kind，采用其他方式搭建的k8s

shikanon / kubeflow-manifests

1.21.4版本中安装kubeflow未成功 #62