While deploying kubeflow on GCP following the instructions everything works well until the make apply part of the kubeflow installation
serviceaccount/kserve-models-web-app created error when creating "build/apps_v1_deployment_kserve-models-web-app.yaml": Post "https://34.155.15.145/apis/apps/v1/namespaces/kubeflow/deployments?fieldManager=kubectl-client-side-apply&fieldValidation=Strict": http2: client connection lost error when retrieving current configuration of: Resource: "cert-manager.io/v1, Resource=certificates", GroupVersionKind: "cert-manager.io/v1, Kind=Certificate" Name: "serving-cert", Namespace: "kubeflow" from server for: "build/cert-manager.io_v1_certificate_serving-cert.yaml": Get "https://34.155.15.145/apis/cert-manager.io/v1/namespaces/kubeflow/certificates/serving-cert": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "cert-manager.io/v1, Resource=issuers", GroupVersionKind: "cert-manager.io/v1, Kind=Issuer" Name: "selfsigned-issuer", Namespace: "kubeflow" from server for: "build/cert-manager.io_v1_issuer_selfsigned-issuer.yaml": Get "https://34.155.15.145/apis/cert-manager.io/v1/namespaces/kubeflow/issuers/selfsigned-issuer": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "networking.istio.io/v1beta1, Resource=virtualservices", GroupVersionKind: "networking.istio.io/v1beta1, Kind=VirtualService" Name: "kserve-models-web-app", Namespace: "kubeflow" from server for: "build/networking.istio.io_v1beta1_virtualservice_kserve-models-web-app.yaml": Get "https://34.155.15.145/apis/networking.istio.io/v1beta1/namespaces/kubeflow/virtualservices/kserve-models-web-app": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "kserve-manager-role", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrole_kserve-manager-role.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterroles/kserve-manager-role": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "kserve-models-web-app-cluster-role", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrole_kserve-models-web-app-cluster-role.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterroles/kserve-models-web-app-cluster-role": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "kserve-proxy-role", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrole_kserve-proxy-role.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterroles/kserve-proxy-role": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "kubeflow-kserve-admin", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrole_kubeflow-kserve-admin.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterroles/kubeflow-kserve-admin": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "kubeflow-kserve-edit", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrole_kubeflow-kserve-edit.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterroles/kubeflow-kserve-edit": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "kubeflow-kserve-view", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrole_kubeflow-kserve-view.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterroles/kubeflow-kserve-view": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterrolebindings", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" Name: "kserve-manager-rolebinding", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrolebinding_kserve-manager-rolebinding.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/kserve-manager-rolebinding": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterrolebindings", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" Name: "kserve-models-web-app-binding", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrolebinding_kserve-models-web-app-binding.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/kserve-models-web-app-binding": net/http: TLS handshake timeout Error from server (InternalError): error when creating "build/security.istio.io_v1beta1_authorizationpolicy_kserve-models-web-app.yaml": Internal error occurred: failed calling webhook "validation.istio.io": failed to call webhook: Post "https://istiod-asm-1162-2.istio-system.svc:443/validate?timeout=10s": No agent available make[1]: *** [Makefile:19: apply] Error 1 make[1]: Leaving directory '/home/<CLOUD_SHELL_HOME_DIRECTORY>/kubeflow-distribution/kubeflow/contrib/kserve' make: *** [Makefile:83: apply] Error 1
When looking at deployments, i have 5 of them on the management cluster, and 45 of them in the kubeflow cluster, with one of them being on a CrashLoopBackOff (workflow-controller). I'm assuming it is in this state due to the installation process not being finished properly and it probably has a dependency not properly set?
Everything is ran from the cloud shell home directory on GCP, so i'm not sure why it's getting a bunch of TLS handhsake timeouts
While deploying kubeflow on GCP following the instructions everything works well until the
make applypart of the kubeflow installationserviceaccount/kserve-models-web-app created error when creating "build/apps_v1_deployment_kserve-models-web-app.yaml": Post "https://34.155.15.145/apis/apps/v1/namespaces/kubeflow/deployments?fieldManager=kubectl-client-side-apply&fieldValidation=Strict": http2: client connection lost error when retrieving current configuration of: Resource: "cert-manager.io/v1, Resource=certificates", GroupVersionKind: "cert-manager.io/v1, Kind=Certificate" Name: "serving-cert", Namespace: "kubeflow" from server for: "build/cert-manager.io_v1_certificate_serving-cert.yaml": Get "https://34.155.15.145/apis/cert-manager.io/v1/namespaces/kubeflow/certificates/serving-cert": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "cert-manager.io/v1, Resource=issuers", GroupVersionKind: "cert-manager.io/v1, Kind=Issuer" Name: "selfsigned-issuer", Namespace: "kubeflow" from server for: "build/cert-manager.io_v1_issuer_selfsigned-issuer.yaml": Get "https://34.155.15.145/apis/cert-manager.io/v1/namespaces/kubeflow/issuers/selfsigned-issuer": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "networking.istio.io/v1beta1, Resource=virtualservices", GroupVersionKind: "networking.istio.io/v1beta1, Kind=VirtualService" Name: "kserve-models-web-app", Namespace: "kubeflow" from server for: "build/networking.istio.io_v1beta1_virtualservice_kserve-models-web-app.yaml": Get "https://34.155.15.145/apis/networking.istio.io/v1beta1/namespaces/kubeflow/virtualservices/kserve-models-web-app": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "kserve-manager-role", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrole_kserve-manager-role.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterroles/kserve-manager-role": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "kserve-models-web-app-cluster-role", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrole_kserve-models-web-app-cluster-role.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterroles/kserve-models-web-app-cluster-role": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "kserve-proxy-role", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrole_kserve-proxy-role.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterroles/kserve-proxy-role": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "kubeflow-kserve-admin", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrole_kubeflow-kserve-admin.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterroles/kubeflow-kserve-admin": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "kubeflow-kserve-edit", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrole_kubeflow-kserve-edit.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterroles/kubeflow-kserve-edit": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "kubeflow-kserve-view", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrole_kubeflow-kserve-view.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterroles/kubeflow-kserve-view": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterrolebindings", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" Name: "kserve-manager-rolebinding", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrolebinding_kserve-manager-rolebinding.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/kserve-manager-rolebinding": net/http: TLS handshake timeout error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterrolebindings", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" Name: "kserve-models-web-app-binding", Namespace: "" from server for: "build/rbac.authorization.k8s.io_v1_clusterrolebinding_kserve-models-web-app-binding.yaml": Get "https://34.155.15.145/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/kserve-models-web-app-binding": net/http: TLS handshake timeout Error from server (InternalError): error when creating "build/security.istio.io_v1beta1_authorizationpolicy_kserve-models-web-app.yaml": Internal error occurred: failed calling webhook "validation.istio.io": failed to call webhook: Post "https://istiod-asm-1162-2.istio-system.svc:443/validate?timeout=10s": No agent available make[1]: *** [Makefile:19: apply] Error 1 make[1]: Leaving directory '/home/<CLOUD_SHELL_HOME_DIRECTORY>/kubeflow-distribution/kubeflow/contrib/kserve' make: *** [Makefile:83: apply] Error 1When looking at deployments, i have 5 of them on the management cluster, and 45 of them in the kubeflow cluster, with one of them being on a CrashLoopBackOff (workflow-controller). I'm assuming it is in this state due to the installation process not being finished properly and it probably has a dependency not properly set?
Everything is ran from the cloud shell home directory on GCP, so i'm not sure why it's getting a bunch of TLS handhsake timeouts