search

kubeflow / kfctl

kfctl is a CLI for deploying and managing Kubeflow
Apache License 2.0
181 stars 138 forks source link

How to enable Envoy proxy on istio-ingress gateway? #339

Open sakaia opened 4 years ago

sakaia commented 4 years ago

I am try to deploy kubeflow on kind. (with Ubuntu 20.04 on AWS EC2) But the istio-ingressgateway is not working with Envoy proxy is NOT ready Is there any way to solve this?

  1. The log (istio-ingressgateway) is attached here.
kubectl logs  istio-ingressgateway-589ff776dd-tp6tk -n istio-system

log.txt

  1. The instruction for deploying is here.

kubeflow_deploy.txt

  1. Current pods status is here 
NAMESPACE            NAME                                                           READY   STATUS             RESTARTS   AGE
cert-manager         cert-manager-5d849b9888-vhg65                                  1/1     Running            0          20m
cert-manager         cert-manager-cainjector-dccb4d7f-rj8mg                         1/1     Running            0          20m
cert-manager         cert-manager-webhook-695df7dbb-qsm5g                           1/1     Running            0          20m
istio-system         cluster-local-gateway-7bf56777fb-fd4rq                         0/1     Running            0          20m
istio-system         grafana-86f89dbd84-nrjvl                                       1/1     Running            0          20m
istio-system         istio-citadel-74966f47d6-tpvc8                                 1/1     Running            0          20m
istio-system         istio-cleanup-secrets-1.1.6-4vj4n                              0/1     Completed          0          20m
istio-system         istio-egressgateway-5c64d575bc-5zdfq                           0/1     Running            0          20m
istio-system         istio-galley-784b9f6d75-p97cc                                  1/1     Running            0          20m
istio-system         istio-grafana-post-install-1.1.6-k9bhp                         0/1     Completed          0          20m
istio-system         istio-ingressgateway-589ff776dd-tp6tk                          0/1     Running            0          20m
istio-system         istio-pilot-677df6b6d4-m2h2f                                   1/2     Running            0          20m
istio-system         istio-policy-6f74d9d95d-r9v6q                                  1/2     CrashLoopBackOff   11         20m
istio-system         istio-security-post-install-1.1.6-gsdm7                        0/1     Completed          0          20m
istio-system         istio-sidecar-injector-866f4b98c7-7nsrb                        1/1     Running            0          20m
istio-system         istio-telemetry-549c8f9dcb-cktf4                               1/2     CrashLoopBackOff   11         20m
istio-system         istio-tracing-555cf644d-mj8lq                                  1/1     Running            0          20m
istio-system         kfserving-ingressgateway-64c7bd9b76-7t8ql                      0/1     Running            0          20m
istio-system         kiali-7db44d6dfb-xj4wj                                         1/1     Running            0          20m
istio-system         prometheus-d44645598-dc78l                                     1/1     Running            0          20m
knative-serving      activator-6dc4884-vd58z                                        0/2     CrashLoopBackOff   7          17m
knative-serving      autoscaler-69bcc99c79-jcsrx                                    0/2     CrashLoopBackOff   6          17m
knative-serving      autoscaler-hpa-68cc87bfb9-98xm4                                1/1     Running            0          17m
knative-serving      controller-95dc7f8bd-2dbdr                                     1/1     Running            0          17m
knative-serving      networking-istio-5b8c5c6cff-dqk88                              1/1     Running            0          17m
knative-serving      webhook-67847fb4b5-zf64k                                       1/1     Running            0          17m
kube-system          coredns-5d4dd4b4db-hrwd4                                       0/1     CrashLoopBackOff   10         26m
kube-system          coredns-5d4dd4b4db-jhzkr                                       0/1     CrashLoopBackOff   9          26m
kube-system          etcd-kind-control-plane                                        1/1     Running            0          25m
kube-system          kindnet-mt88h                                                  1/1     Running            0          26m
kube-system          kube-apiserver-kind-control-plane                              1/1     Running            0          25m
kube-system          kube-controller-manager-kind-control-plane                     1/1     Running            0          25m
kube-system          kube-proxy-jcrfh                                               1/1     Running            0          26m
kube-system          kube-scheduler-kind-control-plane                              1/1     Running            0          25m
kubeflow             admission-webhook-bootstrap-stateful-set-0                     1/1     Running            0          17m
kubeflow             admission-webhook-deployment-569558c8b6-ssl7w                  1/1     Running            0          16m
kubeflow             application-controller-stateful-set-0                          1/1     Running            0          20m
kubeflow             argo-ui-7ffb9b6577-vf88h                                       1/1     Running            0          17m
kubeflow             centraldashboard-659bd78c-87lhx                                1/1     Running            0          17m
kubeflow             jupyter-web-app-deployment-679d5f5dc4-lwvc4                    1/1     Running            0          17m
kubeflow             katib-controller-7f58569f7d-wnrtb                              1/1     Running            1          17m
kubeflow             katib-db-manager-54b66f9f9d-lgg2f                              0/1     Running            6          17m
kubeflow             katib-mysql-dcf7dcbd5-dt4xg                                    1/1     Running            0          17m
kubeflow             katib-ui-6f97756598-mmfzx                                      1/1     Running            0          17m
kubeflow             kfserving-controller-manager-0                                 2/2     Running            1          17m
kubeflow             metacontroller-0                                               1/1     Running            0          17m
kubeflow             metadata-db-65fb5b695d-qv5ss                                   1/1     Running            0          17m
kubeflow             metadata-deployment-65ccddfd4c-xgkfv                           0/1     Running            0          17m
kubeflow             metadata-envoy-deployment-7754f56bff-gn6z4                     1/1     Running            0          17m
kubeflow             metadata-grpc-deployment-5c6db9749-88nj8                       0/1     CrashLoopBackOff   6          17m
kubeflow             metadata-ui-7c85545947-sqnt4                                   1/1     Running            0          17m
kubeflow             minio-6b67f98977-lmktn                                         1/1     Running            0          17m
kubeflow             ml-pipeline-6cf777c7bc-kmplt                                   1/1     Running            1          17m
kubeflow             ml-pipeline-ml-pipeline-visualizationserver-6d744dd449-lzb6n   1/1     Running            0          17m
kubeflow             ml-pipeline-persistenceagent-5c549847fd-kwwp5                  1/1     Running            4          17m
kubeflow             ml-pipeline-scheduledworkflow-674777d89c-x4n9v                 1/1     Running            0          17m
kubeflow             ml-pipeline-ui-549b5b6744-bvch9                                1/1     Running            0          17m
kubeflow             ml-pipeline-viewer-controller-deployment-fc7f7cb65-swfqk       1/1     Running            0          17m
kubeflow             mysql-85bc64f5c4-lxjg6                                         1/1     Running            0          17m
kubeflow             notebook-controller-deployment-5c55f5845b-b2d6j                1/1     Running            0          17m
kubeflow             profiles-deployment-c775584c7-qksqm                            2/2     Running            0          17m
kubeflow             pytorch-operator-cf8c5c497-mmlsf                               1/1     Running            0          17m
kubeflow             seldon-controller-manager-6b4b969447-8zw65                     1/1     Running            0          17m
kubeflow             spark-operatorsparkoperator-76dd5f5688-6xzhb                   1/1     Running            0          17m
kubeflow             spartakus-volunteer-5dc96f4447-mkzvl                           1/1     Running            0          17m
kubeflow             tensorboard-5f685f9d79-8jvss                                   1/1     Running            0          17m
kubeflow             tf-job-operator-5fb85c5fb7-665l7                               1/1     Running            0          17m
kubeflow             workflow-controller-689d6c8846-ltt4r                           1/1     Running            0          17m
local-path-storage   local-path-provisioner-d4947b89c-858f7                         1/1     Running            0          26m
issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the labels:

Label Probability
kind/question 0.87
area/kfctl 0.61
area/inference 0.61

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

jlewi commented 4 years ago

If you are using AWS why not run on EKS?

It looks like the problem is with installing ISTIO on your system and not Kubeflow per se. You might have better luck trying a well supported combination of K8s and ISTIO and ensuring that is deployed correctly.