I have followed the instruction for setting up kubeflow on minikube on linux (ubuntu 20.04)
I can directly access the central dashboard but my ingress gateway is not working.
It refuses to connect and logs point to authentication failures.
Here are the logs:
2021-01-12T20:50:45.819257Z info FLAG: --connectTimeout="10s"
2021-01-12T20:50:45.819259Z info FLAG: --controlPlaneAuthPolicy="MUTUAL_TLS"
2021-01-12T20:50:45.819263Z info FLAG: --controlPlaneBootstrap="true"
2021-01-12T20:50:45.819266Z info FLAG: --customConfigFile=""
2021-01-12T20:50:45.819268Z info FLAG: --datadogAgentAddress=""
2021-01-12T20:50:45.819271Z info FLAG: --disableInternalTelemetry="false"
2021-01-12T20:50:45.819274Z info FLAG: --discoveryAddress="istio-pilot:15011"
2021-01-12T20:50:45.819276Z info FLAG: --dnsRefreshRate="300s"
2021-01-12T20:50:45.819288Z info FLAG: --domain="istio-system.svc.cluster.local"
2021-01-12T20:50:45.819293Z info FLAG: --drainDuration="45s"
2021-01-12T20:50:45.819295Z info FLAG: --envoyAccessLogService=""
2021-01-12T20:50:45.819298Z info FLAG: --envoyMetricsServiceAddress=""
2021-01-12T20:50:45.819301Z info FLAG: --help="false"
2021-01-12T20:50:45.819303Z info FLAG: --id=""
2021-01-12T20:50:45.819309Z info FLAG: --ip=""
2021-01-12T20:50:45.819312Z info FLAG: --lightstepAccessToken=""
2021-01-12T20:50:45.819315Z info FLAG: --lightstepAddress=""
2021-01-12T20:50:45.819317Z info FLAG: --lightstepCacertPath=""
2021-01-12T20:50:45.819320Z info FLAG: --lightstepSecure="false"
2021-01-12T20:50:45.819323Z info FLAG: --log_as_json="false"
2021-01-12T20:50:45.819326Z info FLAG: --log_caller=""
2021-01-12T20:50:45.819328Z info FLAG: --log_output_level="default:info"
2021-01-12T20:50:45.819331Z info FLAG: --log_rotate=""
2021-01-12T20:50:45.819334Z info FLAG: --log_rotate_max_age="30"
2021-01-12T20:50:45.819337Z info FLAG: --log_rotate_max_backups="1000"
2021-01-12T20:50:45.819340Z info FLAG: --log_rotate_max_size="104857600"
2021-01-12T20:50:45.819342Z info FLAG: --log_stacktrace_level="default:none"
2021-01-12T20:50:45.819349Z info FLAG: --log_target="[stdout]"
2021-01-12T20:50:45.819352Z info FLAG: --mixerIdentity=""
2021-01-12T20:50:45.819355Z info FLAG: --parentShutdownDuration="1m0s"
2021-01-12T20:50:45.819357Z info FLAG: --pilotIdentity=""
2021-01-12T20:50:45.819361Z info FLAG: --proxyAdminPort="15000"
2021-01-12T20:50:45.819364Z info FLAG: --proxyComponentLogLevel="misc:error"
2021-01-12T20:50:45.819367Z info FLAG: --proxyLogLevel="warning"
2021-01-12T20:50:45.819370Z info FLAG: --serviceCluster="istio-ingressgateway"
2021-01-12T20:50:45.819373Z info FLAG: --serviceregistry="Kubernetes"
2021-01-12T20:50:45.819375Z info FLAG: --statsdUdpAddress=""
2021-01-12T20:50:45.819379Z info FLAG: --statusPort="15020"
2021-01-12T20:50:45.819382Z info FLAG: --templateFile=""
2021-01-12T20:50:45.819384Z info FLAG: --trust-domain=""
2021-01-12T20:50:45.819387Z info FLAG: --zipkinAddress="zipkin:9411"
2021-01-12T20:50:45.819403Z info Version root@07253e1c-4f13-11ea-97de-0e8f370df5e1-gcr.io/istio-release-release-1.3-20200214-10-15-3db95dfc23ffc081803b42549934915ba3b0a3d5-Clean
2021-01-12T20:50:45.819487Z info Obtained private IP [REDACTED]
2021-01-12T20:50:45.819527Z info Proxy role: &model.Proxy{ClusterID:"", Type:"router", IPAddresses:[]string{"REDACTED", "REDACTED"}, ID:"istio-ingressgateway-7bb77f89b8-fmkmm.istio-system", Locality:(*core.Locality)(nil), DNSDomain:"istio-system.svc.cluster.local", TrustDomain:"cluster.local", PilotIdentity:"", MixerIdentity:"", ConfigNamespace:"", Metadata:map[string]string{}, SidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), WorkloadLabels:labels.Collection(nil), IstioVersion:(*model.IstioVersion)(nil)}
2021-01-12T20:50:45.819540Z info PilotSAN []string{"spiffe://cluster.local/ns/istio-system/sa/istio-pilot-service-account"}
2021-01-12T20:50:45.819936Z info Effective config: binaryPath: /usr/local/bin/envoy
configPath: /etc/istio/proxy
connectTimeout: 10s
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istio-pilot:15011
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: istio-ingressgateway
statNameLength: 189
tracing:
zipkin:
address: zipkin:9411
2021-01-12T20:50:45.819966Z info waiting 1m0s for /var/run/sds/uds_path
2021-01-12T20:50:45.819991Z info PilotSAN []string{"spiffe://cluster.local/ns/istio-system/sa/istio-pilot-service-account"}
2021-01-12T20:50:45.820133Z info Starting proxy agent
2021-01-12T20:50:45.820225Z info Opening status port 15020
2021-01-12T20:50:45.820313Z info watching /etc/certs for changes
2021-01-12T20:50:45.820316Z info Received new config, resetting budget
2021-01-12T20:50:45.820386Z info Reconciling retry (budget 10)
2021-01-12T20:50:45.820495Z info Epoch 0 starting
2021-01-12T20:50:45.846274Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~REDACTED~istio-ingressgateway-7bb77f89b8-fmkmm.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --allow-unknown-fields -l warning --component-log-level misc:error]
[2021-01-12 20:50:45.860][26][warning][config] [external/envoy/source/server/options_impl.cc:193] --allow-unknown-fields is deprecated, use --allow-unknown-static-fields instead.
[2021-01-12 20:50:45.925][26][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, no healthy upstream
[2021-01-12 20:50:45.925][26][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:50] Unable to establish new stream
[2021-01-12 20:50:46.432][26][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure
[2021-01-12 20:50:46.935][26][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 2, failed to get root cert
[2021-01-12 20:50:46.951][26][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
[2021-01-12 20:50:46.951][26][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure
[2021-01-12 20:50:47.297][26][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure
[2021-01-12 20:50:47.927][26][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure
[2021-01-12 20:50:48.098][26][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
[2021-01-12 20:50:48.291][26][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 2, failed to get root cert
2021-01-12T20:50:48.466710Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[2021-01-12 20:50:49.151][26][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure
[2021-01-12 20:50:49.291][26][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure
[2021-01-12 20:50:49.329][26][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 2, failed to get root cert
[2021-01-12 20:50:50.138][26][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2021-01-12T20:50:50.466306Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
I have followed the instruction for setting up kubeflow on minikube on linux (ubuntu 20.04) I can directly access the central dashboard but my ingress gateway is not working. It refuses to connect and logs point to authentication failures.
Here are the logs: