search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.28k stars 4.87k forks source link

Istio addon not working on multinode #10248

Closed timhughes closed 2 years ago

timhughes commented 3 years ago

When trying to run istio on a multi node minikube the istio-proxy containers do not start.

Steps to reproduce the issue:

  1. minikube start --nodes=4 --cpus=4 --memory=8G
  2. minikube addons enable istio-provisioner
  3. minikube addons enable istio

Full output of minikube start command used, if not already included: minikube-start-nodes4-cpus4-memory8g.log minikube-addons-enable-istio-provisioner.log minikube-addons-enable-istio.log

Full output of minikube logs command: minikube-logs.log

Full container logs from istio-operator and istio-system namespaces:

istio-operator-istio-operator-6dbfd4446f-xxqnn-1611512011468363254.log istio-system-istiod-6ccd677dc7-gmzsw-1611512018108421785.log istio-system-istio-ingressgateway-8577c95547-v25xk-1611512014880656996.log istio-system-prometheus-7767dfd55-kd9c8-1611512021807342019.log istio-system-prometheus-7767dfd55-kd9c8-1611512024337276220.log

Relevant Logs from istio-proxy container:

2021-01-24T18:11:40.046139Z    info    FLAG: --binaryPath="/usr/local/bin/envoy"
2021-01-24T18:11:40.046165Z    info    FLAG: --concurrency="0"
2021-01-24T18:11:40.046167Z    info    FLAG: --configPath="/etc/istio/proxy"
2021-01-24T18:11:40.046170Z    info    FLAG: --connectTimeout="10s"
2021-01-24T18:11:40.046172Z    info    FLAG: --controlPlaneAuthPolicy="NONE"
2021-01-24T18:11:40.046175Z    info    FLAG: --controlPlaneBootstrap="true"
2021-01-24T18:11:40.046177Z    info    FLAG: --customConfigFile=""
2021-01-24T18:11:40.046178Z    info    FLAG: --datadogAgentAddress=""
2021-01-24T18:11:40.046180Z    info    FLAG: --disableInternalTelemetry="false"
2021-01-24T18:11:40.046182Z    info    FLAG: --discoveryAddress="istio-pilot.istio-system.svc:15012"
2021-01-24T18:11:40.046184Z    info    FLAG: --dnsRefreshRate="300s"
2021-01-24T18:11:40.046191Z    info    FLAG: --domain="istio-system.svc.cluster.local"
2021-01-24T18:11:40.046193Z    info    FLAG: --drainDuration="45s"
2021-01-24T18:11:40.046195Z    info    FLAG: --envoyAccessLogService=""
2021-01-24T18:11:40.046196Z    info    FLAG: --envoyMetricsService=""
2021-01-24T18:11:40.046198Z    info    FLAG: --help="false"
2021-01-24T18:11:40.046199Z    info    FLAG: --id=""
2021-01-24T18:11:40.046201Z    info    FLAG: --ip=""
2021-01-24T18:11:40.046203Z    info    FLAG: --lightstepAccessToken=""
2021-01-24T18:11:40.046204Z    info    FLAG: --lightstepAddress=""
2021-01-24T18:11:40.046206Z    info    FLAG: --lightstepCacertPath=""
2021-01-24T18:11:40.046208Z    info    FLAG: --lightstepSecure="false"
2021-01-24T18:11:40.046210Z    info    FLAG: --log_as_json="false"
2021-01-24T18:11:40.046212Z    info    FLAG: --log_caller=""
2021-01-24T18:11:40.046214Z    info    FLAG: --log_output_level="default:info"
2021-01-24T18:11:40.046215Z    info    FLAG: --log_rotate=""
2021-01-24T18:11:40.046218Z    info    FLAG: --log_rotate_max_age="30"
2021-01-24T18:11:40.046220Z    info    FLAG: --log_rotate_max_backups="1000"
2021-01-24T18:11:40.046223Z    info    FLAG: --log_rotate_max_size="104857600"
2021-01-24T18:11:40.046225Z    info    FLAG: --log_stacktrace_level="default:none"
2021-01-24T18:11:40.046229Z    info    FLAG: --log_target="[stdout]"
2021-01-24T18:11:40.046231Z    info    FLAG: --mixerIdentity=""
2021-01-24T18:11:40.046233Z    info    FLAG: --outlierLogPath=""
2021-01-24T18:11:40.046235Z    info    FLAG: --parentShutdownDuration="1m0s"
2021-01-24T18:11:40.046237Z    info    FLAG: --pilotIdentity=""
2021-01-24T18:11:40.046239Z    info    FLAG: --proxyAdminPort="15000"
2021-01-24T18:11:40.046241Z    info    FLAG: --proxyComponentLogLevel="misc:error"
2021-01-24T18:11:40.046243Z    info    FLAG: --proxyLogLevel="warning"
2021-01-24T18:11:40.046245Z    info    FLAG: --serviceCluster="istio-ingressgateway"
2021-01-24T18:11:40.046247Z    info    FLAG: --serviceregistry="Kubernetes"
2021-01-24T18:11:40.046249Z    info    FLAG: --statsdUdpAddress=""
2021-01-24T18:11:40.046251Z    info    FLAG: --statusPort="15020"
2021-01-24T18:11:40.046253Z    info    FLAG: --stsPort="0"
2021-01-24T18:11:40.046255Z    info    FLAG: --templateFile=""
2021-01-24T18:11:40.046257Z    info    FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2021-01-24T18:11:40.046259Z    info    FLAG: --trust-domain="cluster.local"
2021-01-24T18:11:40.046261Z    info    FLAG: --zipkinAddress="zipkin.istio-system:9411"
2021-01-24T18:11:40.046283Z    info    Version 1.5.0-c3c353285578eb68b334fc8766746b754b6b3789-Clean
2021-01-24T18:11:40.046388Z    info    Obtained private IP [10.244.3.3]
2021-01-24T18:11:40.046412Z    info    Proxy role: &model.Proxy{ClusterID:"", Type:"router", IPAddresses:[]string{"10.244.3.3", "10.244.3.3"}, ID:"istio-ingressgateway-8577c95547-v25xk.istio-system", Locality:(*envoy_api_v2_core.Locality)(nil), DNSDomain:"istio-system.svc.cluster.local", ConfigNamespace:"", Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), WorkloadLabels:labels.Collection(nil), IstioVersion:(*model.IstioVersion)(nil)}
2021-01-24T18:11:40.046416Z    info    PilotSAN []string(nil)
2021-01-24T18:11:40.046418Z    info    MixerSAN []string(nil)
2021-01-24T18:11:40.046742Z    info    Effective config: binaryPath: /usr/local/bin/envoy
configPath: /etc/istio/proxy
connectTimeout: 10s
discoveryAddress: istio-pilot.istio-system.svc:15012
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: istio-ingressgateway
statNameLength: 189
tracing:
  zipkin:
    address: zipkin.istio-system:9411
2021-01-24T18:11:40.046751Z    info    JWT policy is third-party-jwt
2021-01-24T18:11:40.046806Z    info    Using user-configured CA istio-pilot.istio-system.svc:15012
2021-01-24T18:11:40.046808Z    info    istiod uses self-issued certificate
2021-01-24T18:11:40.046836Z    info    the CA cert of istiod is: -----BEGIN CERTIFICATE-----
MIIC3jCCAcagAwIBAgIRAP3mz2eowkXT6AkWrglnGJQwDQYJKoZIhvcNAQELBQAw
GDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDAeFw0yMTAxMjQxODExMDZaFw0zMTAx
MjIxODExMDZaMBgxFjAUBgNVBAoTDWNsdXN0ZXIubG9jYWwwggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQCV6xd981Y2QyhbxLohoqh3Vidxkr3V+uJd70Wg
zKJFEepQrkLx6sL27BHRqmSfB/po6CTVYmNUFHES9z+XQPQ8DFJtg446+JlwT5u9
BVLGpuskdb2dp/YdT0ZeXSAETYMOlsT6zjK3zAWATgDGYxif8yu80jwy/DvTzIDO
VRi2yIyjKTNT80G4i9M3iYjdHwGaFdFlek3ZbskhGpe75R84uXE9zR1OS+HkDTN5
FVZiENmEI62Qz6BXpmHTMX4Oy9u1nvaDFauuyxlFSdSPnm1296Pz5kesNAyB7F+r
fsTEpjMtf9NNcK6NGeHVzVAWNHqWk4tY1L1RnSflaE6TAo2HAgMBAAGjIzAhMA4G
A1UdDwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IB
AQAxMxbEqwK3pr5wzp6lTPQlXn4gZjMlUFhPlecFmo+6J9SYH07tDvY5LdAhEGE/
fVES2AHTKV7KZa5AxXCRqFqShD51l68aCyg2N1w0Mrev5H3KjT0UuxVDaJ/M09Na
m+VIT1VsPiEkPBdFj+inSYQP0BWsMBXhe1Mg9oJyXZCKzNDHYCMoxi5Vde3KGZ6L
Si+2l7kd3zwssIzgF8LaUqwcN0WC0fl/zgH/JYo+1hz/kcMGj2K6zcOjsvXAyZ/0
hi2T4ZUL/DV+1SBb1Q9hLLs1vzUsmennxuEC940fXc3krMw3bKtRtG0R8bgcWbPb
wCPIQvsx2EmSwdtSYOqof9Wv
-----END CERTIFICATE-----
2021-01-24T18:11:40.046922Z    info    parsed scheme: ""
2021-01-24T18:11:40.046927Z    info    scheme "" not registered, fallback to default scheme
2021-01-24T18:11:40.046938Z    info    ccResolverWrapper: sending update to cc: {[{istio-pilot.istio-system.svc:15012   0 }]  }
2021-01-24T18:11:40.046942Z    info    ClientConn switching balancer to "pick_first"
2021-01-24T18:11:40.047057Z    info    pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING }
2021-01-24T18:11:40.085873Z    info    Starting gateway SDS
2021-01-24T18:11:40.187466Z    info    sds    SDS gRPC server for workload UDS starts, listening on "/etc/istio/proxy/SDS" 
2021-01-24T18:11:40.187542Z    info    sds    SDS gRPC server for ingress gateway controller starts, listening on "/var/run/ingress_gateway/sds" 
2021-01-24T18:11:40.187531Z    info    sds    Start SDS grpc server
2021-01-24T18:11:40.187567Z    info    sds    Start SDS grpc server for ingress gateway proxy
2021-01-24T18:11:40.187571Z    info    PilotSAN []string{"istio-pilot.istio-system.svc"}
2021-01-24T18:11:40.187621Z    info    Starting proxy agent
2021-01-24T18:11:40.187662Z    info    Opening status port 15020
2021-01-24T18:11:40.187693Z    info    Received new config, creating new Envoy epoch 0
2021-01-24T18:11:40.187738Z    info    Epoch 0 starting
2021-01-24T18:11:41.406363Z    info    Envoy proxy is NOT ready: Get http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager.cds|listener_manager.lds).(update_success|update_rejected)$: dial tcp 127.0.0.1:15000: connect: connection refused
2021-01-24T18:11:42.189868Z    info    Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --log-format [Envoy (Epoch 0)] [%Y-%m-%d %T.%e][%t][%l][%n] %v -l warning --component-log-level misc:error]
[Envoy (Epoch 0)] [2021-01-24 18:11:42.224][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:11:42.224][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:11:43.406922Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:11:43.589][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:11:43.589][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:11:45.406869Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:11:46.011][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:11:46.011][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:11:47.406933Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:11:47.444][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:11:47.444][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:11:49.406768Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:11:51.407092Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:11:53.406907Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:11:55.406834Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:11:57.407212Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:11:59.406811Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:00.047149Z    info    grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012   0 }. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
2021-01-24T18:12:00.047245Z    info    pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"}
2021-01-24T18:12:01.047412Z    info    pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING }
2021-01-24T18:12:01.406750Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:03.406980Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:05.407071Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:12:07.274][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:12:07.274][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:12:07.406742Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:09.406889Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:11.406554Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:13.406755Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:15.406844Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:17.406537Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:19.406795Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:21.047495Z    info    grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012   0 }. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
2021-01-24T18:12:21.047542Z    info    pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"}
2021-01-24T18:12:21.406880Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:22.813360Z    info    pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING }
2021-01-24T18:12:23.406873Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:25.406651Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:27.406715Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:29.406651Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:31.406598Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:12:32.515][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:12:32.515][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:12:33.406663Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:35.407079Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:12:35.718][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:12:35.718][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:12:37.407151Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:39.406682Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:41.406570Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:12:42.660][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:12:42.660][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:12:42.813440Z    info    grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012   0 }. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
2021-01-24T18:12:42.813483Z    info    pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"}
2021-01-24T18:12:43.406825Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:45.406872Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:45.857720Z    info    pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING }
2021-01-24T18:12:47.407005Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:49.406834Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:51.406617Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:53.407196Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:55.406961Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:57.228538Z    info    sds    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default new connection
2021-01-24T18:12:57.407219Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:59.406660Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:00.903701Z    info    grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012   0 }. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout". Reconnecting...
2021-01-24T18:13:00.903785Z    info    pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"}
2021-01-24T18:13:00.903816Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"
2021-01-24T18:13:00.953958Z    warn    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 50 millisec
2021-01-24T18:13:00.954081Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"
2021-01-24T18:13:01.054248Z    warn    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 100 millisec
2021-01-24T18:13:01.054333Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"
2021-01-24T18:13:01.254467Z    warn    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 200 millisec
2021-01-24T18:13:01.254537Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"
2021-01-24T18:13:01.406843Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:01.654701Z    warn    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 400 millisec
2021-01-24T18:13:01.654778Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"
2021-01-24T18:13:02.454924Z    warn    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 800 millisec
2021-01-24T18:13:02.455019Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"
2021-01-24T18:13:03.406835Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:04.055157Z    warn    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 1600 millisec
2021-01-24T18:13:04.055244Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"
2021-01-24T18:13:04.734418Z    info    pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING }
2021-01-24T18:13:05.406606Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:13:07.019][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:13:07.019][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:13:07.255368Z    warn    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 3200 millisec
2021-01-24T18:13:07.406709Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:09.406937Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:11.406790Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:13.407097Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:15.406754Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:17.407000Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:19.406913Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:21.406575Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:23.406549Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:24.734524Z    info    grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012   0 }. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
2021-01-24T18:13:24.734584Z    info    pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"}
2021-01-24T18:13:24.734613Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:24.734637Z    error    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR retrial timed out: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:24.734668Z    error    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default failed to generate secret for proxy: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:24.734684Z    error    sds    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default Close connection. Failed to get secret for proxy "router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local" from secret cache: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:24.734770Z    info    sds    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
[Envoy (Epoch 0)] [2021-01-24 18:13:24.734][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:24.776637Z    info    sds    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default new connection
2021-01-24T18:13:24.846060Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:24.896173Z    warn    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 50 millisec
2021-01-24T18:13:24.896212Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:24.996341Z    warn    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 100 millisec
2021-01-24T18:13:24.996435Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:25.196564Z    warn    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 200 millisec
2021-01-24T18:13:25.196607Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:25.406711Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:25.596787Z    warn    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 400 millisec
2021-01-24T18:13:25.596889Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:26.397058Z    warn    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 800 millisec
2021-01-24T18:13:26.397135Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:27.406821Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:27.997277Z    warn    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 1600 millisec
2021-01-24T18:13:27.997362Z    error    citadelclient    Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:29.406690Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:31.021823Z    info    pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING }
2021-01-24T18:13:31.197481Z    warn    cache    node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 3200 millisec
2021-01-24T18:13:31.406707Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:33.406815Z    info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
RA489 commented 3 years ago

/kind support

mikala3 commented 3 years ago

I'm running into the same issue. Is this possible with istio and minikube?

medyagh commented 3 years ago

@mikala3 what error do u get ? have tried allocating more memory ? istio needs a lot of memory

timhughes commented 3 years ago

Tried again on latest minikube
minikube version: v1.18.1 commit: 09ee84d530de4a92f00f1c5dbc34cead092b95bc

kubectl logs -n istio-system pod/prometheus-7767dfd55-9wg5k -c istio-proxy

Still getting the same errors

[Envoy (Epoch 0)] [2021-03-11 15:20:56.272][20][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-03-11T15:20:57.812763Z info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-03-11T15:20:58.961102Z warn    cache   node:sidecar~10.244.3.2~prometheus-7767dfd55-9wg5k.istio-system~istio-system.svc.cluster.local-13 resource:default request:7bb64c9f-4125-45ba-bf24-0356cd717956 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.2:38023->10.96.0.10:53: i/o timeout", retry in 6400 millisec
2021-03-11T15:20:58.961207Z error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.2:38023->10.96.0.10:53: i/o timeout"
2021-03-11T15:20:58.961224Z error   cache   node:sidecar~10.244.3.2~prometheus-7767dfd55-9wg5k.istio-system~istio-system.svc.cluster.local-13 resource:default request:7bb64c9f-4125-45ba-bf24-0356cd717956 CSR retrial timed out: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.2:38023->10.96.0.10:53: i/o timeout"
2021-03-11T15:20:58.961237Z error   cache   node:sidecar~10.244.3.2~prometheus-7767dfd55-9wg5k.istio-system~istio-system.svc.cluster.local-13 resource:default failed to generate secret for proxy: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.2:38023->10.96.0.10:53: i/o timeout"
2021-03-11T15:20:58.961249Z error   sds node:sidecar~10.244.3.2~prometheus-7767dfd55-9wg5k.istio-system~istio-system.svc.cluster.local-13 resource:default Close connection. Failed to get secret for proxy "sidecar~10.244.3.2~prometheus-7767dfd55-9wg5k.istio-system~istio-system.svc.cluster.local" from secret cache: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.2:38023->10.96.0.10:53: i/o timeout"
2021-03-11T15:20:58.961300Z info    sds node:sidecar~10.244.3.2~prometheus-7767dfd55-9wg5k.istio-system~istio-system.svc.cluster.local-13 resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
[Envoy (Epoch 0)] [2021-03-11 15:20:58.961][20][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.2:38023->10.96.0.10:53: i/o timeout"
2021-03-11T15:20:59.812614Z info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-03-11T15:21:01.812450Z info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
medyagh commented 3 years ago

@timhughes hw much memory did you allocate to your Docker Desktop and minikube ?

I nticed u are using multi node --nodes=4 --cpus=4

is there a reason you are using 4 nodes ?

it is unlikely that you need multi node on local cluster, how much memory your system has ?

medyagh commented 3 years ago

/triage needs-information /kind support

timhughes commented 3 years ago

@timhughes hw much memory did you allocate to your Docker Desktop and minikube ?

I am on Linux so no need for Docker Desktop, My Local machine has 24 cores and 64GB ram and the disks are nvme rated at 3500MB/s so I am not worried about resources

I nticed u are using multi node --nodes=4 --cpus=4

is there a reason you are using 4 nodes ?

I am attempting to run rook-ceph which requires 3 worker nodes

it is unlikely that you need multi node on local cluster, how much memory your system has ?

mikala3 commented 3 years ago

@mikala3 what error do u get ? have tried allocating more memory ? istio needs a lot of memory

✔ Istio core installed                                                                                                                                                                                     
✔ Istiod installed                                                                                                                                                                                         
✘ Ingress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the conditionressgateway                                                       
Deployment/istio-system/istio-eastwestgateway
Deployment/istio-system/istio-ingressgateway
- Pruning removed resources                                                                                                                                                                                Error: failed to install manifests: errors occurred during operation

minikube start --profile=${CTX_CLUSTER1} --cpus=6 --memory=20g --driver=kvm2 --nodes=2 --addons storage-provisioner,default-storageclass,metallb

timhughes commented 3 years ago

This may just be a DNS issue.

If I kill the coredns pod and then the istio pods, it appears to fix it.

As a workaround, this is fine for me but coming up with a way if it working first go would be better

mikala3 commented 3 years ago

This may just be a DNS issue.

If I kill the coredns pod and then the istio pods, it appears to fix it.

As a workaround, this is fine for me but coming up with a way if it working first go would be better

So you are able to run with istio and minikube ( with multiple nodes)?

timhughes commented 3 years ago

Yes

On Thu, 22 Apr 2021, 17:10 mikala3, @.***> wrote:

This may just be a DNS issue.

If I kill the coredns pod and then the istio pods, it appears to fix it.

As a workaround, this is fine for me but coming up with a way if it working first go would be better

So you are able to run with istio and minikube ( with multiple nodes)?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/minikube/issues/10248#issuecomment-824978374, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJFZD77HPTJN7YBZNWY5J3TKBC7PANCNFSM4WQW24ZA .

medyagh commented 3 years ago

@timhughes I wonder if the binary in this Pr helps ? here is the link to the binary from a PR that I think might fix this issue, do you mind trying it out ?

http://storage.googleapis.com/minikube-builds/11731/minikube-linux-amd64 http://storage.googleapis.com/minikube-builds/11731/minikube-darwin-amd64 http://storage.googleapis.com/minikube-builds/11731/minikube-windows-amd64.exe

this PR by @andriyDev is doing improvements on multinode

https://github.com/kubernetes/minikube/pull/11731

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

spowelljr commented 2 years ago

Hi @timhughes, we haven't heard back from you, do you still have this issue? There isn't enough information in this issue to make it actionable, and a long enough duration has passed, so this issue is likely difficult to replicate.

I will close this issue for now but feel free to reopen when you feel ready to provide more information.