search

istio / old_issues_repo

Deprecated issue-tracking repo, please post new issues or feature requests to istio/istio instead.
36 stars 9 forks source link

Pod becomes unreachable after istio installation. #152

Open kbalbi opened 6 years ago

kbalbi commented 6 years ago

Is this a BUG or FEATURE REQUEST?: I do not know is this a bug or mis-configuration. I doubt mis-configuration because I did not do any customization just installed the istio with the standard yaml mentioned in the site (istio.io).

Did you review existing epics or issues to identify if this already being worked on? (please try to add the correct labels and epics):

This looks similar to the issue of 147 but with a different erro message.

Bug: Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version :  0.3.0
kubectl version : 1.8.5

Is Istio Auth enabled or not ? No Did you install istio.yaml, istio-auth.yaml.... yep, installed only istio-all.yaml

What happened: After installation, the service become inaccessible. i was trying to troubleshoot and find the following logs on the envoy (istio-proxy).

========================

ERROR: logging before flag.Parse: I0105 21:00:57.213727 1 main.go:68] Version root@24c944bda24b-0.3.0-24ec6a3ac3a1d592d1873d2d8198278a849b8301 ERROR: logging before flag.Parse: I0105 21:00:57.216302 1 main.go:109] Proxy role: proxy.Node{Type:"sidecar", IPAddress:"10.233.70.14", ID:"my-service-64d44fdb49-j45vk.com-my-own-dev", Domain:"com-my-own-dev.svc.cluster.local"} ERROR: logging before flag.Parse: I0105 21:00:57.217070 1 resolve.go:35] Attempting to lookup address: istio-mixer.istio-system ERROR: logging before flag.Parse: I0105 21:00:57.230458 1 resolve.go:45] Addr resolved to: 10.233.7.3:9125 ERROR: logging before flag.Parse: I0105 21:00:57.230546 1 resolve.go:46] Finished lookup of address: istio-mixer.istio-system ERROR: logging before flag.Parse: I0105 21:00:57.232351 1 main.go:169] Effective config: binaryPath: /usr/local/bin/envoy configPath: /etc/istio/proxy connectTimeout: 10.000s discoveryAddress: istio-pilot.istio-system:15003 discoveryRefreshDelay: 1.000s drainDuration: 45.000s parentShutdownDuration: 60.000s proxyAdminPort: 15000 serviceCluster: my-service statsdUdpAddress: 10.233.7.3:9125 zipkinAddress: zipkin.istio-system:9411 ERROR: logging before flag.Parse: I0105 21:00:57.232403 1 main.go:186] Monitored certs: []envoy.CertSource{envoy.CertSource{Directory:"/etc/certs/", Files:[]string{"cert-chain.pem", "key.pem", "root-cert.pem"}}} ERROR: logging before flag.Parse: I0105 21:00:57.234737 1 agent.go:164] Starting proxy agent ERROR: logging before flag.Parse: I0105 21:00:57.235171 1 agent.go:186] Received new config, resetting budget ERROR: logging before flag.Parse: I0105 21:00:57.235232 1 agent.go:261] Reconciling configuration (budget 10) ERROR: logging before flag.Parse: I0105 21:00:57.235672 1 agent.go:281] Epoch 0 starting ERROR: logging before flag.Parse: I0105 21:00:57.235787 1 config.go:49] writing configuration to /etc/istio/proxy/envoy-rev0.json { "listeners": [], "lds": { "cluster": "lds", "refresh_delay_ms": 1000 }, "admin": { "access_log_path": "/dev/stdout", "address": "tcp://127.0.0.1:15000" }, "cluster_manager": { "clusters": [ { "name": "rds", "connect_timeout_ms": 10000, "type": "strict_dns", "lb_type": "round_robin", "hosts": [ { "url": "tcp://istio-pilot.istio-system:15003" } ] }, { "name": "lds", "connect_timeout_ms": 10000, "type": "strict_dns", "lb_type": "round_robin", "hosts": [ { "url": "tcp://istio-pilot.istio-system:15003" } ] }, { "name": "zipkin", "connect_timeout_ms": 10000, "type": "strict_dns", "lb_type": "round_robin", "hosts": [ { "url": "tcp://zipkin.istio-system:9411" } ] } ], "sds": { "cluster": { "name": "sds", "connect_timeout_ms": 10000, "type": "strict_dns", "lb_type": "round_robin", "hosts": [ { "url": "tcp://istio-pilot.istio-system:15003" } ] }, "refresh_delay_ms": 1000 }, "cds": { "cluster": { "name": "cds", "connect_timeout_ms": 10000, "type": "strict_dns", "lb_type": "round_robin", "hosts": [ { "url": "tcp://istio-pilot.istio-system:15003" } ] }, "refresh_delay_ms": 1000 } }, "statsd_udp_ip_address": "10.233.7.3:9125", "tracing": { "http": { "driver": { "type": "zipkin", "config": { "collector_cluster": "zipkin", "collector_endpoint": "/api/v1/spans" } } } } }ERROR: logging before flag.Parse: I0105 21:00:57.239063 1 watcher.go:270] Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster my-service --service-node sidecar~10.233.70.14~my-service-64d44fdb49-j45vk.com-my-own-dev~com-my-own-dev.svc.cluster.local --max-obj-name-len 189] [2018-01-05 21:00:57.366][17][warning][main] external/envoy/source/server/server.cc:157] initializing epoch 0 (hot restart version=9.200.16384.256) [2018-01-05 21:00:57.724][17][warning][main] external/envoy/source/server/server.cc:344] starting main dispatch loop [2018-01-05 21:01:00.070][17][warning][main] external/envoy/source/server/server.cc:328] all clusters initialized. initializing init manager [2018-01-05 21:01:00.599][17][warning][config] external/envoy/source/server/listener_manager_impl.cc:448] all dependencies initialized. starting workers [2018-01-05 21:02:00.590][17][warning][main] external/envoy/source/server/drain_manager_impl.cc:62] shutting down parent after drain ERROR: logging before flag.Parse: I0105 21:07:03.280126 1 watcher.go:123] watchFileEvents: "/etc/certs//..1981_05_01_13_07_03.283210190": CREATE ERROR: logging before flag.Parse: I0105 21:07:03.280513 1 watcher.go:123] watchFileEvents: "/etc/certs//..1981_05_01_13_07_03.283210190": MODIFY|ATTRIB ERROR: logging before flag.Parse: I0105 21:07:03.280563 1 watcher.go:123] watchFileEvents: "/etc/certs//..data_tmp": RENAME ERROR: logging before flag.Parse: I0105 21:07:03.280720 1 watcher.go:123] watchFileEvents: "/etc/certs//..data": CREATE ERROR: logging before flag.Parse: I0105 21:07:03.280779 1 watcher.go:123] watchFileEvents: "/etc/certs//..1981_05_01_13_00_50.960532478": DELETE ERROR: logging before flag.Parse: I0105 21:07:13.280731 1 watcher.go:136] watchFileEvents: notifying ERROR: logging before flag.Parse: I0105 21:07:13.281745 1 agent.go:186] Received new config, resetting budget ERROR: logging before flag.Parse: I0105 21:07:13.281786 1 agent.go:261] Reconciling configuration (budget 10) ERROR: logging before flag.Parse: I0105 21:07:13.282030 1 agent.go:281] Epoch 1 starting ERROR: logging before flag.Parse: I0105 21:07:13.282094 1 config.go:49] writing configuration to /etc/istio/proxy/envoy-rev1.json { "listeners": [], "lds": { "cluster": "lds", "refresh_delay_ms": 1000 }, "admin": { "access_log_path": "/dev/stdout", "address": "tcp://127.0.0.1:15000" }, "cluster_manager": { "clusters": [ { "name": "rds", "connect_timeout_ms": 10000, "type": "strict_dns", "lb_type": "round_robin", "hosts": [ { "url": "tcp://istio-pilot.istio-system:15003" } ] }, { "name": "lds", "connect_timeout_ms": 10000, "type": "strict_dns", "lb_type": "round_robin", "hosts": [ { "url": "tcp://istio-pilot.istio-system:15003" } ] }, { "name": "zipkin", "connect_timeout_ms": 10000, "type": "strict_dns", "lb_type": "round_robin", "hosts": [ { "url": "tcp://zipkin.istio-system:9411" } ] } ], "sds": { "cluster": { "name": "sds", "connect_timeout_ms": 10000, "type": "strict_dns", "lb_type": "round_robin", "hosts": [ { "url": "tcp://istio-pilot.istio-system:15003" } ] }, "refresh_delay_ms": 1000 }, "cds": { "cluster": { "name": "cds", "connect_timeout_ms": 10000, "type": "strict_dns", "lb_type": "round_robin", "hosts": [ { "url": "tcp://istio-pilot.istio-system:15003" } ] }, "refresh_delay_ms": 1000 } }, "statsd_udp_ip_address": "10.233.7.3:9125", "tracing": { "http": { "driver": { "type": "zipkin", "config": { "collector_cluster": "zipkin", "collector_endpoint": "/api/v1/spans" } } } } }ERROR: logging before flag.Parse: I0105 21:07:13.282872 1 watcher.go:270] Envoy command: [-c /etc/istio/proxy/envoy-rev1.json --restart-epoch 1 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster my-service --service-node sidecar~10.233.70.14~my-service-64d44fdb49-j45vk.com-my-own-dev~com-my-own-dev.svc.cluster.local --max-obj-name-len 189] [2018-01-05 21:07:13.318][27][warning][main] external/envoy/source/server/server.cc:157] initializing epoch 1 (hot restart version=9.200.16384.256) [2018-01-05 21:07:13.356][17][warning][main] external/envoy/source/server/server.cc:381] shutting down admin due to child startup [2018-01-05 21:07:13.356][17][warning][main] external/envoy/source/server/server.cc:386] terminating parent process [2018-01-05 21:07:13.965][27][warning][main] external/envoy/source/server/server.cc:344] starting main dispatch loop [2018-01-05 21:07:16.163][27][warning][main] external/envoy/source/server/server.cc:328] all clusters initialized. initializing init manager [2018-01-05 21:07:16.609][27][warning][config] external/envoy/source/server/listener_manager_impl.cc:448] all dependencies initialized. starting workers [2018-01-05 21:07:16.610][17][warning][main] external/envoy/source/server/server.cc:91] closing and draining listeners [2018-01-05 21:08:16.598][27][warning][main] external/envoy/source/server/drain_manager_impl.cc:62] shutting down parent after drain [2018-01-05 21:08:16.598][17][warning][main] external/envoy/source/server/hot_restart_impl.cc:420] shutting down due to child request [2018-01-05 21:08:16.598][17][warning][main] external/envoy/source/server/server.cc:303] caught SIGTERM [2018-01-05 21:08:16.598][17][warning][main] external/envoy/source/server/server.cc:348] main dispatch loop exited [2018-01-05 21:08:16.627][17][warning][main] external/envoy/source/server/server.cc:369] exiting ERROR: logging before flag.Parse: I0105 21:08:16.642212 1 agent.go:211] Epoch 0 exited normally [2018-01-05 21:31:16.777][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

[2018-01-05 21:31:18.673][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

[2018-01-05 21:31:20.640][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

[2018-01-05 21:31:21.944][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

[2018-01-05 21:31:23.766][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

[2018-01-05 21:31:25.382][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

[2018-01-05 21:31:27.140][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

[2018-01-05 21:31:28.590][27][warning][upstream] external/envoy/source/common/router/rds_subscription.cc:65] rds: fetch failure: JSON supplied is not valid. Error(offset 4, line 1): Terminate parsing due to Handler error.

===================================

What you expected to happen: Service is expected to be reachable after creating service mesh with istio.

How to reproduce it: Nothing done, it is a clean installation of istio on the existing system.

sandylss commented 6 years ago

I also have the same problem.Do anyone know what the problem is?

louiscryan commented 6 years ago

This is pretty stale. Can you retest with 0.8 ?