search

istio / istio

Connect, secure, control, and observe services.
https://istio.io
Apache License 2.0
35.95k stars 7.76k forks source link

virtualservice and Ingressgateway not work using consul platform registry #7492

Closed wansuiye closed 5 years ago

wansuiye commented 6 years ago

Describe the bug A clear and concise description of what the bug is.

I edit the pilot deployment as blew, thus can use consul registry on kubernetes

         - name: discovery
          image: "docker.io/istio/pilot:0.8.0"
          imagePullPolicy: IfNotPresent
          args:
          - "discovery"
          - "--consulserverURL"
          - "10.20.30.40:8500"
          - "--registries"
          - "Consul,Kubernetes"

services can be accessed by domain names（eg xxx.service.consul, however ingressgateway and virtualservice not work.

virtualservice :

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: hello
spec:
  hosts:
  - "hello.abc.io"
  gateways:
  - hello-gateway
  - mesh
  http:
  - route:
    - destination:
         host: hello.service.consul

when i curl in the pod, proxy log:

[2018-07-30T09:49:49.028Z] "GET /hello HTTP/1.1" 404 NR 0 0 0 - "-" "curl/7.35.0" "9234ecfa-8a13-94e7-bbae-d36a1c2abe5e" "hello.abc.io" "-"

ingresgateway seems to be fail, but still live, start log as below. when i access outside, "Connection refused" ：

[2018-07-30 09:45:42.665][136][info][main] external/envoy/source/server/server.cc:376] all clusters initialized. initializing init manager
[2018-07-30 09:45:42.668][136][info][config] external/envoy/source/server/listener_manager_impl.cc:781] all dependencies initialized. starting workers
[2018-07-30 09:45:42.716][35][info][main] external/envoy/source/server/server.cc:97] closing and draining listeners
[2018-07-30 09:46:42.668][136][info][main] external/envoy/source/server/drain_manager_impl.cc:63] shutting down parent after drain
[2018-07-30 09:46:42.668][35][info][main] external/envoy/source/server/hot_restart_impl.cc:435] shutting down due to child request
[2018-07-30 09:46:42.668][35][warning][main] external/envoy/source/server/server.cc:346] caught SIGTERM
[2018-07-30 09:46:42.668][35][info][main] external/envoy/source/server/server.cc:400] main dispatch loop exited
[2018-07-30 09:46:42.740][35][info][main] external/envoy/source/server/server.cc:435] exiting
2018-07-30T09:46:42.750468Z info    Epoch 0 exited normally

Expected behavior A clear and concise description of what you expected to happen.

ingressgateway and virtualservice work well Steps to reproduce the bug Steps to reproduce the behavior.

Version What version of istio and Kubernetes are you using? Use istioctl version and kubectl version

Is Istio Auth enabled or not? Did you install the stable istio.yaml, istio-auth.yaml.... or if using the Helm chart please provide full command line input.

Environment Which environment, cloud vendor, OS, etc are you using? kubernetes:1.9 istio:0.8.0

wansuiye commented 6 years ago

update: in istio:1.0 the virtualservice can work for host xxx.service.consul, however ingressgateway still can not work, i execute the ingressgateway pod and found that envoy no listening on port 80. ingressgateway's log is below:

Flag shorthand -v has been deprecated, please use --log_output_level instead
Flag --v has been deprecated, please use --log_output_level instead
2018-08-02T15:10:20.909713Z info    Version root@71a9470ea93c-docker.io/istio-1.0.0-3a136c90ec5e308f236e0d7ebb5c4c5e405217f4-Clean
2018-08-02T15:10:20.909776Z info    Proxy role: model.Proxy{ClusterID:"", Type:"router", IPAddress:"10.20.30.132", ID:"istio-ingressgateway-597f7dbcd9-sw442.istio-system", Domain:"istio-system.svc.cluster.local", Metadata:map[string]string(nil)}
2018-08-02T15:10:20.909789Z info    Attempting to lookup address: istio-statsd-prom-bridge
2018-08-02T15:10:20.910883Z info    Addr resolved to: 192.168.254.90:9125
2018-08-02T15:10:20.910907Z info    Finished lookup of address: istio-statsd-prom-bridge
2018-08-02T15:10:20.918699Z info    Effective config: binaryPath: /usr/local/bin/envoy
configPath: /etc/istio/proxy
connectTimeout: 10s
discoveryAddress: istio-pilot.istio-system:8080
discoveryRefreshDelay: 1s
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: istio-ingressgateway
statsdUdpAddress: 192.168.254.90:9125
zipkinAddress: zipkin:9411

2018-08-02T15:10:20.918721Z info    Monitored certs: []envoy.CertSource{envoy.CertSource{Directory:"/etc/certs/", Files:[]string{"cert-chain.pem", "key.pem", "root-cert.pem"}}}
2018-08-02T15:10:20.918827Z info    Starting proxy agent
2018-08-02T15:10:20.918878Z info    Received new config, resetting budget
2018-08-02T15:10:20.918889Z info    Reconciling configuration (budget 10)
2018-08-02T15:10:20.918906Z info    Epoch 0 starting
2018-08-02T15:10:20.919571Z info    Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~10.20.30.132~istio-ingressgateway-597f7dbcd9-sw442.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 -l warn --v2-config-only]
[2018-08-02 15:10:20.978][25][info][main] external/envoy/source/server/server.cc:183] initializing epoch 0 (hot restart version=10.200.16384.256.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=4882536)
[2018-08-02 15:10:20.978][25][info][main] external/envoy/source/server/server.cc:185] statically linked extensions:
[2018-08-02 15:10:20.978][25][info][main] external/envoy/source/server/server.cc:187]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2018-08-02 15:10:20.978][25][info][main] external/envoy/source/server/server.cc:190]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash,istio_authn,jwt-auth,mixer
[2018-08-02 15:10:20.978][25][info][main] external/envoy/source/server/server.cc:193]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2018-08-02 15:10:20.978][25][info][main] external/envoy/source/server/server.cc:196]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy,mixer
[2018-08-02 15:10:20.978][25][info][main] external/envoy/source/server/server.cc:198]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2018-08-02 15:10:20.978][25][info][main] external/envoy/source/server/server.cc:200]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin
[2018-08-02 15:10:20.978][25][info][main] external/envoy/source/server/server.cc:203]   transport_sockets.downstream: alts,envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-02 15:10:20.978][25][info][main] external/envoy/source/server/server.cc:206]   transport_sockets.upstream: alts,envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-02 15:10:20.988][25][info][config] external/envoy/source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2018-08-02 15:10:21.000][25][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:240] gRPC config stream closed: 14, no healthy upstream
[2018-08-02 15:10:21.000][25][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:41] Unable to establish new stream
[2018-08-02 15:10:21.000][25][info][config] external/envoy/source/server/configuration_impl.cc:60] loading 0 listener(s)
[2018-08-02 15:10:21.000][25][info][config] external/envoy/source/server/configuration_impl.cc:94] loading tracing configuration
[2018-08-02 15:10:21.000][25][info][config] external/envoy/source/server/configuration_impl.cc:103]   loading tracing driver: envoy.zipkin
[2018-08-02 15:10:21.001][25][info][config] external/envoy/source/server/configuration_impl.cc:116] loading stats sink configuration
[2018-08-02 15:10:21.010][25][info][main] external/envoy/source/server/server.cc:410] starting main dispatch loop
[2018-08-02 15:10:21.067][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:129] cm init: initializing cds
[2018-08-02 15:10:24.107][25][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:240] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers
[2018-08-02 15:10:29.255][25][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:240] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers
[2018-08-02 15:10:39.008][25][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:240] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers
[2018-08-02 15:10:49.066][25][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:240] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers
[2018-08-02 15:11:19.439][25][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:240] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers
[2018-08-02 15:11:22.978][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8300||consul.service.consul during init
[2018-08-02 15:11:22.979][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|10833||example-foo.service.consul during init
[2018-08-02 15:11:22.980][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9102||istio-statsd-prom-bridge.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.980][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9125||istio-statsd-prom-bridge.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.980][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||weave-scope-app.weave.svc.cluster.local during init
[2018-08-02 15:11:22.980][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9411||zipkin.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.980][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9090||prometheus.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.981][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|3000||grafana.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.981][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|14267||jaeger-collector.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.981][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|14268||jaeger-collector.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.981][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|53||kube-dns.kube-system.svc.cluster.local during init
[2018-08-02 15:11:22.981][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8088||servicegraph.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.982][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15010||istio-pilot.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.982][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15011||istio-pilot.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.982][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8080||istio-pilot.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.982][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||istio-pilot.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.982][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||istio-sidecar-injector.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.982][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||istio-egressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.983][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||istio-egressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.983][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||istio-galley.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.983][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||istio-galley.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.984][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9091||istio-telemetry.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.984][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15004||istio-telemetry.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.984][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||istio-telemetry.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.984][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|42422||istio-telemetry.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.984][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|30080||istio-ingressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.985][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|30443||istio-ingressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.985][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|31400||istio-ingressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.985][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15011||istio-ingressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.985][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8060||istio-ingressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.985][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15030||istio-ingressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.985][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15031||istio-ingressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.986][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||kubernetes-dashboard.kube-system.svc.cluster.local during init
[2018-08-02 15:11:22.986][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|16686||jaeger-query.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.986][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9091||istio-policy.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.986][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15004||istio-policy.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.986][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||istio-policy.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.987][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8060||istio-citadel.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.987][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||istio-citadel.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.987][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||sidecar-injector-webhook-svc.default.svc.cluster.local during init
[2018-08-02 15:11:22.987][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||kubernetes.default.svc.cluster.local during init
[2018-08-02 15:11:22.987][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|5775||jaeger-agent.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.987][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|6831||jaeger-agent.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.988][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|6832||jaeger-agent.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.988][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||tracing.istio-system.svc.cluster.local during init
[2018-08-02 15:11:22.988][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster BlackHoleCluster during init
[2018-08-02 15:11:22.988][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:110] cm init: initializing secondary clusters
[2018-08-02 15:11:22.992][25][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:133] cm init: all clusters initialized
[2018-08-02 15:11:22.992][25][info][main] external/envoy/source/server/server.cc:390] all clusters initialized. initializing init manager
[2018-08-02 15:11:22.995][25][info][config] external/envoy/source/server/listener_manager_impl.cc:903] all dependencies initialized. starting workers
2018-08-02T15:11:32.618754Z info    watchFileEvents: "/etc/certs//..2018_08_02_15_11_32.397600258": CREATE
2018-08-02T15:11:32.618808Z info    watchFileEvents: "/etc/certs//..2018_08_02_15_11_32.397600258": MODIFY|ATTRIB
2018-08-02T15:11:32.618818Z info    watchFileEvents: "/etc/certs//cert-chain.pem": CREATE
2018-08-02T15:11:32.618857Z info    watchFileEvents: "/etc/certs//key.pem": CREATE
2018-08-02T15:11:32.618867Z info    watchFileEvents: "/etc/certs//root-cert.pem": CREATE
2018-08-02T15:11:32.618874Z info    watchFileEvents: "/etc/certs//..data_tmp": RENAME
2018-08-02T15:11:32.618961Z info    watchFileEvents: "/etc/certs//..data": CREATE
2018-08-02T15:11:32.618990Z info    watchFileEvents: "/etc/certs//..2018_08_02_15_10_19.511582955": DELETE
2018-08-02T15:11:42.619764Z info    watchFileEvents: notifying
2018-08-02T15:11:42.619929Z info    Received new config, resetting budget
2018-08-02T15:11:42.619941Z info    Reconciling configuration (budget 10)
2018-08-02T15:11:42.619968Z info    Epoch 1 starting
2018-08-02T15:11:42.620609Z info    Envoy command: [-c /etc/istio/proxy/envoy-rev1.json --restart-epoch 1 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~10.20.30.132~istio-ingressgateway-597f7dbcd9-sw442.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 -l warn --v2-config-only]
[2018-08-02 15:11:42.677][126][info][main] external/envoy/source/server/server.cc:183] initializing epoch 1 (hot restart version=10.200.16384.256.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=4882536)
[2018-08-02 15:11:42.677][126][info][main] external/envoy/source/server/server.cc:185] statically linked extensions:
[2018-08-02 15:11:42.677][126][info][main] external/envoy/source/server/server.cc:187]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2018-08-02 15:11:42.677][126][info][main] external/envoy/source/server/server.cc:190]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash,istio_authn,jwt-auth,mixer
[2018-08-02 15:11:42.677][126][info][main] external/envoy/source/server/server.cc:193]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2018-08-02 15:11:42.678][126][info][main] external/envoy/source/server/server.cc:196]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy,mixer
[2018-08-02 15:11:42.678][126][info][main] external/envoy/source/server/server.cc:198]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2018-08-02 15:11:42.678][126][info][main] external/envoy/source/server/server.cc:200]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin
[2018-08-02 15:11:42.678][126][info][main] external/envoy/source/server/server.cc:203]   transport_sockets.downstream: alts,envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-02 15:11:42.678][126][info][main] external/envoy/source/server/server.cc:206]   transport_sockets.upstream: alts,envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-02 15:11:42.679][25][warning][main] external/envoy/source/server/server.cc:461] shutting down admin due to child startup
[2018-08-02 15:11:42.679][25][warning][main] external/envoy/source/server/server.cc:469] terminating parent process
[2018-08-02 15:11:42.682][126][info][config] external/envoy/source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2018-08-02 15:11:42.689][126][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:240] gRPC config stream closed: 14, no healthy upstream
[2018-08-02 15:11:42.689][126][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:41] Unable to establish new stream
[2018-08-02 15:11:42.689][126][info][config] external/envoy/source/server/configuration_impl.cc:60] loading 0 listener(s)
[2018-08-02 15:11:42.689][126][info][config] external/envoy/source/server/configuration_impl.cc:94] loading tracing configuration
[2018-08-02 15:11:42.689][126][info][config] external/envoy/source/server/configuration_impl.cc:103]   loading tracing driver: envoy.zipkin
[2018-08-02 15:11:42.689][126][info][config] external/envoy/source/server/configuration_impl.cc:116] loading stats sink configuration
[2018-08-02 15:11:42.689][126][info][main] external/envoy/source/server/server.cc:410] starting main dispatch loop
[2018-08-02 15:11:42.706][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:129] cm init: initializing cds
[2018-08-02 15:11:43.677][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|10833||example-foo.service.consul during init
[2018-08-02 15:11:43.677][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8300||consul.service.consul during init
[2018-08-02 15:11:43.677][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||istio-egressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.678][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||istio-egressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.678][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|3000||grafana.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.678][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|14267||jaeger-collector.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.678][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|14268||jaeger-collector.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.678][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|53||kube-dns.kube-system.svc.cluster.local during init
[2018-08-02 15:11:43.679][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8088||servicegraph.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.679][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15010||istio-pilot.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.679][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15011||istio-pilot.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.679][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8080||istio-pilot.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.679][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||istio-pilot.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.680][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||istio-sidecar-injector.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.680][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|16686||jaeger-query.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.680][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||istio-galley.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.680][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||istio-galley.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.680][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9091||istio-telemetry.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.680][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15004||istio-telemetry.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.680][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||istio-telemetry.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.682][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|42422||istio-telemetry.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.685][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|30080||istio-ingressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.685][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|30443||istio-ingressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.685][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|31400||istio-ingressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.685][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15011||istio-ingressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.685][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8060||istio-ingressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.686][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15030||istio-ingressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.686][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15031||istio-ingressgateway.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.686][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||kubernetes-dashboard.kube-system.svc.cluster.local during init
[2018-08-02 15:11:43.686][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||tracing.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.687][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9091||istio-policy.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.687][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15004||istio-policy.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.687][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||istio-policy.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.687][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8060||istio-citadel.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.687][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||istio-citadel.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.687][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||sidecar-injector-webhook-svc.default.svc.cluster.local during init
[2018-08-02 15:11:43.687][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||kubernetes.default.svc.cluster.local during init
[2018-08-02 15:11:43.688][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|5775||jaeger-agent.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.688][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|6831||jaeger-agent.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.688][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|6832||jaeger-agent.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.688][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9090||prometheus.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.688][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9102||istio-statsd-prom-bridge.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.689][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9125||istio-statsd-prom-bridge.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.689][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||weave-scope-app.weave.svc.cluster.local during init
[2018-08-02 15:11:43.689][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9411||zipkin.istio-system.svc.cluster.local during init
[2018-08-02 15:11:43.689][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster BlackHoleCluster during init
[2018-08-02 15:11:43.689][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:110] cm init: initializing secondary clusters
[2018-08-02 15:11:43.700][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:133] cm init: all clusters initialized
[2018-08-02 15:11:43.700][126][info][main] external/envoy/source/server/server.cc:390] all clusters initialized. initializing init manager
[2018-08-02 15:11:43.710][126][info][config] external/envoy/source/server/listener_manager_impl.cc:903] all dependencies initialized. starting workers
[2018-08-02 15:11:43.900][25][info][main] external/envoy/source/server/server.cc:98] closing and draining listeners
[2018-08-02 15:12:22.995][25][info][main] external/envoy/source/server/drain_manager_impl.cc:63] shutting down parent after drain
[2018-08-02 15:12:43.713][126][info][main] external/envoy/source/server/drain_manager_impl.cc:63] shutting down parent after drain
[2018-08-02 15:12:43.718][25][info][main] external/envoy/source/server/hot_restart_impl.cc:438] shutting down due to child request
[2018-08-02 15:12:43.719][25][warning][main] external/envoy/source/server/server.cc:360] caught SIGTERM
[2018-08-02 15:12:43.719][25][info][main] external/envoy/source/server/server.cc:414] main dispatch loop exited
[2018-08-02 15:12:43.930][25][info][main] external/envoy/source/server/server.cc:449] exiting
2018-08-02T15:12:43.945829Z info    Epoch 0 exited normally
[2018-08-02 15:19:02.279][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:394] add/update cluster outbound|9080||example-foo.example.svc.cluster.local starting warming
[2018-08-02 15:19:02.280][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:394] add/update cluster outbound|9090||example-foo.example.svc.cluster.local starting warming
[2018-08-02 15:19:02.280][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:401] warming cluster outbound|9090||example-foo.example.svc.cluster.local complete
[2018-08-02 15:19:02.280][126][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:401] warming cluster outbound|9080||example-foo.example.svc.cluster.local complete
wansuiye commented 6 years ago

At the same time, there is also a problem that bothers me. There are two services, example-a and exampleb-b. They use the GRPC protocol between them.

when example-a request example-b using a kubernetes fqdn domain , example-a get error "UNAVAILABLE: upstream connect error or disconnect/reset before headers", istio-proxy's log is:

[2018-08-03T06:52:23.442Z] "POST /example-b.BService/get HTTP/2" 200 UC 7 0 0 - "-" "grpc-java-netty/1.6.1" "23d02c0b-3716-988f-ba64-04250fdb988d" "example-b" "10.20.30.2:20090"

When the number of instances of example-b is increased, this error occurs when accessing some pods of example-b, but it does not occur when accessing other pods of example-b.

I can guarantee that it will work in kubernetes. This problem will come up in the environment of kubernetes and consul.

I don't know why adding a registry would result in a failed request to use GRPC between services. You can see in the proxy's log that the target address has been resolved through service discovery. Also i don't know if it's associated with the problem ingress not work.

wish to the reply~

wansuiye commented 6 years ago

@GregHanson could u help me confirm this problem? thanks~

mingweiLIU commented 6 years ago

@wansuiye Hi, I am encountering almost the same issue with istio-gateway log code: 200 UC, have you fixed this issue? can you share the solution?

jeffcai commented 6 years ago

I have the same issue, deployed the consul and related services outside of K8s cluster, and configure the pilot discovery to discover service from the consul, services are listed by running: istioctl proxy-config clusters -n istio-system istio-ingressgateway-xxxxx, and configured the gateway and virtualservice, when accessing the service it return 404 and check the logs which outputs:

[2018-11-14T02:51:32.685Z] "GET /servicebHTTP/1.1" 404 - 0 306 5 5 "192.168.65.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36" "7618ed6c-0146-4a8b-8578-0262620b9e22" "localhost" "9.111.30.117:8071" outbound|8071||service-b.service.consul - 10.1.0.61:80 192.168.65.3:44540

exec into the istio-proxy container, and failed to curl the service with domain: service-b.service.consul:8071 because could not resolve hos.

Update on 11/15/2018: I made a mistake to the virtualservice rule which is not redirecting to service endpoint, after correcting it it works.

wansuiye commented 5 years ago

@mingweiLIU hi, late to reply. do you use the consul and k8s platform adaptor at the same time? if the service is registered in both registry, there is the problem.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

stale[bot] commented 5 years ago

This issue has been automatically closed because it has not had activity in the last month and a half. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.