2019/01/28 19:02:55 Sleeping for 60s to allow Envoy to bootstrap
2019/01/28 19:03:55 Starting server, listening on port 9080
2019/01/28 19:03:55 Using color-teller at colorteller.default.svc.cluster.local:9080
2019/01/28 19:03:55 Using tcp-echo at tcpecho.default.svc.cluster.local:2701
A communication error occurred: "Operation timed out"
The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.
It looks like your Envoy Proxy may not be stood up completely. The /config_dump path should return the serialized JSON configuration of the proxy (see here).
Could you review the logs of your Envoy Proxy? There may be a bootstrapping error.
Sorry, is there anything after 16:57:05? The last log line looks like your Envoy is initializing the bi-directional gRPC stream, so there should be a number of DiscoveryRequests and DiscoveryResponses that occur after it. This is likely where the problem occurred, if anywhere.
This is the remaining log from the envoy
[2019-02-01 16:57:05.991][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
It looks like there is an error occurring on Envoy's end when establishing the gRPC connection to the Envoy Management Service.
Could be a connectivity issue, or maybe an SSL issue.
Looking at the /config_dump output from your Envoy may reveal an issue in your bootstrap config. Perhaps the Envoy Management Service endpoint is being misconfigured?
You could also set some gRPC enviroment variables in your colorapp.yaml. To possibly see why the Envoy container is failing to establish the gRPC stream.
Maybe try GRPC_VERBOSITY=INFO and GRPC_TRACE=transport_security,connectivity_state,client_channel,server_channel.
why do we need to connect to public endpoint ?Do we need proxy for reaching that ip ?I tried from ec2 by sourcing my proxy config unable to connect that ip .
source proxy.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 32 100 32 0 0 29547 0 --:--:-- --:--:-- --:--:-- 32000
####################################################################################################\n
Hostname:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12 100 12 0 0 15503 0 --:--:-- --:--:-- --:--:-- 12000
####################################################################################################\n
Local IP:
curl 34.203.106.141:443
Network Error
Network Error (tcp_error)
A communication error occurred: ""
The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.
@daparthi001
Your Envoy containers need to be able to reach appmesh-envoy-management.us-east-1.amazonaws.com:443 (assuming us-east-1 from your logs) to download the configuration for the nodes in your service mesh. So I believe an internet gateway for your VPC is necessary right now.
Adding proxy to the envoy helped to connect to envoy .
curl on gateway is getting me the same color tried more than 10000 times
curl 10.111.8.209:9080/color
{"color":"white", "stats": {"white":1}}root@curler-cc4fff495-m7zzr:/# curl 10.111.8.209:9080/color
{"color":"white", "stats": {"white":1}}root@curler-cc4fff495-m7zzr:/# curl 10.111.8.209:9080/color
{"color":"white", "stats": {"white":1}}root@curler-cc4fff495-m7zzr:/# curl 10.111.8.209:9080/color
{"color":"white", "stats": {"white":1}}root@curler-cc4fff495-m7zzr:/# curl 10.111.8.209:9080/color
{"color":"white", "stats": {"white":1}}root@curler-cc4fff495-m7zzr:/#
@Y0Username I have switched traffic using canary.sh ,its taking more time to display just red ,being trying more than 15 mins ,is there way we can display output earlier ,it just displays blue only
We hosted kubernetes cluster on single node ,deployed the colorapp.yaml to get the app running . Failing to get the desired results.
root@curler-65d5db4cd9-mz9c2:/# bash root@curler-65d5db4cd9-mz9c2:/# echo ${SERVICES_DOMAIN} default.svc.cluster.local root@curler-65d5db4cd9-mz9c2:/# curl -s http://colorgateway.${SERVICES_DOMAIN}:9080/color root@curler-65d5db4cd9-mz9c2:/# curl -v -s http://colorgateway.${SERVICES_DOMAIN}:9080/color
We used proxy configuration but unable to resolve the hostname.
Network Error (dns_unresolved_hostname)
kubectl logs colorgateway-6988677b74-6c9gb colorgateway
2019/01/28 19:02:55 Sleeping for 60s to allow Envoy to bootstrap 2019/01/28 19:03:55 Starting server, listening on port 9080 2019/01/28 19:03:55 Using color-teller at colorteller.default.svc.cluster.local:9080 2019/01/28 19:03:55 Using tcp-echo at tcpecho.default.svc.cluster.local:2701
kubectl logs colorteller-black-79b7784f7-7zk95 colorteller
2019/01/28 19:02:55 starting server, listening on port 9080
Could you provide the output of the following command?
root@curler-65d5db4cd9-7vp9j:/# curl -v http://colorgateway.${SERVICES_DOMAIN}:9901/config_dump
if i use proxy
It looks like your Envoy Proxy may not be stood up completely. The
/config_dumppath should return the serialized JSON configuration of the proxy (see here).Could you review the logs of your Envoy Proxy? There may be a bootstrapping error.
Sorry, is there anything after 16:57:05? The last log line looks like your Envoy is initializing the bi-directional gRPC stream, so there should be a number of DiscoveryRequests and DiscoveryResponses that occur after it. This is likely where the problem occurred, if anywhere.
This is the remaining log from the envoy [2019-02-01 16:57:05.991][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 16:58:05.736][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 16:58:05.736][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 16:58:05.736][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 16:58:05.736][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 16:58:05.736][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 16:58:05.736][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 16:58:05.772][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 16:59:05.908][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 16:59:05.908][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 16:59:05.908][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 16:59:05.908][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 16:59:05.908][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 16:59:05.908][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 16:59:07.605][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 17:00:06.079][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 17:00:06.079][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 17:00:06.079][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 17:00:06.079][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 17:00:06.079][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 17:00:06.079][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 17:00:08.153][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 17:01:06.332][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 17:01:06.332][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 17:01:06.332][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 17:01:06.332][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 17:01:06.332][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 17:01:06.332][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 17:01:14.673][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 17:02:17.124][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 17:02:17.124][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 17:02:17.124][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 17:02:17.124][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 17:02:17.124][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 17:02:17.124][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 17:02:30.652][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 17:04:16.164][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 17:04:16.164][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 17:04:16.164][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 17:04:16.164][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 17:04:16.164][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 17:04:16.164][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 17:04:26.271][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 17:07:56.698][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 17:07:56.698][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 17:07:56.698][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 17:07:56.698][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 17:07:56.698][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 17:07:56.698][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 17:08:19.095][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 17:13:28.813][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 17:13:28.813][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 17:13:28.813][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 17:13:28.813][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 17:13:28.813][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 17:13:28.813][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 17:13:30.307][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 17:19:43.179][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 17:19:43.179][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 17:19:43.179][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 17:19:43.179][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 17:19:43.179][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 17:19:43.179][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 17:20:13.179][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 17:25:37.524][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 17:25:37.524][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 17:25:37.524][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 17:25:37.524][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 17:25:37.524][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 17:25:37.524][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 17:25:49.102][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 17:31:40.156][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 17:31:40.156][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 17:31:40.156][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 17:31:40.156][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 17:31:40.156][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 17:31:40.156][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 17:32:07.671][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 17:37:09.690][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 17:37:09.690][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 17:37:09.690][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 17:37:09.690][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 17:37:09.690][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 17:37:09.690][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 17:37:21.261][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 17:42:33.163][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 17:42:33.163][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 17:42:33.163][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 17:42:33.163][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 17:42:33.163][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 17:42:33.163][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 17:42:54.138][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 17:46:47.627][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 17:46:47.627][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 17:46:47.627][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 17:46:47.627][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 17:46:47.627][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 17:46:47.627][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 17:46:57.778][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
[2019-02-01 17:52:42.006][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 17:52:42.006][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 17:52:42.006][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 17:52:42.006][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags [2019-02-01 17:52:42.006][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 17:52:42.006][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct [2019-02-01 17:53:00.565][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
I think we're seeing Envoy reach this case.
It looks like there is an error occurring on Envoy's end when establishing the gRPC connection to the Envoy Management Service.
Could be a connectivity issue, or maybe an SSL issue.
Looking at the
/config_dumpoutput from your Envoy may reveal an issue in your bootstrap config. Perhaps the Envoy Management Service endpoint is being misconfigured?You could also set some gRPC enviroment variables in your
colorapp.yaml. To possibly see why the Envoy container is failing to establish the gRPC stream.Maybe try
GRPC_VERBOSITY=INFOandGRPC_TRACE=transport_security,connectivity_state,client_channel,server_channel.IS corporate Proxy need to be configured for communication ?
New logs after adding envoy parameters.
I0201 22:07:12.894940803 1 client_channel.cc:1144] chand=0x33c4c30 calld=0x34c4b20: adding pending batch at index 0 I0201 22:07:12.894958094 1 client_channel.cc:3093] chand=0x33c4c30 calld=0x34c4b20: entering client_channel combiner I0201 22:07:12.894966056 1 client_channel.cc:218] chand=0x33c4c30: starting name resolution I0201 22:07:12.894981181 1 client_channel.cc:2857] chand=0x33c4c30 calld=0x34c4b20: deferring pick pending resolver result [2019-02-01 22:07:12.895][1][info][config] source/server/configuration_impl.cc:61] loading 0 listener(s) [2019-02-01 22:07:12.895][1][info][config] source/server/configuration_impl.cc:95] loading tracing configuration [2019-02-01 22:07:12.895][1][info][config] source/server/configuration_impl.cc:113] loading stats sink configuration [2019-02-01 22:07:12.895][1][info][main] source/server/server.cc:452] starting main dispatch loop I0201 22:07:13.072173878 10 client_channel.cc:485] chand=0x33c4c30: got resolver result: resolver_result=0x33c7060 error="No Error" I0201 22:07:13.169226346 10 client_channel.cc:398] chand=0x33c4c30: created new LB policy "pick_first" (0x3538c10) I0201 22:07:13.169259289 10 connectivity_state.cc:92] CONWATCH: 0x3538df8 pick_first: get IDLE I0201 22:07:13.169264604 10 connectivity_state.cc:116] CONWATCH: 0x3538df8 pick_first: from IDLE [cur=IDLE] notify=0x385a698 I0201 22:07:13.169271682 10 client_channel.cc:177] chand=0x33c4c30: setting connectivity state to IDLE I0201 22:07:13.169276102 10 connectivity_state.cc:164] SET: 0x33c4ce0 client_channel: IDLE --> IDLE [resolver_result] error=(nil) "No Error" I0201 22:07:13.169283208 10 client_channel.cc:2946] chand=0x33c4c30 calld=0x34c4b20: resolver returned, doing LB pick I0201 22:07:13.169286689 10 client_channel.cc:2785] chand=0x33c4c30 calld=0x34c4b20: applying service config to call I0201 22:07:13.169290097 10 client_channel.cc:2693] chand=0x33c4c30 calld=0x34c4b20: starting pick on lb_policy=0x3538c10 I0201 22:07:13.169295113 10 connectivity_state.cc:92] CONWATCH: 0x36d3320 subchannel: get IDLE I0201 22:07:13.169299740 10 connectivity_state.cc:116] CONWATCH: 0x36d3320 subchannel: from IDLE [cur=IDLE] notify=0x37e1118 I0201 22:07:13.169304390 10 connectivity_state.cc:164] SET: 0x36d3320 subchannel: IDLE --> CONNECTING [connecting] error=(nil) "No Error" I0201 22:07:13.169307424 10 connectivity_state.cc:190] NOTIFY: 0x36d3320 subchannel: 0x37e1118 I0201 22:07:13.169518745 10 connectivity_state.cc:164] SET: 0x3538df8 pick_first: IDLE --> CONNECTING [connecting_changed] error=(nil) "No Error" I0201 22:07:13.169533199 10 connectivity_state.cc:190] NOTIFY: 0x3538df8 pick_first: 0x385a698 I0201 22:07:13.169537467 10 connectivity_state.cc:116] CONWATCH: 0x36d3320 subchannel: from CONNECTING [cur=CONNECTING] notify=0x37e1118 I0201 22:07:13.169542001 10 client_channel.cc:189] chand=0x33c4c30: lb_policy=0x3538c10 state changed to CONNECTING I0201 22:07:13.169545158 10 client_channel.cc:177] chand=0x33c4c30: setting connectivity state to CONNECTING I0201 22:07:13.169548173 10 connectivity_state.cc:164] SET: 0x33c4ce0 client_channel: IDLE --> CONNECTING [lb_changed] error=(nil) "No Error" I0201 22:07:13.169551239 10 connectivity_state.cc:116] CONWATCH: 0x3538df8 pick_first: from CONNECTING [cur=CONNECTING] notify=0x385a708 [2019-02-01 22:07:17.895][1][debug][main] source/server/server.cc:139] flushing stats I0201 22:07:33.169230757 11 connectivity_state.cc:164] SET: 0x36d3320 subchannel: CONNECTING --> TRANSIENT_FAILURE [connect_failed] error=0x3807a40 {"created":"@1549058853.169123779","description":"Connect Failed","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/subchannel.cc","file_line":663,"grpc_status":14,"referenced_errors":[{"created":"@1549058853.169063019","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058853.169041310","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:34.203.106.141:443"}]} I0201 22:07:33.169263354 11 connectivity_state.cc:190] NOTIFY: 0x36d3320 subchannel: 0x37e1118 I0201 22:07:33.169268739 11 subchannel.cc:668] Connect failed: {"created":"@1549058853.169063019","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058853.169041310","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:34.203.106.141:443"} I0201 22:07:33.169281331 11 connectivity_state.cc:92] CONWATCH: 0x37a87e0 subchannel: get IDLE I0201 22:07:33.169287689 11 connectivity_state.cc:116] CONWATCH: 0x37a87e0 subchannel: from IDLE [cur=IDLE] notify=0x37e1118 I0201 22:07:33.169292046 11 connectivity_state.cc:164] SET: 0x37a87e0 subchannel: IDLE --> CONNECTING [connecting] error=(nil) "No Error" I0201 22:07:33.169297555 11 connectivity_state.cc:190] NOTIFY: 0x37a87e0 subchannel: 0x37e1118 I0201 22:07:33.169574912 11 connectivity_state.cc:164] SET: 0x3538df8 pick_first: CONNECTING --> CONNECTING [connecting_changed] error=(nil) "No Error" I0201 22:07:33.169590148 11 connectivity_state.cc:116] CONWATCH: 0x37a87e0 subchannel: from CONNECTING [cur=CONNECTING] notify=0x37e1118 I0201 22:07:53.169164061 22 connectivity_state.cc:164] SET: 0x37a87e0 subchannel: CONNECTING --> TRANSIENT_FAILURE [connect_failed] error=0x3807c80 {"created":"@1549058873.169118742","description":"Connect Failed","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/subchannel.cc","file_line":663,"grpc_status":14,"referenced_errors":[{"created":"@1549058873.169079751","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058873.169060228","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:18.215.208.173:443"}]} I0201 22:07:53.169273351 22 connectivity_state.cc:190] NOTIFY: 0x37a87e0 subchannel: 0x37e1118 I0201 22:07:53.169279765 22 subchannel.cc:668] Connect failed: {"created":"@1549058873.169079751","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058873.169060228","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:18.215.208.173:443"} I0201 22:07:53.169291412 22 connectivity_state.cc:92] CONWATCH: 0x3849c20 subchannel: get IDLE I0201 22:07:53.169296658 22 connectivity_state.cc:116] CONWATCH: 0x3849c20 subchannel: from IDLE [cur=IDLE] notify=0x37e1118 I0201 22:07:53.169300997 22 connectivity_state.cc:164] SET: 0x3849c20 subchannel: IDLE --> CONNECTING [connecting] error=(nil) "No Error" I0201 22:07:53.169306608 22 connectivity_state.cc:190] NOTIFY: 0x3849c20 subchannel: 0x37e1118 I0201 22:07:53.169496437 22 connectivity_state.cc:164] SET: 0x3538df8 pick_first: CONNECTING --> CONNECTING [connecting_changed] error=(nil) "No Error" I0201 22:07:53.169510861 22 connectivity_state.cc:116] CONWATCH: 0x3849c20 subchannel: from CONNECTING [cur=CONNECTING] notify=0x37e1118 I0201 22:08:13.169156242 11 connectivity_state.cc:164] SET: 0x3849c20 subchannel: CONNECTING --> TRANSIENT_FAILURE [connect_failed] error=0x3807b00 {"created":"@1549058893.169113220","description":"Connect Failed","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/subchannel.cc","file_line":663,"grpc_status":14,"referenced_errors":[{"created":"@1549058893.169073524","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058893.169050292","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:54.204.207.83:443"}]} I0201 22:08:13.169247248 11 connectivity_state.cc:190] NOTIFY: 0x3849c20 subchannel: 0x37e1118 I0201 22:08:13.169253853 11 subchannel.cc:668] Connect failed: {"created":"@1549058893.169073524","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058893.169050292","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:54.204.207.83:443"} I0201 22:08:13.169265694 11 connectivity_state.cc:164] SET: 0x3538df8 pick_first: CONNECTING --> TRANSIENT_FAILURE [exhausted_subchannels] error=0x3807b00 {"created":"@1549058893.169113220","description":"Connect Failed","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/subchannel.cc","file_line":663,"grpc_status":14,"referenced_errors":[{"created":"@1549058893.169073524","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058893.169050292","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:54.204.207.83:443"}]} I0201 22:08:13.169269548 11 connectivity_state.cc:190] NOTIFY: 0x3538df8 pick_first: 0x385a708 I0201 22:08:13.169273330 11 connectivity_state.cc:92] CONWATCH: 0x36d3320 subchannel: get TRANSIENT_FAILURE I0201 22:08:13.169278351 11 connectivity_state.cc:116] CONWATCH: 0x36d3320 subchannel: from TRANSIENT_FAILURE [cur=TRANSIENT_FAILURE] notify=0x37e1118 I0201 22:08:13.169281711 11 subchannel.cc:490] Subchannel 0x36d3200: Retry immediately I0201 22:08:13.169287469 11 subchannel.cc:452] Failed to connect to channel, retrying I0201 22:08:13.169292330 11 connectivity_state.cc:164] SET: 0x36d3320 subchannel: TRANSIENT_FAILURE --> CONNECTING [connecting] error=(nil) "No Error" I0201 22:08:13.169301470 11 connectivity_state.cc:190] NOTIFY: 0x36d3320 subchannel: 0x37e1118 I0201 22:08:13.169527575 11 client_channel.cc:372] chand=0x33c4c30: started name re-resolving I0201 22:08:13.169553354 11 client_channel.cc:189] chand=0x33c4c30: lb_policy=0x3538c10 state changed to TRANSIENT_FAILURE I0201 22:08:13.169559277 11 client_channel.cc:177] chand=0x33c4c30: setting connectivity state to TRANSIENT_FAILURE I0201 22:08:13.169564392 11 connectivity_state.cc:164] SET: 0x33c4ce0 client_channel: CONNECTING --> TRANSIENT_FAILURE [lb_changed] error=0x3807b00 {"created":"@1549058893.169113220","description":"Connect Failed","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/subchannel.cc","file_line":663,"grpc_status":14,"referenced_errors":[{"created":"@1549058893.169073524","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058893.169050292","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:54.204.207.83:443"}]} I0201 22:08:13.169569225 11 connectivity_state.cc:116] CONWATCH: 0x3538df8 pick_first: from TRANSIENT_FAILURE [cur=TRANSIENT_FAILURE] notify=0x385ae78 I0201 22:08:13.169577327 11 connectivity_state.cc:164] SET: 0x3538df8 pick_first: TRANSIENT_FAILURE --> CONNECTING [connecting_changed] error=(nil) "No Error" I0201 22:08:13.169580487 11 connectivity_state.cc:190] NOTIFY: 0x3538df8 pick_first: 0x385ae78 I0201 22:08:13.169584514 11 connectivity_state.cc:116] CONWATCH: 0x36d3320 subchannel: from CONNECTING [cur=CONNECTING] notify=0x37e1998 I0201 22:08:13.169590871 11 client_channel.cc:2749] chand=0x33c4c30 calld=0x34c4b20: pick completed asynchronously I0201 22:08:13.169645069 11 client_channel.cc:2638] chand=0x33c4c30 calld=0x34c4b20: failed to create subchannel: error={"created":"@1549058893.169596568","description":"Failed to create subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":2636,"referenced_errors":[{"created":"@1549058893.169558164","description":"Pick Cancelled","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":241,"referenced_errors":[{"created":"@1549058893.169113220","description":"Connect Failed","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/subchannel.cc","file_line":663,"grpc_status":14,"referenced_errors":[{"created":"@1549058893.169073524","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058893.169050292","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:54.204.207.83:443"}]}]}]} I0201 22:08:13.169656202 11 client_channel.cc:1235] chand=0x33c4c30 calld=0x34c4b20: failing 1 pending batches: {"created":"@1549058893.169596568","description":"Failed to create subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":2636,"referenced_errors":[{"created":"@1549058893.169558164","description":"Pick Cancelled","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":241,"referenced_errors":[{"created":"@1549058893.169113220","description":"Connect Failed","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/subchannel.cc","file_line":663,"grpc_status":14,"referenced_errors":[{"created":"@1549058893.169073524","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058893.169050292","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:54.204.207.83:443"}]}]}]} I0201 22:08:13.169678202 11 client_channel.cc:3055] chand=0x33c4c30 calld=0x34c4b20: recording cancel_error={"created":"@1549058893.169596568","description":"Failed to create subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":2636,"referenced_errors":[{"created":"@1549058893.169558164","description":"Pick Cancelled","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":241,"referenced_errors":[{"created":"@1549058893.169113220","description":"Connect Failed","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/subchannel.cc","file_line":663,"grpc_status":14,"referenced_errors":[{"created":"@1549058893.169073524","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058893.169050292","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:54.204.207.83:443"}]}]}]} I0201 22:08:13.169682431 11 client_channel.cc:1235] chand=0x33c4c30 calld=0x34c4b20: failing 0 pending batches: {"created":"@1549058893.169596568","description":"Failed to create subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":2636,"referenced_errors":[{"created":"@1549058893.169558164","description":"Pick Cancelled","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":241,"referenced_errors":[{"created":"@1549058893.169113220","description":"Connect Failed","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/subchannel.cc","file_line":663,"grpc_status":14,"referenced_errors":[{"created":"@1549058893.169073524","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058893.169050292","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:54.204.207.83:443"}]}]}]} I0201 22:08:13.169686700 11 client_channel.cc:189] chand=0x33c4c30: lb_policy=0x3538c10 state changed to CONNECTING I0201 22:08:13.169689693 11 client_channel.cc:177] chand=0x33c4c30: setting connectivity state to CONNECTING I0201 22:08:13.169692806 11 connectivity_state.cc:164] SET: 0x33c4ce0 client_channel: TRANSIENT_FAILURE --> CONNECTING [lb_changed] error=(nil) "No Error" I0201 22:08:13.169696198 11 connectivity_state.cc:116] CONWATCH: 0x3538df8 pick_first: from CONNECTING [cur=CONNECTING] notify=0x385a708 I0201 22:08:13.169700551 11 client_channel.cc:2767] chand=0x33c4c30 calld=0x34c4b20: cancelling pick from LB policy 0x3538c10 [2019-02-01 22:08:13.169][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:182] notifyRemoteClose 13 [2019-02-01 22:08:13.169][1][warning][upstream] source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 13, [2019-02-01 22:08:13.169][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:203] resetStream [2019-02-01 22:08:13.169][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:365] Stream cleanup with 0 in-flight tags I0201 22:08:13.169930439 1 client_channel.cc:3036] chand=0x33c4c30 calld=0x34c4b20: failing batch with error: {"created":"@1549058893.169596568","description":"Failed to create subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":2636,"referenced_errors":[{"created":"@1549058893.169558164","description":"Pick Cancelled","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":241,"referenced_errors":[{"created":"@1549058893.169113220","description":"Connect Failed","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/subchannel.cc","file_line":663,"grpc_status":14,"referenced_errors":[{"created":"@1549058893.169073524","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058893.169050292","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:54.204.207.83:443"}]}]}]} [2019-02-01 22:08:13.169][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:354] Deferred delete [2019-02-01 22:08:13.169][1][debug][grpc] source/common/grpc/google_async_client_impl.cc:135] GoogleAsyncStreamImpl destruct I0201 22:08:13.169994019 1 client_channel.cc:3036] chand=0x33c4c30 calld=0x34c4b20: failing batch with error: {"created":"@1549058893.169596568","description":"Failed to create subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":2636,"referenced_errors":[{"created":"@1549058893.169558164","description":"Pick Cancelled","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":241,"referenced_errors":[{"created":"@1549058893.169113220","description":"Connect Failed","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/subchannel.cc","file_line":663,"grpc_status":14,"referenced_errors":[{"created":"@1549058893.169073524","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058893.169050292","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:54.204.207.83:443"}]}]}]} I0201 22:08:13.241532598 10 client_channel.cc:485] chand=0x33c4c30: got resolver result: resolver_result=0x33c7070 error="No Error" I0201 22:08:13.241569799 10 client_channel.cc:521] chand=0x33c4c30: updating existing LB policy "pick_first" (0x3538c10) [2019-02-01 22:08:13.242][1][debug][upstream] source/common/config/grpc_mux_impl.cc:40] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);
I0201 22:08:13.242218567 1 client_channel.cc:1144] chand=0x33c4c30 calld=0x34c4b20: adding pending batch at index 0 I0201 22:08:13.242236497 1 client_channel.cc:3093] chand=0x33c4c30 calld=0x34c4b20: entering client_channel combiner I0201 22:08:13.340316275 10 connectivity_state.cc:113] CONWATCH: 0x36d3320 subchannel: unsubscribe notify=0x37e1998 I0201 22:08:13.340360591 10 connectivity_state.cc:92] CONWATCH: 0x36d3320 subchannel: get CONNECTING I0201 22:08:13.340368402 10 connectivity_state.cc:116] CONWATCH: 0x36d3320 subchannel: from CONNECTING [cur=CONNECTING] notify=0x37e1398 I0201 22:08:13.340383367 10 client_channel.cc:2785] chand=0x33c4c30 calld=0x34c4b20: applying service config to call I0201 22:08:13.340387673 10 client_channel.cc:2693] chand=0x33c4c30 calld=0x34c4b20: starting pick on lb_policy=0x3538c10 I0201 22:08:33.169159727 22 connectivity_state.cc:164] SET: 0x36d3320 subchannel: CONNECTING --> TRANSIENT_FAILURE [connect_failed] error=0x3807d40 {"created":"@1549058913.169114409","description":"Connect Failed","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/subchannel.cc","file_line":663,"grpc_status":14,"referenced_errors":[{"created":"@1549058913.169073627","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058913.169050290","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:34.203.106.141:443"}]} I0201 22:08:33.169260472 22 connectivity_state.cc:190] NOTIFY: 0x36d3320 subchannel: 0x37e1398 I0201 22:08:33.169266333 22 subchannel.cc:668] Connect failed: {"created":"@1549058913.169073627","description":"Failed to connect to remote host: FD Shutdown","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/lockfree_event.cc","file_line":194,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1549058913.169050290","description":"connect() timed out","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_client_posix.cc","file_line":114}],"target_address":"ipv4:34.203.106.141:443"} I0201 22:08:33.169277616 22 connectivity_state.cc:92] CONWATCH: 0x3849c20 subchannel: get TRANSIENT_FAILURE I0201 22:08:33.169283633 22 connectivity_state.cc:116] CONWATCH: 0x3849c20 subchannel: from TRANSIENT_FAILURE [cur=TRANSIENT_FAILURE] notify=0x37e1398 I0201 22:08:33.169287337 22 subchannel.cc:490] Subchannel 0x3849b00: Retry immediately I0201 22:08:33.169293005 22 subchannel.cc:452] Failed to connect to channel, retrying I0201 22:08:33.169297794 22 connectivity_state.cc:164] SET: 0x3849c20 subchannel: TRANSIENT_FAILURE --> CONNECTING [connecting] error=(nil) "No Error" I0201 22:08:33.169302914 22 connectivity_state.cc:190] NOTIFY: 0x3849c20 subchannel: 0x37e1398 I0201 22:08:33.169519525 22 connectivity_state.cc:164] SET: 0x3538df8 pick_first: CONNECTING --> CONNECTING [connecting_changed] error=(nil) "No Error" I0201 22:08:33.169533477 22 connectivity_state.cc:116] CONWATCH: 0x3849c20 subchannel: from CONNECTING [cur=CONNECTING] notify=0x37e1398
From your log:
34.203.106.141:443is a public endpoint for our Envoy Management Service. Looks like your Envoy can't reach this IP from inside your cluster.why do we need to connect to public endpoint ?Do we need proxy for reaching that ip ?I tried from ec2 by sourcing my proxy config unable to connect that ip .
source proxy.sh % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 32 100 32 0 0 29547 0 --:--:-- --:--:-- --:--:-- 32000 ####################################################################################################\n Hostname: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 12 100 12 0 0 15503 0 --:--:-- --:--:-- --:--:-- 12000 ####################################################################################################\n Local IP:
curl 34.203.106.141:443
telnet 34.203.106.141 443 Trying 34.203.106.141...
Any update on this ?Do i need to open security groups to any subnet ?Can i bounce envoy on any private ip's ?
@daparthi001 Your Envoy containers need to be able to reach
appmesh-envoy-management.us-east-1.amazonaws.com:443(assuming us-east-1 from your logs) to download the configuration for the nodes in your service mesh. So I believe an internet gateway for your VPC is necessary right now.Adding proxy to the envoy helped to connect to envoy . curl on gateway is getting me the same color tried more than 10000 times curl 10.111.8.209:9080/color {"color":"white", "stats": {"white":1}}root@curler-cc4fff495-m7zzr:/# curl 10.111.8.209:9080/color {"color":"white", "stats": {"white":1}}root@curler-cc4fff495-m7zzr:/# curl 10.111.8.209:9080/color {"color":"white", "stats": {"white":1}}root@curler-cc4fff495-m7zzr:/# curl 10.111.8.209:9080/color {"color":"white", "stats": {"white":1}}root@curler-cc4fff495-m7zzr:/# curl 10.111.8.209:9080/color {"color":"white", "stats": {"white":1}}root@curler-cc4fff495-m7zzr:/#
@daparthi001 you need to update the routes using this route canary script for the color gateway to hit the other color tellers.
@Y0Username I have switched traffic using canary.sh ,its taking more time to display just red ,being trying more than 15 mins ,is there way we can display output earlier ,it just displays blue only
aws appmesh update-route --mesh-name appmesh --profile default --cli-input-json file:////root/aws-app-mesh-examples/examples/apps/colorapp/servicemesh/config/update_routes//colorteller-route-red-100.json --query route.metadata.uid --output text
curl 10.111.29.151:9080/color {"color":"blue", "stats": {"black":0.04938271604938271,"blue":0.654320987654321,"red":0.2962962962962963}}
This might need further investigation. Do you see your envoy configs(
/config_dump) being updated as soon as you update routes?Resolving old issues. Please open back up if further help is needed.