sidecar-proxy container remains in "Pending" status when using retry_join

PedroGuerraPT commented 3 years ago

I'm trying to connect the example mesh-task to a previously set up Consul server hosted in an EC2 instance, using the retry_join input in the mesh-task definition:

retry_join = "provider=aws tag_key=Name tag_value=consul"

However, when the task starts only the mesh-init and the consul-client enter the Running status, while the sidecar-proxy and example-client-app remain in Pending status:

If I switch back to the configuration using the consul_server_service_name input, every container starts successfully.

Things I've already checked that might be causing the issue:

Is Server accepting connections? -> Consul server is up and already connected to another sidecar service hosted in an EC2 instance
Are Security Groups correct? -> Both Server and ECS Tasks have all ports/protocols open to "0.0.0.0/0" connections
Is the app starting in Docker? -> I'm using the provided app from the examples

I'm not quite sure how I can debug the issue, as there are no log files for pending containers on ECS (or at least I believe there aren't any).

Could there be any config missing on my Consul server, or is there any issue when using retry_join with the AWS provider?

lkysow commented 3 years ago

Hi Pedro, Thanks so much for trying this out!

So your example-client isn't starting because the sidecar-proxy container is still in pending (see startup order https://www.consul.io/docs/ecs/architecture#task-startup). And the sidecar-proxy container is still in pending because mesh-init has not yet completed.

So the first step here is to look into the logs for mesh-init. And I anticipate we'll also need the logs for consul-client.

PedroGuerraPT commented 3 years ago

@lkysow Thanks for leading me into the right path!

According to the logs, mesh-init appears to have started successfully (?):

2021-07-06T11:39:11.081Z [INFO]  registering service
2021-07-06T11:39:11.081Z [ERROR] Put "http://127.0.0.1:8500/v1/agent/service/register": dial tcp 127.0.0.1:8500: connect: connection refused: retry=1s
2021-07-06T11:39:12.083Z [INFO]  registering service
2021-07-06T11:39:12.083Z [ERROR] Put "http://127.0.0.1:8500/v1/agent/service/register": dial tcp 127.0.0.1:8500: connect: connection refused: retry=1s
2021-07-06T11:39:13.083Z [INFO]  registering service
2021-07-06T11:39:13.095Z [INFO]  registering proxy
2021-07-06T11:39:13.287Z [INFO]  service and proxy registered successfully: name=example-client-app id=example-client-app-924d62223ea94fdca540626e0ec3a909
2021-07-06T11:39:13.688Z [INFO]  envoy bootstrap config written: file=/consul/envoy-bootstrap.json
<End of log>

consul-client appears to have started up the connection successfully as well:

==> Starting Consul agent...
Version: '1.9.5'
Node ID: 'eff48609-2388-5767-44ce-2a4b3738db20'
Node name: 'ip-10-0-0-50.ec2.internal'
Datacenter: 'dc1' (Segment: '')
Server: false (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
Cluster Addr: 10.0.0.50 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
==> Log data will now stream in as it occurs:
2021-07-06T11:39:12.179Z [WARN]  agent: Node name "ip-10-0-0-50.ec2.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
2021-07-06T11:39:12.191Z [WARN]  agent.auto_config: Node name "ip-10-0-0-50.ec2.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
2021-07-06T11:39:12.280Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-10-0-0-50.ec2.internal 10.0.0.50
2021-07-06T11:39:12.280Z [INFO]  agent.router: Initializing LAN area manager
2021-07-06T11:39:12.280Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=tcp
2021-07-06T11:39:12.281Z [INFO]  agent: Started DNS server: address=0.0.0.0:8600 network=udp
2021-07-06T11:39:12.281Z [INFO]  agent: Starting server: address=[::]:8500 network=tcp protocol=http
2021-07-06T11:39:12.281Z [INFO]  agent: started state syncer
==> Consul agent running!
2021-07-06T11:39:12.281Z [INFO]  agent: Started gRPC server: address=[::]:8502 network=tcp
2021-07-06T11:39:12.281Z [INFO]  agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
2021-07-06T11:39:12.281Z [INFO]  agent: Joining cluster...: cluster=LAN
2021-07-06T11:39:12.281Z [INFO]  agent: (LAN) joining: lan_addresses=[10.0.0.239]
2021-07-06T11:39:12.281Z [WARN]  agent.router.manager: No servers available
2021-07-06T11:39:12.281Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
2021-07-06T11:39:12.283Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-10-0-0-239.ec2.internal 10.0.0.239
2021-07-06T11:39:12.283Z [INFO]  agent: (LAN) joined: number_of_nodes=1
2021-07-06T11:39:12.283Z [INFO]  agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=1
2021-07-06T11:39:12.284Z [INFO]  agent.client: adding server: server="ip-10-0-0-239.ec2.internal (Addr: tcp/10.0.0.239:8300) (DC: dc1)"
2021-07-06T11:39:13.092Z [INFO]  agent: Synced node info
2021-07-06T11:39:13.094Z [INFO]  agent: Synced service: service=example-client-app-924d62223ea94fdca540626e0ec3a909
2021-07-06T11:39:13.286Z [INFO]  agent: Synced service: service=example-client-app-924d62223ea94fdca540626e0ec3a909-sidecar-proxy
2021-07-06T11:39:22.098Z [INFO]  agent: Synced check: check=service:example-client-app-924d62223ea94fdca540626e0ec3a909-sidecar-proxy:1
2021-07-06T11:39:40.594Z [INFO]  agent: Newer Consul version available: new_version=1.10.0 current_version=1.9.5
<End of log>

But it turns out that the pending sidecar-proxy is in fact generating logs (apologies for the wall of text):

[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:305] initializing epoch 0 (base id=0, hot restart version=11.104)
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:307] statically linked extensions:
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.dynamo, envoy.filters.http.ext_authz, envoy.filters.http.fault, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.gzip, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.oauth, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.squash, envoy.filters.http.tap, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.quic_server_codec: quiche
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.upstreams: envoy.filters.connection_pools.http.generic, envoy.filters.connection_pools.http.http, envoy.filters.connection_pools.http.tcp
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.udp_listeners: quiche_quic_listener, raw_udp_listener
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.mysql_proxy, envoy.filters.network.postgres_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.rocketmq_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.dubbo_proxy.protocols: dubbo
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.resolvers: envoy.ip
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, tls
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.thrift_proxy.filters: envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.dubbo_proxy.route_matchers: default
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.stats_sinks: envoy.dog_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.statsd
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.udp_packet_writers: udp_default_writer, udp_gso_batch_writer
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.bootstrap: envoy.extensions.network.socket_interface.default_socket_interface
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.compression.decompressor: envoy.compression.gzip.decompressor
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.quic_client_codec: quiche
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.tcp_grpc, envoy.file_access_log, envoy.http_grpc_access_log, envoy.tcp_grpc_access_log
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.compression.compressor: envoy.compression.gzip.compressor
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.health_checkers: envoy.health_checkers.redis
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.http.cache: envoy.extensions.http.cache.simple
[2021-07-06 11:39:14.975][8][info][main] [source/server/server.cc:309]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2021-07-06 11:39:14.981][8][warning][misc] [source/common/protobuf/utility.cc:294] Configuration does not parse cleanly as v3. v2 configuration is deprecated and will be removed from Envoy at the start of Q1 2021: Unknown field in: {
"admin": {
"access_log_path": "/dev/null",
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 19000
}
}
},
"node": {
"cluster": "example-client-app",
"id": "example-client-app-924d62223ea94fdca540626e0ec3a909-sidecar-proxy",
"metadata": {
"namespace": "default",
"envoy_version": "1.16.2"
}
},
"static_resources": {
"clusters": [
{
"name": "local_agent",
"connect_timeout": "1s",
"type": "STATIC",
"http2_protocol_options": {},
"hosts": [
{
"socket_address": {
"address": "127.0.0.1",
"port_value": 8502
}
}
]
}
]
},
"stats_config": {
"stats_tags": [
{
"regex": "^cluster\\.((?:([^.]+)~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
"tag_name": "consul.destination.custom_hash"
},
{
"regex": "^cluster\\.((?:[^.]+~)?(?:([^.]+)\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
"tag_name": "consul.destination.service_subset"
},
{
"regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?([^.]+)\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
"tag_name": "consul.destination.service"
},
{
"regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
"tag_name": "consul.destination.namespace"
},
{
"regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.([^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)",
"tag_name": "consul.destination.datacenter"
},
{
"regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.([^.]+)\\.[^.]+\\.consul\\.)",
"tag_name": "consul.destination.routing_type"
},
{
"regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.([^.]+)\\.consul\\.)",
"tag_name": "consul.destination.trust_domain"
},
{
"regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)",
"tag_name": "consul.destination.target"
},
{
"regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+)\\.consul\\.)",
"tag_name": "consul.destination.full_target"
},
{
"regex": "^(?:tcp\|http)\\.upstream\\.(([^.]+)(?:\\.[^.]+)?\\.[^.]+\\.)",
"tag_name": "consul.upstream.service"
},
{
"regex": "^(?:tcp\|http)\\.upstream\\.([^.]+(?:\\.[^.]+)?\\.([^.]+)\\.)",
"tag_name": "consul.upstream.datacenter"
},
{
"regex": "^(?:tcp\|http)\\.upstream\\.([^.]+(?:\\.([^.]+))?\\.[^.]+\\.)",
"tag_name": "consul.upstream.namespace"
},
{
"regex": "^cluster\\.((?:([^.]+)~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
"tag_name": "consul.custom_hash"
},
{
"regex": "^cluster\\.((?:[^.]+~)?(?:([^.]+)\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
"tag_name": "consul.service_subset"
},
{
"regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?([^.]+)\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
"tag_name": "consul.service"
},
{
"regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
"tag_name": "consul.namespace"
},
{
"regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.([^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)",
"tag_name": "consul.datacenter"
},
{
"regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.([^.]+)\\.[^.]+\\.consul\\.)",
"tag_name": "consul.routing_type"
},
{
"regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.([^.]+)\\.consul\\.)",
"tag_name": "consul.trust_domain"
},
{
"regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)",
"tag_name": "consul.target"
},
{
"regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.[^.]+\\.[^.]+)\\.consul\\.)",
"tag_name": "consul.full_target"
},
{
"tag_name": "local_cluster",
"fixed_value": "example-client-app"
},
{
"tag_name": "consul.source.service",
"fixed_value": "example-client-app"
},
{
"tag_name": "consul.source.namespace",
"fixed_value": "default"
},
{
"tag_name": "consul.source.datacenter",
"fixed_value": "dc1"
}
],
"use_all_default_tags": true
},
"dynamic_resources": {
"lds_config": {
"ads": {}
},
"cds_config": {
"ads": {}
},
"ads_config": {
"api_type": "GRPC",
"grpc_services": {
"initial_metadata": [
{
"key": "x-consul-token",
"value": ""
}
],
"envoy_grpc": {
"cluster_name": "local_agent"
}
}
}
},
"layered_runtime": {
"layers": [
{
"name": "static_layer",
"static_layer": {
"envoy.deprecated_features:envoy.api.v2.Cluster.tls_context": true,
"envoy.deprecated_features:envoy.config.trace.v2.ZipkinConfig.HTTP_JSON_V1": true,
"envoy.deprecated_features:envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager.Tracing.operation_name": true
}
}
]
}
}
[2021-07-06 11:39:14.982][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.Cluster Using deprecated option 'envoy.api.v2.Cluster.hosts' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:39:14.982][8][info][main] [source/server/server.cc:325] HTTP header map info:
[2021-07-06 11:39:14.984][8][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2021-07-06 11:39:14.984][8][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2021-07-06 11:39:14.984][8][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2021-07-06 11:39:14.984][8][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2021-07-06 11:39:14.984][8][info][main] [source/server/server.cc:328]   request header map: 608 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-method,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-proto,x-ot-span-context,x-request-id
[2021-07-06 11:39:14.984][8][info][main] [source/server/server.cc:328]   request trailer map: 128 bytes:
[2021-07-06 11:39:14.984][8][info][main] [source/server/server.cc:328]   response header map: 424 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2021-07-06 11:39:14.984][8][info][main] [source/server/server.cc:328]   response trailer map: 152 bytes: grpc-message,grpc-status
[2021-07-06 11:39:15.079][8][info][main] [source/server/server.cc:448] admin address: 127.0.0.1:19000
[2021-07-06 11:39:15.081][8][info][main] [source/server/server.cc:583] runtime: layers:
- name: static_layer
static_layer:
envoy.deprecated_features:envoy.config.trace.v2.ZipkinConfig.HTTP_JSON_V1: true
envoy.deprecated_features:envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager.Tracing.operation_name: true
envoy.deprecated_features:envoy.api.v2.Cluster.tls_context: true
[2021-07-06 11:39:15.082][8][info][config] [source/server/configuration_impl.cc:95] loading tracing configuration
[2021-07-06 11:39:15.082][8][info][config] [source/server/configuration_impl.cc:70] loading 0 static secret(s)
[2021-07-06 11:39:15.082][8][info][config] [source/server/configuration_impl.cc:76] loading 1 cluster(s)
[2021-07-06 11:39:15.483][8][info][config] [source/server/configuration_impl.cc:80] loading 0 listener(s)
[2021-07-06 11:39:15.483][8][info][config] [source/server/configuration_impl.cc:121] loading stats sink configuration
[2021-07-06 11:39:15.483][8][info][runtime] [source/common/runtime/runtime_impl.cc:421] RTDS has finished initialization
[2021-07-06 11:39:15.483][8][info][upstream] [source/common/upstream/cluster_manager_impl.cc:174] cm init: initializing cds
[2021-07-06 11:39:15.484][8][warning][main] [source/server/server.cc:565] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
[2021-07-06 11:39:15.484][8][info][main] [source/server/server.cc:679] starting main dispatch loop
[2021-07-06 11:39:15.579][8][warning][misc] [source/common/protobuf/utility.cc:294] Configuration does not parse cleanly as v3. v2 configuration is deprecated and will be removed from Envoy at the start of Q1 2021: envoy.api.v2.Cluster
[2021-07-06 11:39:15.579][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.Cluster Using runtime overrides to continue using now fatal-by-default deprecated option 'envoy.api.v2.Cluster.tls_context' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:39:15.580][8][info][upstream] [source/common/upstream/cds_api_impl.cc:64] cds: add 2 cluster(s), remove 1 cluster(s)
[2021-07-06 11:39:15.977][8][info][upstream] [source/common/upstream/cds_api_impl.cc:79] cds: add/update cluster 'local_app'
[2021-07-06 11:39:16.578][8][info][upstream] [source/common/upstream/cds_api_impl.cc:79] cds: add/update cluster 'example-server-app.default.dc1.internal.cce5f9df-8e0a-aaac-e896-4ecdaa2f3c0b.consul'
[2021-07-06 11:39:16.578][8][info][upstream] [source/common/upstream/cluster_manager_impl.cc:154] cm init: initializing secondary clusters
[2021-07-06 11:39:16.585][8][warning][misc] [source/common/protobuf/utility.cc:294] Configuration does not parse cleanly as v3. v2 configuration is deprecated and will be removed from Envoy at the start of Q1 2021: envoy.api.v2.ClusterLoadAssignment
[2021-07-06 11:39:16.585][8][info][upstream] [source/common/upstream/cluster_manager_impl.cc:178] cm init: all clusters initialized
[2021-07-06 11:39:16.586][8][info][main] [source/server/server.cc:660] all clusters initialized. initializing init manager
[2021-07-06 11:39:16.681][8][warning][misc] [source/common/protobuf/utility.cc:294] Configuration does not parse cleanly as v3. v2 configuration is deprecated and will be removed from Envoy at the start of Q1 2021: envoy.api.v2.Listener
[2021-07-06 11:39:16.681][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.FilterChain Using deprecated option 'envoy.api.v2.listener.FilterChain.tls_context' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:39:16.681][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:39:16.681][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:39:16.681][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:39:16.685][8][warning][misc] [bazel-out/k8-opt/bin/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:65] Using deprecated extension name 'envoy.tcp_proxy' for 'envoy.filters.network.tcp_proxy'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2021-07-06 11:39:16.686][8][info][upstream] [source/server/lds_api.cc:79] lds: add/update listener 'public_listener:0.0.0.0:20000'
[2021-07-06 11:39:16.687][8][warning][misc] [bazel-out/k8-opt/bin/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:65] Using deprecated extension name 'envoy.tcp_proxy' for 'envoy.filters.network.tcp_proxy'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2021-07-06 11:39:16.687][8][info][upstream] [source/server/lds_api.cc:79] lds: add/update listener 'example-server-app:127.0.0.1:1234'
[2021-07-06 11:39:16.687][8][info][config] [source/server/listener_manager_impl.cc:888] all dependencies initialized. starting workers
[2021-07-06 11:44:19.734][8][warning][misc] [source/common/protobuf/utility.cc:294] Configuration does not parse cleanly as v3. v2 configuration is deprecated and will be removed from Envoy at the start of Q1 2021: envoy.api.v2.Cluster
[2021-07-06 11:44:19.734][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.Cluster Using runtime overrides to continue using now fatal-by-default deprecated option 'envoy.api.v2.Cluster.tls_context' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:44:19.734][8][info][upstream] [source/common/upstream/cds_api_impl.cc:64] cds: add 2 cluster(s), remove 1 cluster(s)
[2021-07-06 11:44:19.735][8][warning][misc] [source/common/protobuf/utility.cc:294] Configuration does not parse cleanly as v3. v2 configuration is deprecated and will be removed from Envoy at the start of Q1 2021: envoy.api.v2.ClusterLoadAssignment
[2021-07-06 11:44:19.735][8][warning][misc] [source/common/protobuf/utility.cc:294] Configuration does not parse cleanly as v3. v2 configuration is deprecated and will be removed from Envoy at the start of Q1 2021: envoy.api.v2.Listener
[2021-07-06 11:44:19.735][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.FilterChain Using deprecated option 'envoy.api.v2.listener.FilterChain.tls_context' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:44:19.735][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:44:19.735][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:44:19.735][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:49:28.365][8][warning][misc] [source/common/protobuf/utility.cc:294] Configuration does not parse cleanly as v3. v2 configuration is deprecated and will be removed from Envoy at the start of Q1 2021: envoy.api.v2.Cluster
[2021-07-06 11:49:28.366][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.Cluster Using runtime overrides to continue using now fatal-by-default deprecated option 'envoy.api.v2.Cluster.tls_context' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:49:28.366][8][info][upstream] [source/common/upstream/cds_api_impl.cc:64] cds: add 2 cluster(s), remove 1 cluster(s)
[2021-07-06 11:49:28.366][8][warning][misc] [source/common/protobuf/utility.cc:294] Configuration does not parse cleanly as v3. v2 configuration is deprecated and will be removed from Envoy at the start of Q1 2021: envoy.api.v2.ClusterLoadAssignment
[2021-07-06 11:49:28.366][8][warning][misc] [source/common/protobuf/utility.cc:294] Configuration does not parse cleanly as v3. v2 configuration is deprecated and will be removed from Envoy at the start of Q1 2021: envoy.api.v2.Listener
[2021-07-06 11:49:28.366][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.FilterChain Using deprecated option 'envoy.api.v2.listener.FilterChain.tls_context' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:49:28.366][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:49:28.366][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:49:28.366][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:49:31.967][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.Cluster Using runtime overrides to continue using now fatal-by-default deprecated option 'envoy.api.v2.Cluster.tls_context' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:49:31.967][8][info][upstream] [source/common/upstream/cds_api_impl.cc:64] cds: add 2 cluster(s), remove 1 cluster(s)
[2021-07-06 11:49:31.967][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.FilterChain Using deprecated option 'envoy.api.v2.listener.FilterChain.tls_context' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:49:31.968][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:49:31.968][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:49:31.968][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:54:16.681][8][info][main] [source/server/drain_manager_impl.cc:70] shutting down parent after drain
[2021-07-06 11:54:35.181][8][warning][misc] [source/common/protobuf/utility.cc:294] Configuration does not parse cleanly as v3. v2 configuration is deprecated and will be removed from Envoy at the start of Q1 2021: envoy.api.v2.Cluster
[2021-07-06 11:54:35.182][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.Cluster Using runtime overrides to continue using now fatal-by-default deprecated option 'envoy.api.v2.Cluster.tls_context' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:54:35.182][8][info][upstream] [source/common/upstream/cds_api_impl.cc:64] cds: add 2 cluster(s), remove 1 cluster(s)
[2021-07-06 11:54:35.182][8][warning][misc] [source/common/protobuf/utility.cc:294] Configuration does not parse cleanly as v3. v2 configuration is deprecated and will be removed from Envoy at the start of Q1 2021: envoy.api.v2.ClusterLoadAssignment
[2021-07-06 11:54:35.182][8][warning][misc] [source/common/protobuf/utility.cc:294] Configuration does not parse cleanly as v3. v2 configuration is deprecated and will be removed from Envoy at the start of Q1 2021: envoy.api.v2.Listener
[2021-07-06 11:54:35.182][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.FilterChain Using deprecated option 'envoy.api.v2.listener.FilterChain.tls_context' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:54:35.182][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:54:35.182][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2021-07-06 11:54:35.182][8][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.api.v2.listener.Filter Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.

I do see a lot of deprecated warnings for the Envoy config, but no error messages at all, which is fairly odd.

lkysow commented 3 years ago

Hmm. Has mesh-init exited yet? This is what the status of the containers should look like:

PedroGuerraPT commented 3 years ago

No, it is still running, unfortunately. It appears that mesh-init stays in pending status even upon starting successfully.

Maybe there is something preventing mesh-init from exiting when using the retry _join input?

Edit: My containers only have the same status as shown in your image when using the consul_server_service_name input.

lkysow commented 3 years ago

Maybe there is something preventing mesh-init from exiting when using the retry _join input?

There shouldn't be! But yeah perhaps there is.

2021-07-06T11:39:13.688Z [INFO] envoy bootstrap config written: file=/consul/envoy-bootstrap.json

This is supposed to be the last log.

We'll try and set up a reproduction and get back to you.

lkysow commented 3 years ago

Question: are you running these tasks on ECS Fargate or ECS EC2.

PedroGuerraPT commented 3 years ago

Question: are you running these tasks on ECS Fargate or ECS EC2.

ECS Fargate.

pglass commented 3 years ago

We weren't able to reproduce this so far, though we tried with the Consul server in Fargate using retry_join (as opposed to running it on an EC2 instance).

Just to double check a couple of things:

Is the example app working or not? Are you able open the client app through the mesh_client_lb_address (found in the Terraform outputs). It should look like the following:
And, your initial screenshot of the task statuses has a "last updated" time of 27m ago, so we wanted to make sure you've been refreshing the task statuses?

Sorry if those are basic questions. Everything looked normal log-wise, aside from the task statuses. Let us know!

PedroGuerraPT commented 3 years ago

@pglass I'll try to test the example app as soon as possible, I wasn't able to work on this today.

But regarding the container status, I do have been pressing the refresh button, waiting for the status update, but all containers remain in the state that I've shown on the original post.

I don't mind the basic questions at all, I'm sure that I'm doing something wrong on my environment, so clearing out the basic stuff helps me a lot!

PedroGuerraPT commented 3 years ago

Hi, unfortunately I'm unable to look into this at the moment, so I'll close the issue for now.

I'll reopen the issue when I'm able to investigate further, thanks anyway for your help.

hashicorp / terraform-aws-consul-ecs

sidecar-proxy container remains in "Pending" status when using retry_join #18