search

solo-io / gloo

The Feature-rich, Kubernetes-native, Next-Generation API Gateway Built on Envoy
https://docs.solo.io/
Apache License 2.0
4.07k stars 437 forks source link

Gateway proxy's listeners get removed and all traffic stops #3106

Closed cweidinger closed 3 months ago

cweidinger commented 4 years ago

Describe the bug

  1. gateway proxy had it's listener removed for unknown reason (probably configuration that it doesn't like) and it didn't automatically come back.

  2. When the gateway proxy boots, it doesn't add a listener if there's configuration that it doesn't like.

To Reproduce We have 2 gloos (gloo-admin and gloo-web) that both watch themselves, default and apps namespaces. We have a virtual service in gloo-admin to a service in "apps" namespace that is apparently causing some intermittent removal of listeners from gateway proxy in gloo-web. gloo-web has a virtualservice that is not working because of the errors. There may be other causes that we just didn't see in the logs.

Expected behavior We expect that the gateway proxy listeners to always be there. If they should fail for one reason or another gloo should automatically restart it so that all traffic is not lost.

We temporarily fixed the issue by restarting all gloo deployments but that's not going to work in production. We also fixed the problem by removing the service in the apps namespace that had a virtual service in gloo-admin but we need that setup.

Additional context Add any other context about the problem here, e.g.


k exec -it gateway-proxy-6bbb9788f6-mlk2n sh
/ # netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:19000         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN
tcp        0      0 10.31.49.125:40518      172.20.191.255:9966     ESTABLISHED
tcp        0      0 10.31.49.125:8081       10.31.49.22:38054       ESTABLISHED
tcp        0      0 127.0.0.1:19000         127.0.0.1:59272         ESTABLISHED
tcp        0      0 127.0.0.1:59272         127.0.0.1:19000         ESTABLISHED
tcp        0      0 10.31.49.125:48294      172.20.191.255:9977     ESTABLISHED
13:31

kubectl logs gateway-proxy-6bbb9788f6-mlk2n -n gloo-web
[2020-06-03 16:24:42.330][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:93] cds: add/update cluster 'apps-argocd-apps-redis-6379_gloo-web'
[2020-06-03 16:28:45.258][6][info][upstream] [external/envoy/source/server/lds_api.cc:58] lds: remove listener 'listener-::-443'
[2020-06-03 16:28:45.258][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:77] cds: add 0 cluster(s), remove 50 cluster(s)
[2020-06-03 16:28:45.258][6][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:626] removing cluster kube-svc:gloo-web-gloo-9966_gloo-web

kubectl logs gloo-778dcbfdf6-xfj56 -n gloo-web
{"level":"warn","ts":1591210328.3338408,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"proxy gloo-web.gateway-proxy was rejected due to invalid config: 3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n\nAttempting to update only EDS information","version":"1.4.0-beta13"}
cweidinger commented 4 years ago

Also we spin down node groups to 0 at night for some of our clusters and when gloo comes back up the gateway proxy gets to "all dependencies initialized. starting workers" but not to "add/update listener"

gloo-admin namespace which is okay

$ k logs gateway-proxy-7fb896fbdc-6x5dl -n gloo-admin
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:293] initializing epoch 0 (hot restart version=disabled)
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:295] statically linked extensions:
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.dynamo, envoy.filters.http.ext_authz, envoy.filters.http.fault, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.gzip, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.lua, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.squash, envoy.filters.http.tap, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash, io.solo.aws_lambda, io.solo.nats_streaming, io.solo.transformation
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.mysql_proxy, envoy.filters.network.postgres_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.rocketmq_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.udp_listeners: raw_udp_listener
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.dubbo_proxy.route_matchers: default
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.resolvers: envoy.ip
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.compression.compressor: envoy.compression.gzip.compressor
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.compression.decompressor: envoy.compression.gzip.decompressor
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.health_checkers: envoy.health_checkers.redis
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.dubbo_proxy.protocols: dubbo
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.tcp_grpc, envoy.file_access_log, envoy.http_grpc_access_log, envoy.tcp_grpc_access_log
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.stats_sinks: envoy.dog_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.statsd
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.thrift_proxy.filters: envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2020-06-04 12:02:59.607][6][info][main] [external/envoy/source/server/server.cc:297]   http_cache_factory: envoy.extensions.http.cache.simple
[2020-06-04 12:02:59.622][6][warning][misc] [external/envoy/source/common/protobuf/utility.cc:200] Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:02:59.622][6][warning][misc] [external/envoy/source/common/protobuf/utility.cc:200] Using deprecated option 'envoy.config.metrics.v2.StatsSink.config' from file stats.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:02:59.623][6][info][main] [external/envoy/source/server/server.cc:378] admin address: 127.0.0.1:19000
[2020-06-04 12:02:59.624][6][info][main] [external/envoy/source/server/server.cc:525] runtime: layers:
  - name: base
    static_layer:
      {}
  - name: admin
    admin_layer:
      {}
[2020-06-04 12:02:59.624][6][info][config] [external/envoy/source/server/configuration_impl.cc:103] loading tracing configuration
[2020-06-04 12:02:59.624][6][info][config] [external/envoy/source/server/configuration_impl.cc:69] loading 0 static secret(s)
[2020-06-04 12:02:59.624][6][info][config] [external/envoy/source/server/configuration_impl.cc:75] loading 4 cluster(s)
[2020-06-04 12:02:59.628][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:02:59.628][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:02:59.628][6][info][config] [external/envoy/source/server/configuration_impl.cc:79] loading 1 listener(s)
[2020-06-04 12:02:59.629][6][warning][misc] [bazel-out/k8-opt/bin/external/envoy/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:66] Using deprecated extension name 'envoy.http_connection_manager' for 'envoy.filters.network.http_connection_manager'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:02:59.631][6][warning][misc] [external/envoy/source/common/protobuf/utility.cc:200] Using deprecated option 'envoy.config.filter.network.http_connection_manager.v2.HttpFilter.config' from file http_connection_manager.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:02:59.633][6][warning][misc] [bazel-out/k8-opt/bin/external/envoy/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:66] Using deprecated extension name 'envoy.router' for 'envoy.filters.http.router'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:02:59.633][6][info][config] [external/envoy/source/server/configuration_impl.cc:129] loading stats sink configuration
[2020-06-04 12:02:59.633][6][warning][misc] [bazel-out/k8-opt/bin/external/envoy/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:66] Using deprecated extension name 'envoy.metrics_service' for 'envoy.stat_sinks.metrics_service'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:02:59.633][6][info][main] [external/envoy/source/server/server.cc:622] starting main dispatch loop
[2020-06-04 12:02:59.732][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:02:59.732][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:03:01.629][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:03:01.629][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:03:09.979][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:03:09.979][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:03:14.643][6][info][runtime] [external/envoy/source/common/runtime/runtime_impl.cc:526] RTDS has finished initialization
[2020-06-04 12:03:14.643][6][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:174] cm init: initializing cds
[2020-06-04 12:03:18.222][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:03:18.222][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:03:23.295][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:03:23.295][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:03:28.915][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:03:28.915][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:03:29.647][6][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:178] cm init: all clusters initialized
[2020-06-04 12:03:29.647][6][info][main] [external/envoy/source/server/server.cc:601] all clusters initialized. initializing init manager
[2020-06-04 12:03:43.659][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:03:43.659][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:03:44.652][6][info][config] [external/envoy/source/server/listener_manager_impl.cc:836] all dependencies initialized. starting workers
[2020-06-04 12:03:54.988][6][warning][misc] [external/envoy/source/common/protobuf/utility.cc:200] Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:03:54.998][6][warning][misc] [bazel-out/k8-opt/bin/external/envoy/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:66] Using deprecated extension name 'envoy.http_connection_manager' for 'envoy.filters.network.http_connection_manager'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:03:55.001][6][warning][misc] [external/envoy/source/common/protobuf/utility.cc:200] Using deprecated option 'envoy.config.filter.network.http_connection_manager.v2.HttpFilter.config' from file http_connection_manager.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:03:55.004][6][warning][misc] [bazel-out/k8-opt/bin/external/envoy/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:66] Using deprecated extension name 'envoy.fault' for 'envoy.filters.http.fault'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:03:55.007][6][warning][misc] [bazel-out/k8-opt/bin/external/envoy/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:66] Using deprecated extension name 'envoy.cors' for 'envoy.filters.http.cors'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:03:55.012][6][warning][misc] [bazel-out/k8-opt/bin/external/envoy/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:66] Using deprecated extension name 'envoy.grpc_web' for 'envoy.filters.http.grpc_web'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:03:55.012][6][warning][misc] [bazel-out/k8-opt/bin/external/envoy/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:66] Using deprecated extension name 'envoy.health_check' for 'envoy.filters.http.health_check'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:03:55.012][6][warning][misc] [bazel-out/k8-opt/bin/external/envoy/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:66] Using deprecated extension name 'envoy.router' for 'envoy.filters.http.router'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:03:55.019][6][info][upstream] [external/envoy/source/server/lds_api.cc:78] lds: add/update listener 'listener-::-443'
[2020-06-04 12:03:55.046][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:78] cds: add 102 cluster(s), remove 4 cluster(s)
[2020-06-04 12:03:55.048][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:apps-podinfo-primary-9898_apps'
[2020-06-04 12:03:55.055][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-argocd-admin-server-80_gloo-admin'
[2020-06-04 12:03:55.056][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-prometheus-operator-8080_gloo-admin'
[2020-06-04 12:03:55.078][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'gloo-admin-gateway-proxy-svc-8081_gloo-admin'
[2020-06-04 12:03:55.084][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:gloo-admin-gloo-9977_gloo-admin'
[2020-06-04 12:03:55.087][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-system-cluster-autoscaler-aws-cluster-autoscaler-8085_gloo-admin'
[2020-06-04 12:03:55.088][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:kube-system-kube-dns-53_kube-system'
[2020-06-04 12:03:55.091][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-alertmanager-operated-9094_admin'
[2020-06-04 12:03:55.093][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-prometheus-alertmanager-9093_admin'
[2020-06-04 12:03:55.094][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-prometheus-kube-state-metrics-8080_admin'
[2020-06-04 12:03:55.096][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:apps-argocd-apps-application-controller-8082_apps'
[2020-06-04 12:03:55.098][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:apps-argocd-apps-dex-server-5557_apps'
[2020-06-04 12:03:55.099][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:apps-podinfo-canary-9898_apps'
[2020-06-04 12:03:55.101][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-flagger-loadtester-80_gloo-admin'
[2020-06-04 12:03:55.102][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'apps-argocd-apps-server-metrics-8083_gloo-admin'
[2020-06-04 12:03:55.104][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-argocd-admin-server-80_admin'
[2020-06-04 12:03:55.106][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:gloo-admin-gloo-9966_gloo-admin'
[2020-06-04 12:03:55.107][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:kube-system-prometheus-coredns-9153_kube-system'
[2020-06-04 12:03:55.109][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'gloo-admin-gateway-proxy-svc-443_gloo-admin'
[2020-06-04 12:03:55.110][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:default-kubernetes-443_default'
[2020-06-04 12:03:55.116][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-alertmanager-operated-9093_gloo-admin'
[2020-06-04 12:03:55.128][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'apps-argocd-apps-repo-server-8081_gloo-admin'
[2020-06-04 12:03:55.129][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'apps-podinfo-primary-9898_gloo-admin'
[2020-06-04 12:03:55.131][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'gloo-admin-gloo-9966_gloo-admin'
[2020-06-04 12:03:55.132][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-system-kube-dns-53_gloo-admin'
[2020-06-04 12:03:55.134][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-prometheus-grafana-80_admin'
[2020-06-04 12:03:55.135][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-prometheus-operated-9090_admin'
[2020-06-04 12:03:55.137][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-alertmanager-operated-9094_gloo-admin'
[2020-06-04 12:03:55.138][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-argocd-admin-dex-server-5557_gloo-admin'
[2020-06-04 12:03:55.140][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-argocd-admin-server-443_gloo-admin'
[2020-06-04 12:03:55.141][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'default-kubernetes-443_gloo-admin'
[2020-06-04 12:03:55.143][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:gloo-admin-gateway-proxy-svc-443_gloo-admin'
[2020-06-04 12:03:55.145][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-argocd-admin-application-controller-8082_admin'
[2020-06-04 12:03:55.147][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:apps-argocd-apps-server-metrics-8083_apps'
[2020-06-04 12:03:55.149][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-prometheus-grafana-80_gloo-admin'
[2020-06-04 12:03:55.150][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'gloo-admin-f5vip-443_gloo-admin'
[2020-06-04 12:03:55.152][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'gloo-admin-gateway-proxy-svc-80_gloo-admin'
[2020-06-04 12:03:55.153][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-argocd-admin-dex-server-5557_admin'
[2020-06-04 12:03:55.155][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-argocd-admin-redis-6379_admin'
[2020-06-04 12:03:55.156][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:apps-argocd-apps-dex-server-5556_apps'
[2020-06-04 12:03:55.176][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:apps-podinfo-9898_apps'
[2020-06-04 12:03:55.178][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-argocd-admin-dex-server-5556_gloo-admin'
[2020-06-04 12:03:55.181][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'default-k8s-health-8080_gloo-admin'
[2020-06-04 12:03:55.185][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:gloo-admin-gateway-proxy-svc-8081_gloo-admin'
[2020-06-04 12:03:55.187][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-alertmanager-operated-9093_admin'
[2020-06-04 12:03:55.189][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:apps-argocd-apps-redis-6379_apps'
[2020-06-04 12:03:55.191][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:apps-argocd-apps-server-443_apps'
[2020-06-04 12:03:55.193][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-argocd-admin-server-metrics-8083_gloo-admin'
[2020-06-04 12:03:55.195][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-godaddy-k8s-external-secr-c579211f1021299f5f82334bcd0678b_gloo-admin'
[2020-06-04 12:03:55.197][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:gloo-admin-cert-manager-gloo-admin-9402_gloo-admin'
[2020-06-04 12:03:55.199][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:kube-system-cluster-autoscaler-aws-cluster-autoscaler-8085_kube-system'
[2020-06-04 12:03:55.201][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-argocd-admin-dex-server-5556_admin'
[2020-06-04 12:03:55.202][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:default-k8s-health-8080_default'
[2020-06-04 12:03:55.204][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-argocd-admin-repo-server-8081_gloo-admin'
[2020-06-04 12:03:55.206][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-prometheus-kube-state-metrics-8080_gloo-admin'
[2020-06-04 12:03:55.207][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'apps-argocd-apps-redis-6379_gloo-admin'
[2020-06-04 12:03:55.210][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'apps-argocd-apps-server-80_gloo-admin'
[2020-06-04 12:03:55.212][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'apps-podinfo-9898_gloo-admin'
[2020-06-04 12:03:55.213][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'gloo-admin-vhi-pod-8080_gloo-admin'
[2020-06-04 12:03:55.214][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-metrics-server-443_admin'
[2020-06-04 12:03:55.216][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'apps-argocd-apps-server-443_gloo-admin'
[2020-06-04 12:03:55.217][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:gloo-admin-gateway-proxy-nodeport-443_gloo-admin'
[2020-06-04 12:03:55.220][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-system-prometheus-adapter-443_gloo-admin'
[2020-06-04 12:03:55.221][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'apps-argocd-apps-dex-server-5556_gloo-admin'
[2020-06-04 12:03:55.223][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:gloo-admin-f5vip-443_gloo-admin'
[2020-06-04 12:03:55.224][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-system-external-dns-7979_gloo-admin'
[2020-06-04 12:03:55.226][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-prometheus-prometheus-node-exporter-9100_admin'
[2020-06-04 12:03:55.227][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-metrics-server-443_gloo-admin'
[2020-06-04 12:03:55.229][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-prometheus-alertmanager-9093_gloo-admin'
[2020-06-04 12:03:55.230][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-prometheus-operated-9090_gloo-admin'
[2020-06-04 12:03:55.232][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:gloo-admin-gateway-proxy-svc-80_gloo-admin'
[2020-06-04 12:03:55.234][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:gloo-admin-gloo-9979_gloo-admin'
[2020-06-04 12:03:55.236][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:gloo-admin-gloo-9988_gloo-admin'
[2020-06-04 12:03:55.277][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-system-prometheus-coredns-9153_gloo-admin'
[2020-06-04 12:03:55.279][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-godaddy-k8s-external-secr-c579211f1021299f5f82334bcd0678b_admin'
[2020-06-04 12:03:55.282][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-argocd-admin-server-metrics-8083_admin'
[2020-06-04 12:03:55.284][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-flagger-loadtester-80_admin'
[2020-06-04 12:03:55.286][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-prometheus-prometheus-9090_admin'
[2020-06-04 12:03:55.288][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-prometheus-prometheus-node-exporter-9100_gloo-admin'
[2020-06-04 12:03:55.291][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-system-prometheus-kubelet-10250_gloo-admin'
[2020-06-04 12:03:55.293][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:kube-system-prometheus-adapter-443_kube-system'
[2020-06-04 12:03:55.296][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-argocd-admin-repo-server-8081_admin'
[2020-06-04 12:03:55.301][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'gloo-admin-cert-manager-gloo-admin-9402_gloo-admin'
[2020-06-04 12:03:55.302][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'gloo-admin-gloo-9977_gloo-admin'
[2020-06-04 12:03:55.304][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'gloo-admin-gloo-9979_gloo-admin'
[2020-06-04 12:03:55.305][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:gloo-admin-gateway-443_gloo-admin'
[2020-06-04 12:03:55.307][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'apps-argocd-apps-dex-server-5557_gloo-admin'
[2020-06-04 12:03:55.324][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-prometheus-prometheus-9090_gloo-admin'
[2020-06-04 12:03:55.329][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'gloo-admin-gloo-9988_gloo-admin'
[2020-06-04 12:03:55.332][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:gloo-admin-vhi-pod-8080_gloo-admin'
[2020-06-04 12:03:55.334][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:kube-system-external-dns-7979_kube-system'
[2020-06-04 12:03:55.336][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:apps-argocd-apps-server-80_apps'
[2020-06-04 12:03:55.340][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:apps-argocd-apps-repo-server-8081_apps'
[2020-06-04 12:03:55.341][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-argocd-admin-application-controller-8082_gloo-admin'
[2020-06-04 12:03:55.343][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'apps-podinfo-canary-9898_gloo-admin'
[2020-06-04 12:03:55.345][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:kube-system-prometheus-kubelet-10250_kube-system'
[2020-06-04 12:03:55.347][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-prometheus-operator-8080_admin'
[2020-06-04 12:03:55.348][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'admin-argocd-admin-redis-6379_gloo-admin'
[2020-06-04 12:03:55.350][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'apps-argocd-apps-application-controller-8082_gloo-admin'
[2020-06-04 12:03:55.352][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'gloo-admin-gateway-443_gloo-admin'
[2020-06-04 12:03:55.353][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'gloo-admin-gateway-proxy-nodeport-443_gloo-admin'
[2020-06-04 12:03:55.355][6][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:94] cds: add/update cluster 'kube-svc:admin-argocd-admin-server-443_admin'
[2020-06-04 12:18:44.656][6][info][main] [external/envoy/source/server/drain_manager_impl.cc:68] shutting down parent after drain

gloo-web gateway proxy which didn't ever get a listener

$ k logs gateway-proxy-6656c5458f-5tvvs -n gloo-web
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:293] initializing epoch 0 (hot restart version=disabled)
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:295] statically linked extensions:
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.compression.decompressor: envoy.compression.gzip.decompressor
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.stats_sinks: envoy.dog_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.statsd
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.thrift_proxy.filters: envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.mysql_proxy, envoy.filters.network.postgres_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.rocketmq_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   http_cache_factory: envoy.extensions.http.cache.simple
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.dubbo_proxy.protocols: dubbo
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.compression.compressor: envoy.compression.gzip.compressor
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.dynamo, envoy.filters.http.ext_authz, envoy.filters.http.fault, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.gzip, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.lua, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.squash, envoy.filters.http.tap, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash, io.solo.aws_lambda, io.solo.nats_streaming, io.solo.transformation
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.udp_listeners: raw_udp_listener
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.tcp_grpc, envoy.file_access_log, envoy.http_grpc_access_log, envoy.tcp_grpc_access_log
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.resolvers: envoy.ip
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.health_checkers: envoy.health_checkers.redis
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.dubbo_proxy.route_matchers: default
[2020-06-04 12:02:37.624][6][info][main] [external/envoy/source/server/server.cc:297]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2020-06-04 12:02:37.636][6][warning][misc] [external/envoy/source/common/protobuf/utility.cc:200] Using deprecated option 'envoy.api.v2.listener.Filter.config' from file listener_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:02:37.636][6][warning][misc] [external/envoy/source/common/protobuf/utility.cc:200] Using deprecated option 'envoy.config.metrics.v2.StatsSink.config' from file stats.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:02:37.637][6][info][main] [external/envoy/source/server/server.cc:378] admin address: 127.0.0.1:19000
[2020-06-04 12:02:37.638][6][info][main] [external/envoy/source/server/server.cc:525] runtime: layers:
  - name: base
    static_layer:
      {}
  - name: admin
    admin_layer:
      {}
[2020-06-04 12:02:37.638][6][info][config] [external/envoy/source/server/configuration_impl.cc:103] loading tracing configuration
[2020-06-04 12:02:37.638][6][info][config] [external/envoy/source/server/configuration_impl.cc:69] loading 0 static secret(s)
[2020-06-04 12:02:37.638][6][info][config] [external/envoy/source/server/configuration_impl.cc:75] loading 4 cluster(s)
[2020-06-04 12:02:37.689][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:02:37.690][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:02:37.690][6][info][config] [external/envoy/source/server/configuration_impl.cc:79] loading 1 listener(s)
[2020-06-04 12:02:37.690][6][warning][misc] [bazel-out/k8-opt/bin/external/envoy/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:66] Using deprecated extension name 'envoy.http_connection_manager' for 'envoy.filters.network.http_connection_manager'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:02:37.694][6][warning][misc] [external/envoy/source/common/protobuf/utility.cc:200] Using deprecated option 'envoy.config.filter.network.http_connection_manager.v2.HttpFilter.config' from file http_connection_manager.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:02:37.695][6][warning][misc] [bazel-out/k8-opt/bin/external/envoy/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:66] Using deprecated extension name 'envoy.router' for 'envoy.filters.http.router'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:02:37.695][6][info][config] [external/envoy/source/server/configuration_impl.cc:129] loading stats sink configuration
[2020-06-04 12:02:37.695][6][warning][misc] [bazel-out/k8-opt/bin/external/envoy/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:66] Using deprecated extension name 'envoy.metrics_service' for 'envoy.stat_sinks.metrics_service'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
[2020-06-04 12:02:37.695][6][info][main] [external/envoy/source/server/server.cc:622] starting main dispatch loop
[2020-06-04 12:02:37.773][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:02:37.773][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:02:41.694][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:02:41.695][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:02:52.705][6][info][runtime] [external/envoy/source/common/runtime/runtime_impl.cc:526] RTDS has finished initialization
[2020-06-04 12:02:52.705][6][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:174] cm init: initializing cds
[2020-06-04 12:02:54.784][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:02:54.784][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:03:07.708][6][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:178] cm init: all clusters initialized
[2020-06-04 12:03:07.708][6][info][main] [external/envoy/source/server/server.cc:601] all clusters initialized. initializing init manager
[2020-06-04 12:03:10.226][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:03:10.226][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:03:18.882][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:03:18.882][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:03:22.712][6][info][config] [external/envoy/source/server/listener_manager_impl.cc:836] all dependencies initialized. starting workers
[2020-06-04 12:03:42.385][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:03:42.385][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:03:51.483][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:03:51.483][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:03:51.944][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
[2020-06-04 12:03:51.944][6][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
[2020-06-04 12:18:22.714][6][info][main] [external/envoy/source/server/drain_manager_impl.cc:68] shutting down parent after drain
cweidinger commented 4 years ago

and the gloo logs for gloo-web which had the gateway-proxy whose listener didn't start up

$ k logs gloo-778dcbfdf6-4k9xf -n gloo-web
{"level":"info","ts":1591272159.8108993,"logger":"gloo.v1.event_loop","caller":"v1/setup_event_loop.sk.go:57","msg":"event loop started","version":"1.4.0-beta13"}
{"level":"info","ts":1591272160.4747615,"logger":"gloo.v1.event_loop.setup.v1.event_loop","caller":"v1/eds_event_loop.sk.go:57","msg":"event loop started","version":"1.4.0-beta13"}
{"level":"info","ts":1591272160.5869262,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/run.go:35","msg":"begin sync 733900675914526315 (46 upstreams)","version":"1.4.0-beta13"}
{"level":"info","ts":1591272160.5870788,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:188","msg":"Received first EDS update from plugin: *ec2.plugin","version":"1.4.0-beta13"}
{"level":"info","ts":1591272160.9874115,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/run.go:65","msg":"end sync 733900675914526315","version":"1.4.0-beta13"}
{"level":"info","ts":1591272160.9877446,"logger":"gloo.v1.event_loop.setup.v1.event_loop.syncer","caller":"discovery/discovery.go:188","msg":"Received first EDS update from plugin: *kubernetes.plugin","version":"1.4.0-beta13"}
{"level":"info","ts":1591272160.9889414,"logger":"gloo.v1.event_loop.setup.v1.event_loop","caller":"v1/api_event_loop.sk.go:57","msg":"event loop started","version":"1.4.0-beta13"}
{"level":"info","ts":1591272161.0465472,"logger":"gloo.v1.event_loop.gloo.metrics","caller":"runner/run.go:50","msg":"Starting metrics server","version":"1.4.0-beta13"}
{"level":"info","ts":1591272161.046597,"logger":"gloo.v1.event_loop.gloo.metrics","caller":"runner/run.go:56","msg":"metrics server running in [gRPC] mode, listening at [:9966]","version":"1.4.0-beta13"}
{"level":"info","ts":1591272161.0501056,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:77","msg":"begin sync 2201467656919314617 (1 proxies, 46 upstreams, 12 endpoints, 2 secrets, 7 artifacts, 0 auth configs)","version":"1.4.0-beta13"}
{"level":"info","ts":1591272161.0506399,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-443","version":"1.4.0-beta13"}
{"level":"info","ts":1591272161.054958,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-80","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272161.070232,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:133","msg":"Proxy had invalid config","version":"1.4.0-beta13","proxy":{"name":"gateway-proxy","namespace":"gloo-web"},"error":"3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n"}
{"level":"warn","ts":1591272161.0708892,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"proxy gloo-web.gateway-proxy was rejected due to invalid config: 3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n\nAttempting to update only EDS information","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272161.0709212,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"endpoint update failed. xDS snapshot for proxy gloo-web.gateway-proxy will not be updated. Error is: no snapshot found for node gloo-web~gateway-proxy","version":"1.4.0-beta13"}
{"level":"info","ts":1591272161.0935626,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:188","msg":"end sync 2201467656919314617","version":"1.4.0-beta13"}
{"level":"info","ts":1591272163.0505056,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:77","msg":"begin sync 1885724498719813910 (1 proxies, 46 upstreams, 20 endpoints, 2 secrets, 7 artifacts, 0 auth configs)","version":"1.4.0-beta13"}
{"level":"info","ts":1591272163.0512984,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-443","version":"1.4.0-beta13"}
{"level":"info","ts":1591272163.0542626,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-80","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272163.0759263,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:133","msg":"Proxy had invalid config","version":"1.4.0-beta13","proxy":{"name":"gateway-proxy","namespace":"gloo-web"},"error":"3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n"}
{"level":"warn","ts":1591272163.0769007,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"proxy gloo-web.gateway-proxy was rejected due to invalid config: 3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n\nAttempting to update only EDS information","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272163.0769467,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"endpoint update failed. xDS snapshot for proxy gloo-web.gateway-proxy will not be updated. Error is: no snapshot found for node gloo-web~gateway-proxy","version":"1.4.0-beta13"}
{"level":"info","ts":1591272163.07856,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:188","msg":"end sync 1885724498719813910","version":"1.4.0-beta13"}
{"level":"info","ts":1591272170.0484598,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:77","msg":"begin sync 8541343198279477792 (1 proxies, 46 upstreams, 22 endpoints, 2 secrets, 7 artifacts, 0 auth configs)","version":"1.4.0-beta13"}
{"level":"info","ts":1591272170.0490637,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-443","version":"1.4.0-beta13"}
{"level":"info","ts":1591272170.0510318,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-80","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272170.0617495,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:133","msg":"Proxy had invalid config","version":"1.4.0-beta13","proxy":{"name":"gateway-proxy","namespace":"gloo-web"},"error":"3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n"}
{"level":"warn","ts":1591272170.0623972,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"proxy gloo-web.gateway-proxy was rejected due to invalid config: 3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n\nAttempting to update only EDS information","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272170.0624309,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"endpoint update failed. xDS snapshot for proxy gloo-web.gateway-proxy will not be updated. Error is: no snapshot found for node gloo-web~gateway-proxy","version":"1.4.0-beta13"}
{"level":"info","ts":1591272170.0634768,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:188","msg":"end sync 8541343198279477792","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272175.031028,"logger":"gloo","caller":"setuputils/main_setup.go:64","msg":"Error while reporting usage","version":"1.4.0-beta13","error":"Encountered error while reporting usage: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup reporting.corp.solo.io on 172.20.0.10:53: read udp 10.31.48.254:54645->172.20.0.10:53: read: connection refused\""}
{"level":"info","ts":1591272183.04849,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:77","msg":"begin sync 15998799953435351923 (1 proxies, 46 upstreams, 26 endpoints, 2 secrets, 7 artifacts, 0 auth configs)","version":"1.4.0-beta13"}
{"level":"info","ts":1591272183.0491004,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-443","version":"1.4.0-beta13"}
{"level":"info","ts":1591272183.0511208,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-80","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272183.0617948,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:133","msg":"Proxy had invalid config","version":"1.4.0-beta13","proxy":{"name":"gateway-proxy","namespace":"gloo-web"},"error":"3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n"}
{"level":"warn","ts":1591272183.0624475,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"proxy gloo-web.gateway-proxy was rejected due to invalid config: 3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n\nAttempting to update only EDS information","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272183.0624804,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"endpoint update failed. xDS snapshot for proxy gloo-web.gateway-proxy will not be updated. Error is: no snapshot found for node gloo-web~gateway-proxy","version":"1.4.0-beta13"}
{"level":"info","ts":1591272183.0635538,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:188","msg":"end sync 15998799953435351923","version":"1.4.0-beta13"}
{"level":"info","ts":1591272185.048447,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:77","msg":"begin sync 1446980669125157496 (1 proxies, 46 upstreams, 28 endpoints, 2 secrets, 7 artifacts, 0 auth configs)","version":"1.4.0-beta13"}
{"level":"info","ts":1591272185.0490432,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-443","version":"1.4.0-beta13"}
{"level":"info","ts":1591272185.0509338,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-80","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272185.0617716,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:133","msg":"Proxy had invalid config","version":"1.4.0-beta13","proxy":{"name":"gateway-proxy","namespace":"gloo-web"},"error":"3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n"}
{"level":"warn","ts":1591272185.0623753,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"proxy gloo-web.gateway-proxy was rejected due to invalid config: 3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n\nAttempting to update only EDS information","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272185.0624065,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"endpoint update failed. xDS snapshot for proxy gloo-web.gateway-proxy will not be updated. Error is: no snapshot found for node gloo-web~gateway-proxy","version":"1.4.0-beta13"}
{"level":"info","ts":1591272185.0633633,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:188","msg":"end sync 1446980669125157496","version":"1.4.0-beta13"}
{"level":"info","ts":1591272210.048459,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:77","msg":"begin sync 16273949872983771330 (1 proxies, 46 upstreams, 28 endpoints, 2 secrets, 7 artifacts, 0 auth configs)","version":"1.4.0-beta13"}
{"level":"info","ts":1591272210.0490901,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-443","version":"1.4.0-beta13"}
{"level":"info","ts":1591272210.051125,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-80","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272210.0619717,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:133","msg":"Proxy had invalid config","version":"1.4.0-beta13","proxy":{"name":"gateway-proxy","namespace":"gloo-web"},"error":"3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n"}
{"level":"warn","ts":1591272210.0625784,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"proxy gloo-web.gateway-proxy was rejected due to invalid config: 3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n\nAttempting to update only EDS information","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272210.0626106,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"endpoint update failed. xDS snapshot for proxy gloo-web.gateway-proxy will not be updated. Error is: no snapshot found for node gloo-web~gateway-proxy","version":"1.4.0-beta13"}
{"level":"info","ts":1591272210.0830503,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:188","msg":"end sync 16273949872983771330","version":"1.4.0-beta13"}
{"level":"info","ts":1591272214.0486517,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:77","msg":"begin sync 1367886256158800126 (1 proxies, 46 upstreams, 32 endpoints, 2 secrets, 7 artifacts, 0 auth configs)","version":"1.4.0-beta13"}
{"level":"info","ts":1591272214.0493631,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-443","version":"1.4.0-beta13"}
{"level":"info","ts":1591272214.0514061,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-80","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272214.0627975,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:133","msg":"Proxy had invalid config","version":"1.4.0-beta13","proxy":{"name":"gateway-proxy","namespace":"gloo-web"},"error":"3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n"}
{"level":"warn","ts":1591272214.0634153,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"proxy gloo-web.gateway-proxy was rejected due to invalid config: 3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n\nAttempting to update only EDS information","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272214.0634475,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"endpoint update failed. xDS snapshot for proxy gloo-web.gateway-proxy will not be updated. Error is: no snapshot found for node gloo-web~gateway-proxy","version":"1.4.0-beta13"}
{"level":"info","ts":1591272214.0834196,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:188","msg":"end sync 1367886256158800126","version":"1.4.0-beta13"}
{"level":"info","ts":1591272219.0486722,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:77","msg":"begin sync 15266886486302123631 (1 proxies, 46 upstreams, 34 endpoints, 2 secrets, 7 artifacts, 0 auth configs)","version":"1.4.0-beta13"}
{"level":"info","ts":1591272219.0493083,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-443","version":"1.4.0-beta13"}
{"level":"info","ts":1591272219.0512125,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-80","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272219.06277,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:133","msg":"Proxy had invalid config","version":"1.4.0-beta13","proxy":{"name":"gateway-proxy","namespace":"gloo-web"},"error":"3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n"}
{"level":"warn","ts":1591272219.0633912,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"proxy gloo-web.gateway-proxy was rejected due to invalid config: 3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n\nAttempting to update only EDS information","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272219.0634263,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"endpoint update failed. xDS snapshot for proxy gloo-web.gateway-proxy will not be updated. Error is: no snapshot found for node gloo-web~gateway-proxy","version":"1.4.0-beta13"}
{"level":"info","ts":1591272219.084079,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:188","msg":"end sync 15266886486302123631","version":"1.4.0-beta13"}
{"level":"info","ts":1591272225.048829,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:77","msg":"begin sync 17302097027792196598 (1 proxies, 46 upstreams, 40 endpoints, 2 secrets, 7 artifacts, 0 auth configs)","version":"1.4.0-beta13"}
{"level":"info","ts":1591272225.0495942,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-443","version":"1.4.0-beta13"}
{"level":"info","ts":1591272225.0516062,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-80","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272225.0641472,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:133","msg":"Proxy had invalid config","version":"1.4.0-beta13","proxy":{"name":"gateway-proxy","namespace":"gloo-web"},"error":"3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n"}
{"level":"warn","ts":1591272225.064825,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"proxy gloo-web.gateway-proxy was rejected due to invalid config: 3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n\nAttempting to update only EDS information","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272225.0648572,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"endpoint update failed. xDS snapshot for proxy gloo-web.gateway-proxy will not be updated. Error is: no snapshot found for node gloo-web~gateway-proxy","version":"1.4.0-beta13"}
{"level":"info","ts":1591272225.085954,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:188","msg":"end sync 17302097027792196598","version":"1.4.0-beta13"}
{"level":"info","ts":1591272227.0489297,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:77","msg":"begin sync 1710484915255174826 (1 proxies, 46 upstreams, 42 endpoints, 2 secrets, 7 artifacts, 0 auth configs)","version":"1.4.0-beta13"}
{"level":"info","ts":1591272227.049663,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-443","version":"1.4.0-beta13"}
{"level":"info","ts":1591272227.0515597,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-80","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272227.064121,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:133","msg":"Proxy had invalid config","version":"1.4.0-beta13","proxy":{"name":"gateway-proxy","namespace":"gloo-web"},"error":"3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n"}
{"level":"warn","ts":1591272227.064765,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"proxy gloo-web.gateway-proxy was rejected due to invalid config: 3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n\nAttempting to update only EDS information","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272227.0647964,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"endpoint update failed. xDS snapshot for proxy gloo-web.gateway-proxy will not be updated. Error is: no snapshot found for node gloo-web~gateway-proxy","version":"1.4.0-beta13"}
{"level":"info","ts":1591272227.084514,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:188","msg":"end sync 1710484915255174826","version":"1.4.0-beta13"}
{"level":"info","ts":1591272234.6481853,"caller":"cache/simple.go:224","msg":"open watch 1 for type.googleapis.com/envoy.api.v2.Cluster[] from nodeID \"gloo-web~gateway-proxy\", version \"\""}
{"level":"info","ts":1591272234.6482437,"caller":"cache/simple.go:224","msg":"open watch 2 for type.googleapis.com/envoy.api.v2.Listener[] from nodeID \"gloo-web~gateway-proxy\", version \"\""}
{"level":"info","ts":1591272237.499601,"logger":"proxy-validator","caller":"validation/server.go:169","msg":"received proxy validation request"}
{"level":"info","ts":1591272237.5010936,"logger":"proxy-validator.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-80"}
{"level":"info","ts":1591272237.5012846,"logger":"proxy-validator.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-443"}
{"level":"info","ts":1591272237.5274065,"logger":"proxy-validator","caller":"validation/server.go:175","msg":"proxy validation report result: listener_reports:<http_listener_report:<> > listener_reports:<http_listener_report:<virtual_host_reports:<route_reports:<> > virtual_host_reports:<route_reports:<> > > > "}
{"level":"info","ts":1591272238.049087,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:77","msg":"begin sync 4597696003959312620 (1 proxies, 46 upstreams, 46 endpoints, 2 secrets, 7 artifacts, 0 auth configs)","version":"1.4.0-beta13"}
{"level":"info","ts":1591272238.050004,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-443","version":"1.4.0-beta13"}
{"level":"info","ts":1591272238.052585,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-80","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272238.0666082,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:133","msg":"Proxy had invalid config","version":"1.4.0-beta13","proxy":{"name":"gateway-proxy","namespace":"gloo-web"},"error":"3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n"}
{"level":"warn","ts":1591272238.0674589,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"proxy gloo-web.gateway-proxy was rejected due to invalid config: 3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n\nAttempting to update only EDS information","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272238.0674918,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"endpoint update failed. xDS snapshot for proxy gloo-web.gateway-proxy will not be updated. Error is: no snapshot found for node gloo-web~gateway-proxy","version":"1.4.0-beta13"}
{"level":"info","ts":1591272238.089077,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:188","msg":"end sync 4597696003959312620","version":"1.4.0-beta13"}
{"level":"info","ts":1591272239.0489657,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:77","msg":"begin sync 4043792216824835196 (1 proxies, 46 upstreams, 48 endpoints, 2 secrets, 7 artifacts, 0 auth configs)","version":"1.4.0-beta13"}
{"level":"info","ts":1591272239.0496917,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-443","version":"1.4.0-beta13"}
{"level":"info","ts":1591272239.051524,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-80","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272239.0645075,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:133","msg":"Proxy had invalid config","version":"1.4.0-beta13","proxy":{"name":"gateway-proxy","namespace":"gloo-web"},"error":"3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n"}
{"level":"warn","ts":1591272239.0651531,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"proxy gloo-web.gateway-proxy was rejected due to invalid config: 3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n\nAttempting to update only EDS information","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272239.0651846,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"endpoint update failed. xDS snapshot for proxy gloo-web.gateway-proxy will not be updated. Error is: no snapshot found for node gloo-web~gateway-proxy","version":"1.4.0-beta13"}
{"level":"info","ts":1591272239.0661485,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:188","msg":"end sync 4043792216824835196","version":"1.4.0-beta13"}
{"level":"info","ts":1591272381.0491786,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:77","msg":"begin sync 2860259612625154686 (1 proxies, 46 upstreams, 50 endpoints, 2 secrets, 7 artifacts, 0 auth configs)","version":"1.4.0-beta13"}
{"level":"info","ts":1591272381.050111,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-443","version":"1.4.0-beta13"}
{"level":"info","ts":1591272381.0549543,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:114","msg":"computing envoy resources for listener: listener-::-80","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272381.0913084,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:133","msg":"Proxy had invalid config","version":"1.4.0-beta13","proxy":{"name":"gateway-proxy","namespace":"gloo-web"},"error":"3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n"}
{"level":"warn","ts":1591272381.0924764,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:142","msg":"proxy gloo-web.gateway-proxy was rejected due to invalid config: 3 errors occurred:\n\t* invalid resource apps.podinfo\n\t* destination # 1: upstream not found: list did not find upstream gloo-admin.apps-podinfo-primary-9898\n\t* destination # 2: upstream not found: list did not find upstream gloo-admin.apps-podinfo-canary-9898\n\n\nAttempting to update only EDS information","version":"1.4.0-beta13"}
{"level":"warn","ts":1591272381.0925207,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:149","msg":"endpoint update failed. xDS snapshot for proxy gloo-web.gateway-proxy will not be updated. Error is: no snapshot found for node gloo-web~gateway-proxy","version":"1.4.0-beta13"}
{"level":"info","ts":1591272381.1138237,"logger":"gloo.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:188","msg":"end sync 2860259612625154686","version":"1.4.0-beta13"}
GrahamGoudeau commented 4 years ago

Hey @cweidinger, this actually looks to be the intended behavior. My guess is that you have an UpstreamGroup in the apps namespace that both Gloos are watching, and that UpstreamGroup refers explicitly to an upstream in the gloo-admin namespace. This causes the gloo-web Gloo to fail to process the UpstreamGroup.

Separately I've filed #3128 because investigating this has exposed the possibility of a race condition when two Gloos manage the same UpstreamGroup.

cweidinger commented 4 years ago

@GrahamGoudeau Hey thanks for explaining. From the looks of our virtualservice that we're defining, I had no idea that gloo's upstreamgroup and upstream would cause that problem.

Name:         podinfo
Namespace:    gloo-admin
Labels:       argocd.argoproj.io/instance=flagger-canary-example
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"gateway.solo.io/v1","kind":"VirtualService","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"flagger-c...
API Version:  gateway.solo.io/v1
Kind:         VirtualService
Metadata:
  Creation Timestamp:  2020-06-03T19:04:32Z
  Generation:          11
  Resource Version:    808915
  Self Link:           /apis/gateway.solo.io/v1/namespaces/gloo-admin/virtualservices/podinfo
  UID:                 0f151269-a5cd-11ea-a57f-0aa06e344997
Spec:
  Ssl Config:
    Secret Ref:
      Name:       gateway-tls
      Namespace:  gloo-admin
  Virtual Host:
    Domains:
      podinfo.admin.ue1.master.sharedservices.development.aws.carfax.io
    Routes:
      Matchers:
        Prefix:  /
      Route Action:
        Upstream Group:
          Name:       podinfo
          Namespace:  apps
Status:
  reported_by:  gateway
  State:        1
  subresource_statuses:
    *v1.Proxy.gloo-admin.gateway-proxy:
      reported_by:  gloo
      State:        1
Events:             <none>

My expectation remains that a misconfiguration like this would register an error and prevent the gloo-web from processing the odd UpstreamGroup (from it's perspective) but that other VirtualServices in gloo-web would continue to serve traffic and further VirtualServices added to gloo-web would start serving traffic. The biggest apprehension our team has about continuing to try to use gloo for our production systems is this pattern we've seen a few times where if one configuration item (virtualservice in this case) contains some error that gloo stops serving traffic for things that are properly configured.

kdorosh commented 4 years ago

My expectation remains that a misconfiguration like this would register an error and prevent the gloo-web from processing the odd UpstreamGroup (from it's perspective) but that other VirtualServices in gloo-web would continue to serve traffic and further VirtualServices added to gloo-web would start serving traffic

This is also my expectation for how Gloo functions (except that the error status may get lost on resources managed by two Gloos, due to the bug @GrahamGoudeau filed -- but this should be status-only which is only read by people). Can you reproduce instances where an invalid virtual service added invalidates routes that previously existed? Further, for misconfigurations within a virtual service, or adding virtual services after an invalid one has been written to storage, have you considered enabling route replacement?

To prevent invalid resources from being written to storage, I would also recommend setting Gloo validation's alwaysAccept setting to false.

cweidinger commented 4 years ago

We did enable route replacement before we got this bug. We'll try alwaysAccept. Thanks for the tip

github-actions[bot] commented 9 months ago

This issue has been marked as stale because of no activity in the last 180 days. It will be closed in the next 180 days unless it is tagged "no stalebot" or other activity occurs.

github-actions[bot] commented 3 months ago

This issue has been closed due to no activity in the last 12 months.