search

kumahq / kuma

🐻 The multi-zone service mesh for containers, Kubernetes and VMs. Built with Envoy. CNCF Sandbox Project.
https://kuma.io/install
Apache License 2.0
3.61k stars 332 forks source link

Duplicated listeners after federating a zone #9006

Closed slonka closed 6 months ago

slonka commented 8 months ago

What happened?

got:

 2024-01-23T09:59:44.403Z    INFO    xds.nack-backoff    config was previously rejected by Envoy. Applying backoff before resending it    {"backoff": "5s", "nodeID": "default.demo-app-c7cd6588b-wxn6t.kuma-demo", "reason": "Error adding/updating listener(s) outbound:2 │
│ 40.0.0.1:80: error adding listener: 'outbound:240.0.0.1:80' has duplicate address '240.0.0.1:80' as existing listener\noutbound:240.0.0.0:80: error adding listener: 'outbound:240.0.0.0:80' has duplicate address '240.0.0.0:80' as existing listener\n"}                 │

after federating a zone with demo installed, @bartsmykla mentioned that this could be related to https://github.com/kumahq/kuma/issues/6717 - that's why I assigned @jijiechen

Steps to reproduce:

  1. deploy a zone
  2. install demo
  3. deploy a global
  4. export resources using kumactl export
  5. federate zone (kumactl install with kds global address)
  6. check logs
jijiechen commented 8 months ago

Reproduced once, but not too much useful information captured.

This issue is a transitional state during the importing step of the federation:

complete data plane logs ``` 2024-01-25T08:30:22.010Z INFO config skipping reading config from file 2024-01-25T08:30:22.010Z INFO kuma-dp.run effective configuration {"config": "{\"controlPlane\":{\"caCert\":\"-----BEGIN CERTIFICATE-----\\nMIIDEDCCAfigAwIBAgIRAKwURu51rSVmOkAAM9tHfqQwDQYJKoZIhvcNAQELBQAw\\nEjEQMA4GA1UEAxMHa3VtYS1jYTAeFw0yNDAxMjUwODI4MzdaFw0zNDAxMjIwODI4\\nMzdaMBIxEDAOBgNVBAMTB2t1bWEtY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw\\nggEKAoIBAQDApuKyYXlcNrrzIMzamWnQahJxYyycQlw43Z8erkmi1kUqqkEWQzfY\\nAmQdHMMk0YUZacCFLcjMyY5VqOkn1zUWJchSNgTYwEIkLyN7HjobwdQ1V4B6SY0E\\n9geGr5pJLoLcULuV5jgAqlgQU0pEDvT6SZ3DaaCS+YxBgZpa1WKujLVMsvz/rIbC\\nVP5UxTd0Pmmfa+h9JCjNzdZJ2FJNPfqwgpIGtiuIaJG0sX/mQgfCnthKHAFxGLf3\\n20awgs7uP+VtW7yG6+Wtzu1ydr+vz9BXtEuL0oBSjH9ytlZ/1hj9qbGSh9xv7681\\nFo7+dkcIc7o3zH8S7rDRLBQKBNXS21zrAgMBAAGjYTBfMA4GA1UdDwEB/wQEAwIC\\npDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDwYDVR0TAQH/BAUwAwEB\\n/zAdBgNVHQ4EFgQUftDh2kgss6k5SsXZzgFnseod48YwDQYJKoZIhvcNAQELBQAD\\nggEBAKYjBUeg7Jng8Q+h0kw2zOcu5IBM7+0F9u49qkcYq0yCQYewCHrWbQwf4KMD\\nlGc4qnsnhI9yvoLe6p4k47QdxP+TUo7vB1DibW1anPQwn30a6IWZNMw5rFWpz9gM\\n+3iiUAB/aywRUVBk4tiu2UxsGbZBBhS8TfayYVcFpcrFHhEMKRk1ufJs53xLxgrg\\nbWm2kikWRzIf3LrXQmWxIysHYunzqmeS5b2Hv3oIZHe8XLJUc7GbpByAB9agRQIU\\nEfoSv5UERYqHSp44Rhi4kSxasUEQXxUQBS0UXH+472ASKfZqU9MEtnn8aM1qP/KN\\ni+9GQvrbEx7TRk3DuS78l8tdwP4=\\n-----END CERTIFICATE-----\\n\",\"caCertFile\":\"\",\"retry\":{\"backoff\":\"3s\",\"maxDuration\":\"5m0s\"},\"url\":\"https://kong-mesh-control-plane.kong-mesh-system:5678\"},\"dataplane\":{\"drainTime\":\"30s\",\"mesh\":\"default\",\"name\":\"demo-app-d789d49c-j86js.kuma-demo\",\"proxyType\":\"dataplane\"},\"dataplaneRuntime\":{\"binaryPath\":\"envoy\",\"concurrency\":2,\"dataplaneTokenPath\":\"/var/run/secrets/kubernetes.io/serviceaccount/token\",\"metrics\":{},\"resources\":{}},\"dns\":{\"coreDnsBinaryPath\":\"coredns\",\"coreDnsEmptyPort\":15054,\"coreDnsPort\":15053,\"enabled\":true,\"envoyDnsPort\":15055,\"prometheusPort\":19153}}"} 2024-01-25T08:30:22.010Z INFO kuma-dp.run generated configurations will be stored in a temporary directory {"dir": "/tmp/kuma-dp-3804747063"} 2024-01-25T08:30:22.010Z INFO config skipping reading config from file 2024-01-25T08:30:22.010Z INFO effective Kong Mesh DP configuration {"config": "{\"auth\":{\"type\":\"\"},\"opa\":{\"addr\":\"localhost:8181\",\"configOverrides\":[],\"configPath\":\"\",\"diagnosticAddr\":\"0.0.0.0:8282\",\"enabled\":true,\"extAuthzAddr\":\"localhost:9191\"}}"} 2024-01-25T08:30:22.034Z INFO kuma-dp.run fetched Envoy version {"version": {"Build":"b5ca88acee3453c9459474b8f22215796eff4dde/1.28.0/Modified/RELEASE/BoringSSL","Version":"1.28.0","KumaDpCompatible":true}} 2024-01-25T08:30:22.034Z INFO kuma-dp.run generating bootstrap configuration 2024-01-25T08:30:22.034Z INFO dataplane trying to fetch bootstrap configuration from the Control Plane 2024-01-25T08:30:22.049Z INFO kuma-dp.run received bootstrap configuration {"adminPort": 9901} 2024-01-25T08:30:22.051Z INFO kuma-dp.run starting Kuma DP {"version": "2.5.1"} 2024-01-25T08:30:22.051Z INFO kuma-dp.run.access-log-streamer starting resilient component ... 2024-01-25T08:30:22.051Z INFO access-log-streamer cleaning existing access log pipe {"file": "/tmp/kuma-dp-3804747063/kuma-al-demo-app-d789d49c-j86js.kuma-demo-default.sock"} 2024-01-25T08:30:22.051Z INFO access-log-streamer creating access log pipe {"file": "/tmp/kuma-dp-3804747063/kuma-al-demo-app-d789d49c-j86js.kuma-demo-default.sock"} 2024-01-25T08:30:22.051Z INFO starting resilient component ... 2024-01-25T08:30:22.051Z INFO opa starting OPADS stream to receive OPA Config 2024-01-25T08:30:22.051Z INFO kuma-dp.run.envoy bootstrap configuration saved to a file {"file": "/tmp/kuma-dp-3804747063/bootstrap.yaml"} 2024-01-25T08:30:22.051Z INFO metrics-hijacker starting Metrics Hijacker Server {"socketPath": "unix:///tmp/kuma-dp-3804747063/kuma-mh-demo-app-d789d49c-j86js.kuma-demo-default.sock"} 2024-01-25T08:30:22.051Z INFO kuma-dp.run.envoy starting Envoy {"path": "/usr/bin/envoy", "arguments": ["--config-path", "/tmp/kuma-dp-3804747063/bootstrap.yaml", "--drain-time-s", "30", "--disable-hot-restart", "--log-level", "info", "--concurrency", "2"]} 2024-01-25T08:30:22.051Z INFO kuma-dp.run.dns-server configuration saved to a file {"file": "/tmp/kuma-dp-3804747063/Corefile"} 2024-01-25T08:30:22.051Z INFO kuma-dp.run.dns-server starting DNS Server (coredns) {"args": ["-conf", "/tmp/kuma-dp-3804747063/Corefile", "-quiet"]} 2024-01-25T08:30:22.051Z INFO opa waiting for OPA Config [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:413] initializing epoch 0 (base id=0, hot restart version=disabled) [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:415] statically linked extensions: [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.geoip_providers: envoy.geoip_providers.maxmind [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.matching.input_matchers: envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip, envoy.matching.matchers.runtime_fraction [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.tracers: envoy.dynamic.ot, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.retry_priorities: envoy.retry_priorities.previous_priorities [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.connection_handler: envoy.connection_handler.default [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.http.header_validators: envoy.http.header_validators.envoy_default [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8 [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.quic.server_preferred_address: quic.server_preferred_address.fixed [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.regex_engines: envoy.regex_engines.google_re2 [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.health_checkers: envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.common.key_value: envoy.key_value.file_based [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.matching.action: envoy.matching.actions.format_string, filter-chain-name [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.quic.connection_id_generator: envoy.quic.deterministic_connection_id_generator [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.route.early_data_policy: envoy.route.early_data_policy.default [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.formatter: envoy.formatter.cel, envoy.formatter.metadata, envoy.formatter.req_without_query [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.health_check.event_sinks: envoy.health_check.event_sink.file [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.trie_matcher [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.dubbo_proxy.serializers: dubbo.hessian2 [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.header [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] network.connection.client: default, envoy_internal [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.http_capsule [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.upstreams: envoy.filters.connection_pools.tcp.generic [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.network.application_protocols, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.load_balancing_policies: envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.filters.http.upstream: envoy.buffer, envoy.filters.http.admission_control, envoy.filters.http.buffer, envoy.filters.http.header_mutation, envoy.filters.http.upstream_codec [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.thrift_proxy.transports: auto, framed, header, unframed [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.path.match: envoy.path.match.uri_template.uri_template_matcher [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.resolvers: envoy.ip [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] quic.http_server_connection: quic.http_server_connection.default [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.dubbo_proxy.protocols: dubbo [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.request_id: envoy.request_id.uuid [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe [2024-01-25 08:30:22.066][26][info][main] [source/server/server.cc:417] envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy [2024-01-25 08:30:22.069][26][info][main] [source/server/server.cc:471] HTTP header map info: [2024-01-25 08:30:22.072][26][info][main] [source/server/server.cc:474] request header map: 680 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-ot-span-context,x-request-id [2024-01-25 08:30:22.072][26][info][main] [source/server/server.cc:474] request trailer map: 128 bytes: [2024-01-25 08:30:22.072][26][info][main] [source/server/server.cc:474] response header map: 440 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id [2024-01-25 08:30:22.072][26][info][main] [source/server/server.cc:474] response trailer map: 152 bytes: grpc-message,grpc-status [2024-01-25 08:30:22.078][26][info][main] [source/server/server.cc:837] runtime: layers: - name: kuma static_layer: re2.max_program_size.warn_level: 1000 re2.max_program_size.error_level: 4294967295 [2024-01-25 08:30:22.079][26][info][admin] [source/server/admin/admin.cc:68] admin address: 127.0.0.1:9901 [2024-01-25 08:30:22.079][26][info][config] [source/server/configuration_impl.cc:159] loading tracing configuration [2024-01-25 08:30:22.079][26][info][config] [source/server/configuration_impl.cc:118] loading 1 static secret(s) [2024-01-25 08:30:22.079][26][info][config] [source/server/configuration_impl.cc:124] loading 2 cluster(s) [2024-01-25 08:30:22.088][26][info][config] [source/server/configuration_impl.cc:128] loading 0 listener(s) [2024-01-25 08:30:22.088][26][info][config] [source/server/configuration_impl.cc:145] loading stats configuration [2024-01-25 08:30:22.089][26][warning][main] [source/server/server.cc:901] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.downstream_connections` resource monitor. [2024-01-25 08:30:22.089][26][info][main] [source/server/server.cc:942] starting main dispatch loop [2024-01-25 08:30:22.090][26][info][runtime] [source/common/runtime/runtime_impl.cc:579] RTDS has finished initialization [2024-01-25 08:30:22.090][26][info][upstream] [source/common/upstream/cluster_manager_impl.cc:222] cm init: initializing cds 2024-01-25T08:30:23.062Z INFO opa received empty OPA Config. OPA is not yet configured for this proxy. Sending ACK to the control plane 2024-01-25T08:30:23.062Z INFO opa waiting for OPA Config [2024-01-25 08:30:23.674][26][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 8 cluster(s), remove 2 cluster(s) [2024-01-25 08:30:23.713][26][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 8 cluster(s), skipped 0 unmodified cluster(s) [2024-01-25 08:30:23.713][26][info][upstream] [source/common/upstream/cluster_manager_impl.cc:200] cm init: initializing secondary clusters [2024-01-25 08:30:23.715][26][info][upstream] [source/common/upstream/cluster_manager_impl.cc:226] cm init: all clusters initialized [2024-01-25 08:30:23.715][26][info][main] [source/server/server.cc:923] all clusters initialized. initializing init manager [2024-01-25 08:30:23.721][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'inbound:10.42.0.14:5000' [2024-01-25 08:30:23.722][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'inbound:passthrough:ipv6' [2024-01-25 08:30:23.723][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'kuma:dns' [2024-01-25 08:30:23.724][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.23.134:5000' [2024-01-25 08:30:23.724][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:240.0.0.0:80' [2024-01-25 08:30:23.725][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:passthrough:ipv6' [2024-01-25 08:30:23.729][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'probe:listener' [2024-01-25 08:30:23.729][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'inbound:passthrough:ipv4' [2024-01-25 08:30:23.738][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'kuma:envoy:admin' [2024-01-25 08:30:23.739][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.244.77:6379' [2024-01-25 08:30:23.739][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:240.0.0.1:80' [2024-01-25 08:30:23.740][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:passthrough:ipv4' [2024-01-25 08:30:23.740][26][info][config] [source/extensions/listener_managers/listener_manager/listener_manager_impl.cc:923] all dependencies initialized. starting workers [2024-01-25 08:30:34.587][26][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 8 cluster(s), remove 2 cluster(s) [2024-01-25 08:30:34.607][26][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 1 cluster(s), skipped 7 unmodified cluster(s) [2024-01-25 08:30:34.613][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.23.134:5000' [2024-01-25 08:30:34.619][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:240.0.0.0:80' [2024-01-25 08:37:52.213][26][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:152] StreamAggregatedResources gRPC config stream to ads_cluster closed: 13, 2024-01-25T08:37:52.216Z INFO opa closing OPADS stream 2024-01-25T08:37:52.216Z ERROR component terminated with an error {"generationID": 1, "error": "could not receive OPA Config: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR", "errorVerbose": "rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR\ncould not receive OPA Config\ngithub.com/Kong/kong-mesh/app/kuma-dp/pkg/opa.(*opaConfigurer).Start\n\tgithub.com/Kong/kong-mesh/app/kuma-dp/pkg/opa/configurer.go:65\ngithub.com/kumahq/kuma/pkg/core/runtime/component.(*resilientComponent).Start.func1\n\tgithub.com/kumahq/kuma@v0.0.0-20231205160531-d2ced55cd241/pkg/core/runtime/component/resilient.go:43\nruntime.goexit\n\truntime/asm_arm64.s:1197"} [2024-01-25 08:37:53.744][26][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 8 cluster(s), remove 2 cluster(s) [2024-01-25 08:37:53.766][26][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 5 cluster(s), skipped 3 unmodified cluster(s) [2024-01-25 08:37:53.767][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:63] lds: remove listener 'outbound:240.0.0.0:80' [2024-01-25 08:37:53.767][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:63] lds: remove listener 'outbound:240.0.0.1:80' [2024-01-25 08:37:53.768][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'kuma:dns' [2024-01-25 08:37:53.775][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.23.134:5000' [2024-01-25 08:37:53.780][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'kuma:envoy:admin' [2024-01-25 08:37:53.781][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.244.77:6379' 2024-01-25T08:37:57.219Z INFO opa starting OPADS stream to receive OPA Config 2024-01-25T08:37:57.236Z INFO opa waiting for OPA Config 2024-01-25T08:37:58.239Z INFO opa received empty OPA Config. OPA is not yet configured for this proxy. Sending ACK to the control plane 2024-01-25T08:37:58.239Z INFO opa waiting for OPA Config [2024-01-25 08:38:09.721][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'inbound:10.42.0.14:5000' [2024-01-25 08:38:09.730][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.23.134:5000' [2024-01-25 08:43:12.103][26][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:152] StreamAggregatedResources gRPC config stream to ads_cluster closed: 13, 2024-01-25T08:43:12.107Z INFO opa closing OPADS stream 2024-01-25T08:43:12.109Z ERROR component terminated with an error {"generationID": 2, "error": "could not receive OPA Config: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR", "errorVerbose": "rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR\ncould not receive OPA Config\ngithub.com/Kong/kong-mesh/app/kuma-dp/pkg/opa.(*opaConfigurer).Start\n\tgithub.com/Kong/kong-mesh/app/kuma-dp/pkg/opa/configurer.go:65\ngithub.com/kumahq/kuma/pkg/core/runtime/component.(*resilientComponent).Start.func1\n\tgithub.com/kumahq/kuma@v0.0.0-20231205160531-d2ced55cd241/pkg/core/runtime/component/resilient.go:43\nruntime.goexit\n\truntime/asm_arm64.s:1197"} [2024-01-25 08:43:13.323][26][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 8 cluster(s), remove 2 cluster(s) [2024-01-25 08:43:13.323][26][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 0 cluster(s), skipped 8 unmodified cluster(s) [2024-01-25 08:43:13.330][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'kuma:envoy:admin' 2024-01-25T08:43:17.112Z INFO opa starting OPADS stream to receive OPA Config 2024-01-25T08:43:17.129Z INFO opa waiting for OPA Config 2024-01-25T08:43:18.132Z INFO opa received empty OPA Config. OPA is not yet configured for this proxy. Sending ACK to the control plane 2024-01-25T08:43:18.132Z INFO opa waiting for OPA Config [2024-01-25 08:43:31.174][26][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 8 cluster(s), remove 2 cluster(s) [2024-01-25 08:43:31.188][26][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 3 cluster(s), skipped 5 unmodified cluster(s) [2024-01-25 08:43:31.194][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.23.134:5000' [2024-01-25 08:43:31.200][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:240.0.0.0:80' [2024-01-25 08:43:31.201][26][warning][config] [source/extensions/listener_managers/listener_manager/listener_manager_impl.cc:1113] error adding listener: 'outbound:240.0.0.1:80' has duplicate address '240.0.0.1:80' as existing listener [2024-01-25 08:43:31.205][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'inbound:10.42.0.14:5000' [2024-01-25 08:43:31.207][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.244.77:6379' [2024-01-25 08:43:31.207][26][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138] gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) outbound:240.0.0.1:80: error adding listener: 'outbound:240.0.0.1:80' has duplicate address '240.0.0.1:80' as existing listener [2024-01-25 08:43:36.214][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:240.0.0.1:80' [2024-01-25 08:43:36.214][26][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 8 cluster(s), remove 2 cluster(s) [2024-01-25 08:43:36.229][26][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 2 cluster(s), skipped 6 unmodified cluster(s) [2024-01-25 08:45:53.162][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:63] lds: remove listener 'outbound:240.0.0.1:80' [2024-01-25 08:45:53.162][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:63] lds: remove listener 'outbound:240.0.0.0:80' [2024-01-25 08:45:53.169][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.23.134:5000' [2024-01-25 08:45:53.171][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.244.77:6379' [2024-01-25 08:45:54.185][26][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:99] lds: add/update listener 'outbound:10.43.23.134:5000' ```
jijiechen commented 8 months ago

There are no unusual logs generated in any control planes:

All of logs generated from above pods are regular entries.

jakubdyszkiewicz commented 7 months ago

Triage: we could double-check this with upgrade 2.5.0 to 2.6.0 without any federation. Increase the number of listeners (by adding more services) to get a better chance of hitting this issue.

ttreptow commented 7 months ago

In my case, the problem does not go away. Some things I've noticed:

Regarding that last point, perhaps simply filtering out duplicate configs before sending them could be a quick fix?

lahabana commented 6 months ago

@ttreptow does this still happen in 2.6.1 ? Seems like there were multiple things at play at the same time here.

ttreptow commented 6 months ago

I didn't see the issue this time when I upgraded from 2.5.2 to 2.6.1

ttreptow commented 6 months ago

I spoke too soon actually, I do still see the issue

jijiechen commented 6 months ago

@ttreptow Thanks for reporting.

Just wanted to confirm a few items. Do you have any headless serviecs in your mesh? On the services of the duplicated listeners, are there any properties different or interesting? Does this problem cause any actual traffic issue in your mesh? We are seeing this occurring on headless services, and this will be fixed in a coming patch release.

If headless services are not in your case and you don't mind, could you send us a copy of your dumpped envoy config? (Please remove any sensetive information like secrets, PII, etc). You may post it on this thread if you'd like, or you can also send it via email to team-mesh@konghq.com. Thank you very much.