search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
24.74k stars 4.75k forks source link

Envoy 1.27: Datadog operation changed from `envoy.proxy` to `ingress` and resource name unset #30235

Closed hiddewie closed 8 months ago

hiddewie commented 11 months ago

Title

Envoy 1.27: Datadog operation changed from envoy.proxy to ingress and resource name unset

Description

Since Envoy version 1.27, the Datadog traces are emitted with a different operation ID: ingress instead of envoy.proxy. In addition, the resource names set by the tracing tag resource.name is no longer propagated to Datadog.

Seems related to the discussion in envoyproxy/envoy#21083, the PR envoyproxy/envoy#26284 which replaced the Datadog tracing implementation with another library.

Envoy versions 1.26.3 or earlier are fine, and produce the correct traces with the correct operation IDs, resources and tags.

Observed behaviour: The Datadog operation name is ingress, and the resource name is ingress

Expected behaviour: The Datadog operation name is envoy.proxy, and the resource name is as configured by the tracing tag resource.name

The release notes of 1.27 contain no deprecations, nor configuration updates that are needed to preserve the 1.26.x behaviour: https://www.envoyproxy.io/docs/envoy/latest/version_history/v1.27/v1.27.0.

Repro steps

Nomad job (contains the Envoy configuration)

job "envoy-test" {
  datacenters = ["*"]
  type = "service"

  group "envoy-test" {
    network {
      mode = "bridge"
    }

    task "envoy-test" {
      driver = "docker"
      kill_timeout = "30s"

      config {
        image = "envoyproxy/envoy:v1.26-latest"
        volumes = [
          "/var/run/datadog-agent:/var/run/datadog-agent",
          "local/envoy.yaml:/etc/envoy/envoy.yaml",
        ]
      }

      env {
        TZ = "Europe/Amsterdam"
        DD_SERVICE = "envoy-test"
        DD_ENV = "development"
        DD_VERSION = "envoy-test"
        DD_TRACE_STARTUP_LOGS = false
      }

      template {
        data = <<-EOF
node:
  id: 'test-envoy'
  cluster: 'cl-1'

admin:
  access_log:
    name: envoy.access_loggers.file
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
      path: "/dev/stdout"
  address:
    socket_address:
      address: '::'
      port_value: 5000
      ipv4_compat: true

static_resources:
  listeners:
    - name: 'listener'
      address:
        socket_address:
          address: '::'
          port_value: 4000
          ipv4_compat: true
      filter_chains:
      - filters:
        - name: envoy.http_connection_manager
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            stat_prefix: 'test-envoy'
            tracing:
              custom_tags:
                - tag: service
                  literal:
                    value: 'test-envoy'
                - tag: env
                  literal:
                    value: 'development'
                - tag: version
                  literal:
                    value: 'test-envoy'
                - tag: resource.name
                  literal:
                    value: 'test-resource'
              provider:
                name: envoy.tracers.datadog
                typed_config:
                  "@type": type.googleapis.com/envoy.config.trace.v3.DatadogConfig
                  collector_cluster: datadog_agent
                  service_name: 'test-envoy'

            route_config:
              name: route_configuration
              virtual_hosts:
                - name: backend
                  domains: [ "*" ]

                  routes:
                    - match:
                        path: "/status"
                      direct_response:
                        status: 200
                        body:
                          inline_string: OK

                    - match:
                        prefix: ''
                      direct_response:
                        status: 503
                        body:
                          inline_string: 'Service Not Available'

            http_filters:
              - name: envoy.router
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                  suppress_envoy_headers: true

  clusters:
    - name: datadog_agent
      connect_timeout: 1s
      type: static
      load_assignment:
        cluster_name: datadog_agent
        endpoints:
          lb_endpoints:
            endpoint:
              address:
                pipe:
                  path: /var/run/datadog-agent/apm.sock
        EOF

        destination = "local/envoy.yaml"
      }

      resources {
        cpu = 100
        memory = 1000
      }
    }
  }
}

In the container:

apt update
apt install curl
curl localhost:4000/status
curl localhost:4000/doesnt-exist

With container envoyproxy/envoy:v1.26-latest:

image

With container envoyproxy/envoy:v1.27-latest:

image

Admin and Stats Output

See attached files:

stats.txt server_info.txt routes.txt clusters.txt

Config

See content of Nomad job

Logs

Access logs for 1.27

[2023-10-16T11:18:37.788Z] "GET / HTTP/1.1" 200 - 0 18578 0 - "172.26.76.38" "curl/7.68.0" "-" "localhost:5000" "-"
[2023-10-16T11:18:40.452Z] "GET /stats HTTP/1.1" 200 - 0 19772 0 - "172.26.76.38" "curl/7.68.0" "-" "localhost:5000" "-"
[2023-10-16T11:20:19.989Z] "GET /stats HTTP/1.1" 200 - 0 20244 0 - "172.26.76.38" "curl/7.68.0" "-" "localhost:5000" "-"
[2023-10-16T11:20:30.485Z] "GET /clusters HTTP/1.1" 200 - 0 1619 0 - "172.26.76.38" "curl/7.68.0" "-" "localhost:5000" "-"
[2023-10-16T11:20:41.518Z] "GET /routes HTTP/1.1" 404 - 0 4079 0 - "172.26.76.38" "curl/7.68.0" "-" "localhost:5000" "-"
[2023-10-16T11:20:46.660Z] "GET /server_info HTTP/1.1" 200 - 0 75569 2 - "172.26.76.38" "curl/7.68.0" "-" "localhost:5000" "-"
[2023-10-16T11:24:51.221Z] "GET /routes HTTP/1.1" 404 - 0 4079 0 - "172.26.76.38" "curl/7.68.0" "-" "localhost:5000" "-"

[2023-10-16 11:15:55.565][1][info][main] [source/server/server.cc:413] initializing epoch 0 (base id=0, hot restart version=11.104)
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:415] statically linked extensions:
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.formatter: envoy.formatter.cel, envoy.formatter.metadata, envoy.formatter.req_without_query
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.route.early_data_policy: envoy.route.early_data_policy.default
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.matching.action: envoy.matching.actions.format_string, filter-chain-name
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.http.header_validators: envoy.http.header_validators.envoy_default
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   quic.http_server_connection: quic.http_server_connection.default
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.load_balancing_policies: envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   network.connection.client: default, envoy_internal
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.matching.input_matchers: envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip, envoy.matching.matchers.runtime_fraction
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.connection_handler: envoy.connection_handler.default
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.resolvers: envoy.ip
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.dubbo_proxy.protocols: dubbo
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.request_id: envoy.request_id.uuid
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.tracers: envoy.dynamic.ot, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.health_check.event_sinks: envoy.health_check.event_sink.file
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.header
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.upstreams: envoy.filters.connection_pools.tcp.generic
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.health_checkers: envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.common.key_value: envoy.key_value.file_based
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.quic.connection_id_generator: envoy.quic.deterministic_connection_id_generator
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.regex_engines: envoy.regex_engines.google_re2
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.path.match: envoy.path.match.uri_template.uri_template_matcher
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.quic.server_preferred_address: quic.server_preferred_address.fixed
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.filters.http.upstream: envoy.buffer, envoy.filters.http.admission_control, envoy.filters.http.buffer, envoy.filters.http.header_mutation, envoy.filters.http.upstream_codec
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2023-10-16 11:15:55.566][1][info][main] [source/server/server.cc:417]   envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso
[2023-10-16 11:15:55.570][1][info][main] [source/server/server.cc:470] HTTP header map info:
[2023-10-16 11:15:55.572][1][info][main] [source/server/server.cc:473]   request header map: 680 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-ot-span-context,x-request-id
[2023-10-16 11:15:55.572][1][info][main] [source/server/server.cc:473]   request trailer map: 128 bytes: 
[2023-10-16 11:15:55.572][1][info][main] [source/server/server.cc:473]   response header map: 440 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2023-10-16 11:15:55.572][1][info][main] [source/server/server.cc:473]   response trailer map: 152 bytes: grpc-message,grpc-status
[2023-10-16 11:15:55.579][1][info][main] [source/server/server.cc:840] runtime: {}
[2023-10-16 11:15:55.579][1][info][admin] [source/server/admin/admin.cc:66] admin address: [::]:5000
[2023-10-16 11:15:55.580][1][info][config] [source/server/configuration_impl.cc:144] loading tracing configuration
[2023-10-16 11:15:55.580][1][info][config] [source/server/configuration_impl.cc:103] loading 0 static secret(s)
[2023-10-16 11:15:55.580][1][info][config] [source/server/configuration_impl.cc:109] loading 1 cluster(s)
[2023-10-16 11:15:55.582][1][info][config] [source/server/configuration_impl.cc:113] loading 1 listener(s)
[2023-10-16 11:15:55.583][1][info][tracing] [source/common/tracing/tracer_manager_impl.cc:42] instantiating a new tracer: envoy.tracers.datadog
[2023-10-16 11:15:55.584][1][info][config] [source/server/configuration_impl.cc:130] loading stats configuration
[2023-10-16 11:15:55.584][1][info][runtime] [source/common/runtime/runtime_impl.cc:577] RTDS has finished initialization
[2023-10-16 11:15:55.584][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:226] cm init: all clusters initialized
[2023-10-16 11:15:55.584][1][warning][main] [source/server/server.cc:814] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
[2023-10-16 11:15:55.585][1][info][main] [source/server/server.cc:918] all clusters initialized. initializing init manager
[2023-10-16 11:15:55.585][1][info][config] [source/extensions/listener_managers/listener_manager/listener_manager_impl.cc:858] all dependencies initialized. starting workers
[2023-10-16 11:15:55.587][1][info][main] [source/server/server.cc:937] starting main dispatch loop
kyessenov commented 11 months ago

CC @cgilmour

cgilmour commented 11 months ago

A fix for this was made by my colleague @dgoffredo in #29932, and looks like it has been backported as well.

hiddewie commented 11 months ago

Thank you for the quick answer, I was not aware of that pull request (somehow it evaded my Github searches). The PR seems to cover my issue, I will wait for the release including the fix. This issue can be closed.

cgilmour commented 11 months ago

A release including the fix shipped very recently (literally - within the last few hours). It's included in both the 1.27.x and 1.26.x releases.

hiddewie commented 11 months ago

I can confirm that Envoy 1.27.2 changes the operation name back from ingress to envoy.proxy. However the resource name is not taken from the tag resource.name like it was, and is still ingress.

The reproduction Nomad job above, but using the image envoyproxy/envoy:v1.27.2 produces the following traces: image

So my main question remains: how can I influence the generated Datadog resource name from the tags configuration in the Envoy tracing configuration so it is not the fixed string ingress?

cgilmour commented 11 months ago

Thanks @hiddewie, you're right it corrects only the operation name and not an additional behavior of setting some details via custom_tags. Me and @dgoffredo will take a look at this.

github-actions[bot] commented 10 months ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

hiddewie commented 10 months ago

Backports in #30892 & #30893

github-actions[bot] commented 9 months ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

hiddewie commented 9 months ago

not stale

dgoffredo commented 9 months ago

Heads up: I found another way that Datadog spans can have the wrong "operation name": https://github.com/envoyproxy/envoy/pull/31366

If Envoy cuts a release before that pull request is backported, then most of the situations where "operation name" is incorrect will be fixed, except for one:

github-actions[bot] commented 8 months ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

hiddewie commented 8 months ago

not stale

phlax commented 8 months ago

@hiddewie does this have a corresponding pr/commit on main - generally any changes should be landed on main and backported - not sure of the context here tho

dgoffredo commented 8 months ago

The fix for this is available in the recently released v1.29.0. That release also contains fixes for other Datadog bugs that were introduced in v1.27.

phlax commented 8 months ago

ok - so iiuc this needs to be backported to ~1.26, and~ 1.28 also ?

if there is anything else that still needs backporting please lmk

dgoffredo commented 8 months ago

All of the currently known Datadog bugs that were introduced with v1.27 have been patched on all of: the main branch, the v1.27 release branch, and the v1.28 release branch.

phlax commented 8 months ago

can we close this?

dgoffredo commented 8 months ago

I'm happy to close this, but let's wait for confirmation from @hiddewie that upgrading to v1.29 is a solution.

hiddewie commented 8 months ago

I will do a test-deployment with envoy 1.29 and see how the result is in datadog.

hiddewie commented 8 months ago

I can confirm the 1.29.0 provides equivalent functionality in Datadog as the 1.26.3 version that we were previously running on. This solves this issue from my point of view.

Thanks for all the help!

zirain commented 8 months ago

@hiddewie @phlax should we backport those fixes?

phlax commented 8 months ago

@zirain my understanding from https://github.com/envoyproxy/envoy/issues/30235#issuecomment-1896593037 is that its not necessary

zirain commented 8 months ago

let me check.

dgoffredo commented 8 months ago

@zirain my understanding from #30235 (comment) is that its not necessary

There were three ways that the operation name could be set to an unexpected value. Each fix was part of its own pull request to main, which is now part of the v1.29 release, and each was backported onto both the v1.27 and v1.28 release branches.

PRs:

zirain commented 8 months ago

@dgoffredo thanks for your reply, make sense to me.