Envoy and Jaeger tracing

pvyaka01 commented 4 years ago

When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.

Overview of the Issue

Trying to get envoy / Jaeger tracing to work. I added tracing configurations to the service and registered the services. I've not configured any applications for spans and such but expecting to see calls from ingress to services show up in Jaeger UI. Traces are not being generated.

Reproduction Steps

Setup service A (no opentracing) with side-car and below envoy configurations

config {
  envoy_prometheus_bind_addr = "0.0.0.0:9102"

  envoy_extra_static_clusters_json = <<EOL
    {
      "connect_timeout": "3.000s",
      "dns_lookup_family": "V4_ONLY",
      "lb_policy": "ROUND_ROBIN",
      "load_assignment": {
          "cluster_name": "jaeger_9411",
          "endpoints": [
              {
                  "lb_endpoints": [
                      {
                          "endpoint": {
                              "address": {
                                  "socket_address": {
                                      "address": "jaeger",
                                      "port_value": 9411,
                                      "protocol": "TCP"
                                  }
                              }
                          }
                      }
                  ]
              }
          ]
      },
      "name": "jaeger_9411",
      "type": "STRICT_DNS"
    }
  EOL

  envoy_tracing_json = <<EOL
    {
        "http": {
            "config": {
                "collector_cluster": "jaeger_9411",
                "collector_endpoint": "/api/v1/spans"
            },
            "name": "envoy.zipkin"
        }
    }
  EOL

Setup ingress gateway with consul connect
start jaeger-all-in-one (./jaeger-all-in-one --log-level=debug)
Make a curl call from ingress to the service
Jaeger logs do not show any traces coming in
curl call shows that call is going through envoy:

$ curl -vvv http://x.x.x.x:8080/<service>/version
* About to connect() to x.x.x.x port 8080 (#0)
*   Trying x.x.x.x...
* Connected to x.x.x.x. (x.x.x.x) port 8080 (#0)
> GET /<service>/version HTTP/1.1
> User-Agent: curl/7.29.0
> Host: x.x.x.x:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< content-length: 31
< date: Wed, 12 Aug 2020 21:39:49 GMT
< x-envoy-upstream-service-time: 6
< server: envoy
<
* Connection #0 to host x.x.x.x left intact
Deployed Version: 20.03.01.1341

config_dump shows this:

{
     "cluster": {
      "@type": "type.googleapis.com/envoy.api.v2.Cluster",
      "name": "jaeger_9411",
      "type": "STRICT_DNS",
      "connect_timeout": "3s",
      "dns_lookup_family": "V4_ONLY",
      "load_assignment": {
       "cluster_name": "jaeger_9411",
       "endpoints": [
        {
         "lb_endpoints": [
          {
           "endpoint": {
            "address": {
             "socket_address": {
              "address": "jaeger",
              "port_value": 9411
             }
            }
           }
          }
         ]
        }
       ]
      }
     },
     "last_updated": "2020-08-12T21:26:29.744Z"
    }

And:

    "tracing": {
     "http": {
      "name": "envoy.zipkin",
      "hidden_envoy_deprecated_config": {
       "collector_endpoint": "/api/v1/spans",
       "collector_cluster": "jaeger_9411"
      }
     }
    }

Consul info for both Client and Server

Server:

Consul v1.8.0
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Envoy version:

$ envoy --version

envoy  version: 1a0363c885c2dbb1e48b03847dbd706d1ba43eba/1.14.2/clean-getenvoy-fbeeb15-envoy/RELEASE/BoringSSL

consul info
agent:
        check_monitors = 3
        check_ttls = 0
        checks = 15
        services = 12
build:
        prerelease =
        revision = 3111cb8c
        version = 1.8.0
consul:
        acl = disabled
        known_servers = 3
        server = false
runtime:
        arch = amd64
        cpu_count = 2
        goroutines = 142
        max_procs = 2
        os = linux
        version = go1.14.4
serf_lan:
        coordinate_resets = 0
        encrypted = true
        event_queue = 0
        event_time = 65
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 10607
        members = 302
        query_queue = 0
        query_time = 1

bbros-dev commented 2 years ago

Information is scattered, so adding this fairly detailed demo/example here for future self and others:

https://github.com/nicholasjackson/demo-consul-service-mesh/

Specifically, where some Jaeger details are worked out:

https://github.com/nicholasjackson/demo-consul-service-mesh/tree/master/metrics_tracing

berylshow commented 2 years ago

信息是可分割的，所以在这里添加非常详细的演示/示例以供用户的自己和其他人使用：

https://github.com/nicholasjackson/demo-consul-service-mesh/

具体来说，其中一些 Jaeger 的细节得到了解决：

https://github.com/nicholasjackson/demo-consul-service-mesh/tree/master/metrics_tracing

@bbros-dev hello ！I tried to use the example you provided to test, but the results were not ideal. The application introduced a span that relied on opentracing and directly sent to Jaeger. I hope to see the span information created by envoy on Jaeger's UI, but I didn't get it. Can you look at the problem for me? thank you very much！I asked a question on GitHub. Do you have time to see it? thank you very much! https://github.com/hashicorp/consul-k8s/issues/967

bbros-dev commented 2 years ago

Sorry I can't be more help @berylshow - this will have to await Hashicorp providing a learning resource or someone providing a blog post. Perhaps you might contribute to the https://github.com/nicholasjackson/demo-consul-service-mesh/ repository by opening an issue there detyailing what you found to be unexpected?

M-Whitaker commented 1 year ago

Hi, I'm also confused by this. It seems all the consul demos manually publish the traces from the application. Surely this means envoy is doing nothing?

hashicorp / consul