Bring-your-own-tracing with the jaeger extension

[adleong]

By default, the linkerd-jaeger extension installs an OpenCensus collector and a Jaeger backend.

However, for users who have an existing Jaeger backend, it should be possible to configure the linkerd-jaeger extension to have the oc-collector send traces to that existing Jaeger backend.

Similarly, for users who have an existing OpenCensus collector, it should be possible to configure the jaeger-injector to point injected proxies at that existing collector.

[adleong]

Fixed by https://github.com/linkerd/linkerd2/pull/5499

[ericjee]

Hi @adleong ,Thx for the great work:)
Quick question, How to handle the second scenario as you described since I have my own Opentelemetry collector.

[arunmat]

Looks like the second scenario as described by @adleong is not covered today.

[adleong]

@ericjee @arunmat good catch that the second scenario isn't covered by the docs. are either of you interested in submitting a PR to fix this omission? it should be as simple as setting the webhook.collectorSvcAddr and webhook.collectorSvcAccount chart values (similarly to how we set the collector.jaegerAddr values in the bring-your-own-jaeger example). For reference, the full list of configurable values can be found in the chart readme

[arunmat]

@adleong I think its more than just the doc. I think the helm chart should be modified NOT to deploy the bundled collector and jaeger when custom values are set for webhook.collectorSvcAddr, webhook.collectorSvcAccount and collector.jaegerAddr.

[adleong]

Even though the bundled collector and jaeger will be unused if custom values are set, deploying them doesn't cause any harm and their resource usage should be very minimal. That said, it would be a nice optimization to add logic in the chart to avoid deploying them when they're not necessary.

In my opinion, updating the docs is a good place to start and then optimizing the chart can be done as a second step. How does that sound?

[whiskeysierra]

@arunmat If you're willing to run a kustomize postrender step, you can apply a $patch: delete to the resources you don't want to deploy. See also https://tech.aabouzaid.com/2021/05/delete-a-manifest-from-kustomize-base.html.

[rucciva]

Hi @adleong , is it also possible to use existing otel collector in another namespace? Currently, i have installed jaeger injector inside linkerd namespace and the otel collector is in another namespace, but i don't see any traffic reaching the otel collector port from linkerd proxy. Here are the log from certain i jected proxy

 [     0.001664s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
 [     0.002029s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
 [     0.002042s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
 [     0.002044s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
 [     0.002045s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
 [     0.002046s]  INFO ThreadId(01) linkerd2_proxy: Local identity is ingress-nginx.ingress-nginx.serviceaccount.identity.linkerd.cluster.local
 [     0.002048s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
 [     0.002049s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
 [     0.002050s]  INFO ThreadId(01) linkerd2_proxy: OpenCensus tracing collector at collector-opentelemetry-collector.opentelemetry.svc.cluster.local:55678 (collector-opentelemetry-collector.opentelemetry.serviceaccount.identity.linkerd.cluster.local)
 [     0.012431s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=ingress-nginx.ingress-nginx.serviceaccount.identity.linkerd.cluster.local

Update: tried moving the jaeger injector into the same namespace with otel collector but it still doesnt produce any trace

[whiskeysierra]

Definitely works, we're running that setup. Make sure your collector is part of the mesh, that threw us off at the beginning. The proxies only support TLS when talking to the collector.

On Sun, 7 May 2023, 01:45 I Putu Ariyasa, @.***> wrote:

Hi @adleong https://github.com/adleong , is it also possible to use existing otel collector in another namespace? Currently, i have installed jaeger injector inside linkerd namespace and the otel collector is in another namespace, but i don't see any traffic reaching the otel collector port from linkerd proxy

— Reply to this email directly, view it on GitHub https://github.com/linkerd/linkerd2/issues/5234#issuecomment-1537245419, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADI7HIIHOS3EFGQFTZWBVDXE3PC3ANCNFSM4TVHA4XQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[rucciva]

hi @whiskeysierra , thanks for replying.

my collector is already part of the mesh, in fact i can confirm that it can receive otlp traffic from ingress-nginx

i use the otel helm chart to install the collector with this values.yaml

mode: "deployment"

config:
  receivers:
    opencensus:
    jaeger: null
    prometheus: null
    zipkin: null
  exporters:
    otlp:
      endpoint: datadog.datadog.svc.cluster.local:4317
      tls:
        insecure: true
    logging: null
  service:
    extensions:
      - health_check
      - memory_ballast
    pipelines:
      logs: null
      metrics: null
      traces:
        receivers:
          - opencensus
          - otlp
        processors:
          - memory_limiter
          - batch
        exporters:
          - otlp

ports:
  opencensus:
    enabled: true
    containerPort: 55678
    servicePort: 55678
    protocol: TCP
    appProtocol: grpc
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false

UPDATE the traces is emitted now, need to change ingress-nginx to use opentracing instead of opentelemetry.