How to export to GCP Cloud Trace?

[cjohansen]

I tried to setup trace exports from my Clojure app to GCP Cloud Trace with official OpenTelemetry tooling, but I'm not sure if this is expected to work with clj-otel. Here's what I did:

;; deps.edn
{:deps {,,,
        com.github.steffan-westcott/clj-otel-api {:mvn/version "0.2.6"}
        com.google.cloud.opentelemetry/exporter-trace {:mvn/version "0.20.0"}
        io.opentelemetry/opentelemetry-api {:mvn/version "1.34.1"}
        io.opentelemetry/opentelemetry-sdk {:mvn/version "1.34.1"}
        io.opentelemetry/opentelemetry-exporter-otlp {:mvn/version "1.34.1"}
,,,}}

And in code:

(ns myapp.prod
  (:gen-class)
  (:import (com.google.cloud.opentelemetry.trace TraceExporter)
           (io.opentelemetry.sdk OpenTelemetrySdk)
           (io.opentelemetry.sdk.trace SdkTracerProvider)
           (io.opentelemetry.sdk.trace.export SimpleSpanProcessor)))

(defn init-opentelemetry []
  (let [trace-exporter (TraceExporter/createWithDefaultConfiguration)
        span-processor (.build (SimpleSpanProcessor/builder trace-exporter))
        tracer-provider (-> (SdkTracerProvider/builder)
                            (.addSpanProcessor span-processor)
                            .build)]
    (-> (OpenTelemetrySdk/builder)
        (.setTracerProvider tracer-provider)
        .buildAndRegisterGlobal)))

This code runs, but I don't see any traces in GCP. I also have not wired it to clj-otel in any way, so am not that surprised it didn't work. Any hints on how to achieve this?

[cjohansen]

Edit: I changed the last .build to .buildAndRegisterGlobal, and now I'm running into some conflict with the clj-otel tooling. The exception message was:

java.lang.IllegalStateException: GlobalOpenTelemetry.set has already been called. GlobalOpenTelemetry.set must be called only once before any calls to GlobalOpenTelemetry.get. If you are using the OpenTelemetrySdk, use OpenTelemetrySdkBuilder.buildAndRegisterGlobal instead. Previous invocation set to cause of this exception

Relevant lines from the stack trace:

Caused by: java.lang.Throwable
    at io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:115)
    at io.opentelemetry.api.GlobalOpenTelemetry.get(GlobalOpenTelemetry.java:85)
    at steffan_westcott.clj_otel.api.otel$get_global_otel_BANG_.invokeStatic(otel.clj:15)
    at steffan_westcott.clj_otel.api.otel$get_global_otel_BANG_.invoke(otel.clj:15)
    at steffan_westcott.clj_otel.api.otel$get_default_otel_BANG_.invokeStatic(otel.clj:46)
    at steffan_westcott.clj_otel.api.otel$get_default_otel_BANG_.invoke(otel.clj:39)
    at steffan_westcott.clj_otel.api.metrics.instrument$get_meter.invokeStatic(instrument.clj:47)
    at steffan_westcott.clj_otel.api.metrics.instrument$get_meter.invoke(instrument.clj:30)
    at steffan_westcott.clj_otel.api.metrics.instrument$get_meter.invokeStatic(instrument.clj:41)
    at steffan_westcott.clj_otel.api.metrics.instrument$get_meter.invoke(instrument.clj:30)
    at steffan_westcott.clj_otel.api.metrics.instrument$get_default_meter_BANG_.invokeStatic(instrument.clj:68)
    at steffan_westcott.clj_otel.api.metrics.instrument$get_default_meter_BANG_.invoke(instrument.clj:62)
    at steffan_westcott.clj_otel.api.metrics.instrument$instrument.invokeStatic(instrument.clj:257)
    at steffan_westcott.clj_otel.api.metrics.instrument$instrument.invoke(instrument.clj:221)
    at steffan_westcott.clj_otel.api.metrics.instrument$instrument.invokeStatic(instrument.clj:254)
    at steffan_westcott.clj_otel.api.metrics.instrument$instrument.invoke(instrument.clj:221)

[steffan-westcott]

I don't have any experience with Google Cloud Platform, so unfortunately I don't have solid advice. However, it may be easier to use Google Cloud Exporter in Collector Contrib rather than programmatically configure the SDK in your application.

[cjohansen]

I don't think GCP specifics are important here - the question is more one of how can I set a TraceExporter instance that clj-otel will use?

I checked out your link, but I honestly don't understand what it is or how to use it. Looks like a lot of go code? Is it a separate service? I really just want to export traces from my JVM-app directly to GCP.

[steffan-westcott]

The OpenTelemetry Collector is a runnable process that forwards telemetry data from applications to telemetry backends. Almost all of the examples in clj-otel demonstrate use of the Collector.

Collector Contrib is an extended version of the standard Collector that has extra receivers, processors and exporters. Google Cloud Exporter is present in Collector Contrib. The Google documentation for working with OpenTelemetry in Java has a quickstart example that uses Google Cloud Exporter.

I suggest getting traces working using the Collector as per the Google quickstart example. Programmatically configuring the SDK is possible, but is the most challenging option to get working correctly.

[steffan-westcott]

If you decide not to use the Collector, the next easiest option is to use the OpenTelemetry autoconfigure SDK extension. Google provide a library that works with this. The shaded variant can, in addition, be used as an extension to the OpenTelemetry instrumentation agent. Here is a full program with manual instrumentation that uses the agent extension:

deps.edn

{:paths   ["src"]
 :deps    {org.clojure/clojure                      {:mvn/version "1.11.3"}
           com.github.steffan-westcott/clj-otel-api {:mvn/version "0.2.7"}}
 :aliases {:otel {:jvm-opts [;; Download the following from
                             ;; https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.5.0/opentelemetry-javaagent.jar
                             "-javaagent:opentelemetry-javaagent.jar"

                             ;; Download the following from
                             ;; https://repo1.maven.org/maven2/com/google/cloud/opentelemetry/exporter-auto/0.30.0-alpha/exporter-auto-0.30.0-alpha-shaded.jar
                             "-Dotel.javaagent.extensions=exporter-auto-0.30.0-alpha-shaded.jar"

                             "-Dotel.resource.attributes=service.name=my-app"
                             "-Dotel.traces.exporter=google_cloud_trace"
                             "-Dotel.metrics.exporter=google_cloud_monitoring"
                             "-Dotel.logs.exporter=none"

                             ;; Supply your configuration details below
                             "-DGOOGLE_CLOUD_PROJECT=<my-project-id>"
                             "-DGOOGLE_APPLICATION_CREDENTIALS=<my-credentials>"]}}}

src/org/example/my_app.clj

(ns org.example.my-app
  (:require [steffan-westcott.clj-otel.api.trace.span :as span]))

(defn foo []
  (span/with-span! "Doing foo things"
    (println "foo")))

[cjohansen]

Thanks a lot for all the pointers 👍 🙏