AlexanderMann commented 4 years ago

Motivation

Looking at instrumenting a bunch of third party software in our stack. Presently, we're on GCP, and leveraging CloudTrace would be really wonderful.

While we realize that this is a BETA tool, being able to toss this into the command instead of forking the 3rd party project would be...wonderful.

Notes

Presently leveraging tracing in GCP for Java, it's suggested that folks use OpenCensus. While this is fine, it requires meddling in the jar etc.

https://cloud.google.com/trace/docs/setup/java

To get Opencensus working, it's a simple matter of building a StackdriverTraceExporter.

My thinking here would be that this project would offer another integration with environment variables informing it to setup with CloudTrace instead of OTLP.

Suggested Musical Pairing

https://www.youtube.com/watch?v=C24hUt18RWY

AlexanderMann commented 4 years ago

Presently I'm trying to see whether I can leverage the exporter.jar

anuraaga commented 4 years ago

Hi @AlexanderMann! Currently users can use the opentelemetry collector to proxy traces into various backends, including Stackdriver.

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/master/exporter/stackdriverexporter

Are you able to use a sidecar daemon for conversion? Of course, being able to export directly without a proxy is also convenient and we should support that eventually. @james-bebbington does GCP team have any plans in this area?

james-bebbington commented 4 years ago

We have just released an alpha version of a Java Cloud Trace exporter for OpenTelemetry. See:

I haven't looked into the Java auto-instrumentation repo before though, so I'm not sure how to (or if this can be) configured to be used with auto-instrumentation. How are vendor exporters expected to be configured? (we will definitely want to support this eventually)

FYI @zoercai

AlexanderMann commented 4 years ago

This is all grand. My attempts with exporter.jar ran quickly into incompatibility with the previous OpenCensus work for Stackdriver.

Running a sidecar daemon is certainly possible, just not as convenient (as you noted 😉 )

For context:

Working at ApolloGraphQL
We're running a massive Druid installation and want tracing on it
Having modest success with Istio for various things, but ultimately it's nice being able to peak at the innards of applications
Instead of continuing to build our own jars so that we can do StackDriver's recommended approach we were hoping to leverage the Java-agent blah blah blah
DataDog already has support for this but we are hoping to keep more of our information closer to the source for the time being

@james-bebbington your project looks promising, and we'll prolly give that a go at some point this week and report back. We might have some bandwidth to throw behind helping with issues here as well depending on how priorities fall with our project.

AlexanderMann commented 4 years ago

Okayyyy, so fresh from the presses:

Dockerfile:

WORKDIR /tracing

...

curl -sSfL --output exporter-trace.jar "https://repo1.maven.org/maven2/com/google/cloud/opentelemetry/exporter-trace/0.7.1/exporter-trace-0.7.1.jar"

java jar options:

-javaagent:/tracing/opentelemetry-javaagent-all.jar
-Dotel.exporter.jar=/tracing/exporter-trace.jar

K8s logs:

[opentelemetry.auto.trace 2020-09-01 19:51:55:835 +0000] [main] WARN io.opentelemetry.auto.tooling.TracerInstaller - No span exporter found in /tracing/exporter-trace.jar

I'll poke into this more later, but I wanted to share logs earlier than later.

AlexanderMann commented 4 years ago

@anuraaga I'm struggling a bit here to figure out where I should implement the required Factory...based on the examples, it looks like I should make a new one in the javaagent-exporters package, but based on the env var for passing in a separate jar, it seems like that should live inside the jar being pulled in? Any recs here?

AlexanderMann commented 4 years ago

I know this is definitely not what we want long term, but this is thus far showing promise. We're ramping up usage/support tonight and will be able to report back whether this is a vetted path forward: https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/1160

AlexanderMann commented 4 years ago

Next steps right now are bumping the Google Code to latest so that we're not worried about running into the type signature changes.

Exception in thread "BatchSpanProcessor_WorkerThread_1" java.lang.AbstractMethodError: com.google.cloud.opentelemetry.trace.TraceExporter.export(Ljava/util/Collection;)Lio/opentelemetry/sdk/common/export/CompletableResultCode;

at io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.exportCurrentBatch(BatchSpanProcessor.java:266)
at io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.run(BatchSpanProcessor.java:217)
at java.lang.Thread.run(Thread.java:748)

anuraaga commented 4 years ago

@AlexanderMann Thanks for the digging, sorry for the late reply. We currently publish snapshots of javaagent-tooling, though we are considering a small artifact shuffle in #1129. We'll want an additional artifact in the https://github.com/GoogleCloudPlatform/opentelemetry-operations-java repo that uses it and publishes the auto-exporter jar. We'll release 0.8.0 soon with this as well, though only to bintray instead of Maven Central for now. @james-bebbington do you think you'd include an auto-exporter in that repo based on our development releases? Or wait for things to settle?

AlexanderMann commented 3 years ago

See: https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/pull/39

neboduus commented 3 years ago

Hi everyone, is there any guide whch could help me in implementing Tracing using Open-Telemetry and Google Trace? I see there is this exporter available, but I should use a JAR right? Can you please give me some advises about how to do that?

I have to specify that i want to implement Tracing into a Scala Play Project, but from my understanding it should work.

Do you have some advises for this direction?

Thanks

zoercai commented 3 years ago

Hi @neboduus, https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/tree/master/exporters/trace contains instructions on sending traces to Google Cloud Trace with OpenTelemetry, and here is a simple example project. Hope that helps!

neboduus commented 3 years ago

@zoercai Thanks a lot for the reply. But, is this working also for Scala?

zoercai commented 3 years ago

We haven't tested this for Scala, and I don't have any experience with Scala nor Play. I did ask someone who has experience with Scala to take a quick look and he said it should work just like the example unless you wanted "automatic" tracing from Play. Do you want to try it out and let us know if you come across any issues?

neboduus commented 3 years ago

Thanks again for your replies. I can say that I have tested it with Scala and Play and the instrumentation works. I have been able to send 1 trace to Google Stackdriver and visualize it, therefore it works. Thanks for your support, Im gonna write here if some issues arise

open-telemetry / opentelemetry-java-instrumentation

Google CloudTrace Support #1099

Motivation

Notes

Suggested Musical Pairing