search

grpc / grpc-java

The Java gRPC implementation. HTTP/2 based RPC
https://grpc.io/docs/languages/java/
Apache License 2.0
11.4k stars 3.83k forks source link

Cannot use Google Cloud Logging and Google Cloud Observability together (`IllegalStateException: Set cannot be called after any get call`) #11048

Open pkbriggs opened 6 months ago

pkbriggs commented 6 months ago

What version of gRPC-Java are you using?

1.62.2

What is your environment?

Ubuntu 20.04.6 LTS, openjdk 17.0.10

What did you expect to see?

Be able to successfully run an application with Google Cloud Logging + Google Cloud Observability

What did you see instead?

An exception is thrown when running the application:

Exception in thread "main" java.lang.IllegalStateException: Set cannot be called after any get call
    at io.grpc.GlobalInterceptors.setInterceptorsTracers(GlobalInterceptors.java:61)
    at io.grpc.InternalGlobalInterceptors.setInterceptorsTracers(InternalGlobalInterceptors.java:29)
    at io.grpc.gcp.observability.GcpObservability.setProducer(GcpObservability.java:164)
    at io.grpc.gcp.observability.GcpObservability.grpcInit(GcpObservability.java:116)
    at io.grpc.gcp.observability.GcpObservability.grpcInit(GcpObservability.java:98)

Steps to reproduce the bug

Minimal app showing issue: https://github.com/pkbriggs/gcp-logging-observability-issue

  1. Create new Java app that uses both Google Cloud Logging (https://cloud.google.com/logging/docs/setup/java) and Google Cloud Observability (https://cloud.google.com/stackdriver/docs/solutions/grpc/set-up-observability)
  2. Add a logback.xml as described in the GCP Logging docs
  3. Run app (GRPC_GCP_OBSERVABILITY_CONFIG="{}" ./gradlew run) and observe that an exception is thrown when GcpObservability.grpcInit() is called

If you delete/rename src/main/resources/logback.xml and run the app, it will run successfully. This seems to indicate there is some issue with com.google.cloud.logging.logback.LoggingAppender and io.grpc.gcp.observability.GcpObservability.

ejona86 commented 6 months ago

For the reproduction, the "simple solution" is "don't log before GcpObservability.grpcInit()". It might even require delay calling LoggerFactory. But I understand that is easier said than done in a real app, and may not be desired.

Even if you get it running, there are serious risks of forming a cycle between GCP Logging and GcpObservability.

@DNVindhya, do you have ideas?

pkbriggs commented 6 months ago

Thanks for the reply! Unfortunately the issue still persists if the logger.info call is moved after GcpObservability.grpcInit(). Deferring LoggerFactory.getLogger does seem to fix the issue but, as you pointed out, would be burdensome in our app.

I'm a bit confused about the intended use case for the Observability and Logging products: given Observability hooks up to GCP Logging, does that mean we can remove our direct dependency on GCP Logging? Is Observability meant to supersede the individual Logging and Metrics integrations? Are they intended to work together? It would be a bit silly to say "you can either have observability or logging, but not both!"