Start using ENV variables for debugging setup

[RicardoMonteiroSimoes]

Is your feature request related to a problem?

Not really, except that it's turning me insane getting adequate logging turned on and moved into production on a kubernetes architecture. Also I can't fathom why we wouldn't also use the same ENVs that are described here: https://grpc.github.io/grpc/cpp/md_doc_environment_variables.html For example, the Node.js implementation works perfectly fine with those and getting adequate debugging running took some minutes.

Describe the solution you'd like

Introduce ENV scanning to the startup phase of GRPC that specifically check variables defined here https://grpc.github.io/grpc/cpp/md_doc_environment_variables.html to setup the complete logging process. This would make life just so much more easier than it is right now. It would also make it easier to set it up, because there is a page that documents it. I've had to scrape together stack overflow answers to get a solution that might work.

Describe alternatives you've considered

The solutions I've scraped together (Setting it in code, using property files etc) work well enough in a local isolated fashion, but it's just not that easy to do so in a big cloud based application.

It might be that this already works for the current version (doesn't seem like it from my research)

[sergiitk]

You can use java.util.logging.config.file and point to a logging.properties file to fine-tune the logs you want to see.

Take a look at this Dockerfile we use for e2e tests: https://github.com/grpc/grpc-java/blob/f229aed538b8cb5bfd2e7b331d246f6b7d8b3a4b/buildscripts/xds-k8s/test-server.Dockerfile#L10-L12

It points to logging-json.properties, which logs in json (useful for indexed logging). There's a few more logging examples files we bundle in the image:

• logging.properties - Plain text logging with some loggers we care about configured to log at FINEST
• logging-debug.properties - Same as above, plus io.netty configured to FINEST (it's pretty verbose).

While this is a Docker/k8s example, you can use this approach to configure logging elsewhere. You can also override JAVA_OPTS ENV variable to point to another file, f.e.JAVA_OPTS="-Djava.util.logging.config.file=another-logging.properties".

[sergiitk]

Also I can't fathom why we wouldn't also use the same ENVs that are described here

Sorry to hear this is confusing. There's a good reason why it's like that: each gRPC implementation is using its native language logging system. The one you linked is only C++. Yes, nodejs mimicked C++'s, but I'd argue node's logging is much less prescriptive and much more flexible that Java's. Keep in mind node is 14 years younger than Java.

See more examples how other languages configure logging here: https://cloud.google.com/traffic-director/docs/troubleshooting-proxyless → Enable logging in gRPC runtime.

[RicardoMonteiroSimoes]

Yes, I've seen that after some time, but I was expecting it to be available at https://grpc.io/docs/languages/java/, but it seems that there are multiple documentation sources.

The file variant works locally perfectly fine, it's just not directly possible with our k8s setup (image build through build folder which isn't synced :/ ). There are workarounds and I'm trying to make it work, but it would be appreciated if there was one unified solution for this. Java supports reading out ENV_VARs, so why not unify it? Would also simplify the documenting it all.

[sergiitk]

We're working on improving the documentation. As usual, that's the tough part 😄 FYI @temawi - could you please double check if we have something for logging docs in our pipeline?

@ejona86 - what do you think about having some predefined loggers configurations based on the contents of GRPC_TRACE (or maybe even GRPC_JAVA_TRACE, similar to node)?

[sergiitk]

The file variant works locally perfectly fine, it's just not directly possible with our k8s setup (image build through build folder which isn't synced :/ ) There are workarounds and I'm trying to make it work, but it would be appreciated if there was one unified solution for this.

While we don't have a solution, consider configuring your loggers programatically. Generally I won't advice this, but with the limitations of your setup, it might be a reasonable solution.

[ejona86]

We are using the "unified" solution. But we are "unified" with Java, not other grpc languages. Yes, logging configuration in Java sucks (even if you consider log4j/slf4j). But "welcome to Java." Or "Java prides itself in not having nice things."

I don't see us making something grpc-specific here. What should happen is some person gets fed up and makes a java.util.logging.config.class that reads from system properties or environment variables. I've used such things before, but I'm not aware of such an OSS class. It'd be just a few lines of code to make one that copies the contents of such a variable to LogManager.readConfiguration(InputStream).

[RicardoMonteiroSimoes]

Thanks for your replies!

I can see your point of view and it certainly is one that makes sense, as always there are many paths and you have to decide on one. Documentation does seem to be the hardest problem, but if it's being worked on, then I guess that's also fine.

In my case I found a different solution: Our parent "service" that all inherit from contains a logback.xml in the resources folder, that has some generic settings for all services set. Using <include resource="logback-service-specific.xml" optional="true"/> and then defining a logback-service-specific.xmlin the resources of a given service, I was able to set everything up as necessary.

Thanks for your time!

I will not close this as maybe there is an inherent discussion to be had about this, but feel free to close it if its unnecessary :)

[ejona86]

Ah, okay. So you didn't need dynamic logging configuration. You needed static logging configuration within your binary. That is something that log4j/slf4j do handle about as well as could be expected.