buildbarn / bb-storage

Storage daemon, capable of storing data for the Remote Execution protocol
Apache License 2.0
142 stars 91 forks source link

Tunable LogLevels? #182

Open masterlongshanks opened 10 months ago

masterlongshanks commented 10 months ago

I've noticed that the serverside logs, unlike some other CICD products, are empty after startup unless some endpoint has disconnected.

Auth issues, for example, are only reported clientside; as are buffer size mismatches. Would it be possible, please, to add tunable verbosity to the code (across all repos, not just storage), so that a) owners of the service can find out about issues without their users having to act as monitors, and b) debugging new deployments might get a little easier?

No argument with the unix philosophy of silence being golden: simply asking for the option of noise. Thank you for considering the request.

EdSchouten commented 10 months ago

All Buildbarn binaries provide pretty decent support for submitting OpenTelemetry trace spans to another service. We prefer using those, as tracing provides richer semantics than plain logging.

masterlongshanks commented 9 months ago

Thanks for picking up. OTEL and tracing is on our to-do list :-). Once we understand OTEL and have it working properly, that will probably address use case a) in the initial problem statement.

Last week or so someone in slack was having trouble working out jmespath issues, and noted the general serverside silence.

_Hey folks, I've been trying to get metadataExtractionJmespathExpression working for JWT based auth and then wiring that up to executeAuthorizer._

Is use case b) supported by their experience?

masterlongshanks commented 6 months ago

I just learn of https://github.com/grpc/grpc-go/?tab=readme-ov-file#how-to-turn-on-logging

export GRPC_GO_LOG_VERBOSITY_LEVEL=99
export GRPC_GO_LOG_SEVERITY_LEVEL=info

Setting those in the k8s pods just made everything a lot more obvious. Sharing here in the hope that other n00bs are helped, but looks like the issue can be closed.