search

KohlsTechnology / prometheus_bigquery_remote_storage_adapter

Prometheus remote storage adapter for Google's BigQuery
Apache License 2.0
44 stars 17 forks source link

Add Tracing For Additional Observability #45

Open seanmalloy opened 2 years ago

seanmalloy commented 2 years ago

What happens?

Currently the prometheus_bigquery_remote_storage_adapter application only supports logs and prometheus metrics for observability.

What were you expecting to happen?

I would like to be able to use tracing to monitor/observe prometheus_bigquery_remote_storage_adapter when running in a production environment.

Steps to reproduce:

Run prometheus_bigquery_remote_storage_adapter and notice that only logs and prometheus metrics are provided.

Any errors, stacktrace, logs?

N/A

Environment:

Additional comments:

From my perspective it seems like OpenTelemetry would provide a nice vendor neutral option for enabling tracing.

wdoogz commented 2 years ago

@seanmalloy were you thinking about offering tracing as a flag in our code when the application is started it'll begin sending the traces to a OTel collector (such as maybe another sidecar running within the same pod as the Prom BQ remote storage adapter), which will send it to whatever backend a user wishes?

seanmalloy commented 2 years ago

@seanmalloy were you thinking about offering tracing as a flag in our code when the application is started it'll begin sending the traces to a OTel collector (such as maybe another sidecar running within the same pod as the Prom BQ remote storage adapter), which will send it to whatever backend a user wishes?

@wdoogz my simple answer is yes I agree with you.

I'm not an OpenTelemetry expert, but I think we would want to keep this feature as vendor agnostic as possible, so using the OpenTelemetry collector seems like a good idea.

https://opentelemetry.io/docs/collector/

Also, having a flag to enable/disable tracing seems like a good idea.