search

3scale-ops / prometheus-exporter-operator

Operator to centralize the setup of 3rd party prometheus exporters on Kubernetes/OpenShift, with a collection of grafana dashboards
Apache License 2.0
42 stars 15 forks source link

Logging level for probe exporter should be configurable #27

Closed ValentinLevitov closed 3 years ago

ValentinLevitov commented 3 years ago

https://github.com/3scale-ops/prometheus-exporter-operator/blob/88759a23a74ad9e8122c6da0761cdf4641187e76/roles/prometheusexporter/exporters/probe/container.yml.j2#L3

At present the logging level is hardcoded to 'debug'. The pressure to the logging subsystem is too high. There should be possibility to lower the level. /kind feature

ValentinLevitov commented 3 years ago

/kind feature

slopezz commented 3 years ago

Hi @ValentinLevitov, thanks for reaching us!

Orginally we harcode it because we thought it was ok to always see that the execution of every target is being correctly done:

ts=2020-11-16T20:36:36.413Z caller=main.go:169 module=http_2xx target=https://kubernetes.io level=debug msg="Beginning probe" probe=http timeout_seconds=5 ts=2020-11-16T20:36:36.413Z caller=main.go:169 module=http_2xx target=https://kubernetes.io level=debug msg="Resolving target address" ip_protocol=ip4 
ts=2020-11-16T20:36:36.442Z caller=main.go:169 module=http_2xx target=https://kubernetes.io level=debug msg="Resolved target address" ip=147.75.40.148 ts=2020-11-16T20:36:36.442Z caller=main.go:169 module=http_2xx target=https://kubernetes.io level=debug msg="Making HTTP request" url=https://147.75.40.148 host=kubernetes.io 
ts=2020-11-16T20:36:36.758Z caller=main.go:169 module=http_2xx target=https://kubernetes.io level=debug msg="Received HTTP response" status_code=200 ts=2020-11-16T20:36:36.766Z caller=main.go:169 module=http_2xx target=https://kubernetes.io level=debug msg="Response timings for roundtrip" roundtrip=0 start=2020-11-16T20:36:36.442358373Z dnsDone=2020-11-16T20:36:36.442358373Z connectDone=2020-11-16T20:36:36.451026327Z gotConn=2020-11-16T20:36:36.460991651Z responseStart=2020-11-16T20:36:36.758412614Z end=2020-11-16T20:36:36.765988114Z 
ts=2020-11-16T20:36:36.766Z caller=main.go:169 module=http_2xx target=https://kubernetes.io level=debug msg="Probe succeeded" duration_seconds=0.352932335

Actually, according to https://github.com/prometheus/blackbox_exporter/issues/399 there is only 2 possible values:

But for sure, depending on how many exporter you have you dont want all those log lines, so we can make possible to config it at PrometheusExporter Custom Resource level with a new field called logLevel, whose default value would be debug, and then you can override it with logLevel=info.

What do you think?

ValentinLevitov commented 3 years ago

I guess it would be great. For now I have to reduce the probation interval for my endpoints (there are dozens of them) from 30 seconds to 2 minutes)